Hard Choices for Individual Situations: Selecting a Course Management System

1
Hard Choices for Individual Situations Selecting
a Course Management System

• By Bruce Landon, Ph.D
• Douglas College
• Western Cooperative for Educational
  Telecommunications

2
Why choices are difficult

• The newness of the course management software as
  an educational innovation
• Too many Products
• Too many Products Features
• You could make a big mistake here

3
Too many products

• Commercial Products
• Not yet profitable
• Open Source Products
• Many Unknowns OKI
• Training Products
• Related and similar, but larger market
• Institutional Products
• Many in-house proprietary systems

4
Too many product features

• 50 plus features
• With each new version there are additional
  features
• Marketing often focuses on special features or
  names of features in the intense competition

5
Difficulties inside of the Decision-Makers

• Overconfidence in historic strategies for making
  decisions
• Too little working memory for decision problems
  this large
• Too many Cognitive Illusions

6
cognitive illusions

• are due to limited focus

7
How Many can you juggle?
8
Even a Mathematician is limited by Memory Span
9
Some are more limited
10
Some can handle more
11
But Seven Nine is Maximum and Ten is only in
your dreams
12
When there are too many
13
Some of them get dropped
14
Cognitive Illusions

• Availability Heuristic
• Representativeness Heuristic
• The Framing Effect

15
Availability Heuristic

• Powerful cognitive distortion
• like Out of Sight - Out of Mind
• Believability is related to ease of recall
• if one cannot remember it
  then it must not be true
• Distorted by vividness of information
• Distorted by number of repetitions, like
  ads on the radio and television

16
Representativeness Heuristic

• Powerful cognitive distortion
• if it looks like one then it is one
• in spite of relevant base rate information
• (you can tell a book by its cover)
• (if looks good then it works well)

17
Does Monday look like this?
18
Or does Monday look like this?
19
The Framing Effect

• Refers to effect of the frame of reference
• Risk Frame people tend to take risks described
  in terms of loss like losing enrolments
• Benefits Frame People tend to avoid risks that
  are described in terms of benefits gaining more
  enrolments

20
Framing is like context for the size of the
circle in the middle

• It only looks different in different contexts

21
Effect of More Options delaying and avoiding

• In a high conflict situation where there are many
  alternatives, the decision makers tend to
  postpone making a decision altogether
• The effect has been observed in physicians, when
  offered more alternative drugs fewer opted to try
  any new drug
• The effect has also been observed in members of
  the Ontario Provincial Parliament when they were
  requested to choose among different types of
  health care expenditures

22
The ultimate fallibility is Overconfidence

• The tendency to be more confident than is
  warranted by the evidence
• To overestimate the accuracy of one's beliefs
  and judgments (availability heuristic again)
• For example, the confidence of an eye witness in
  their testimony is unrelated to the accuracy of
  that testimony
• This overestimation of confidence enhances
  personal self-esteem and contributes to the
  resistance to being persuaded otherwise
• Ironically, people often are most overconfident
  when most incompetent Myers (2001, p 527)

23
How people normally Make Decisions
The 5 Basic Strategies

• One Reason strategy
• Elimination by aspect strategy
• Satisficing strategy
• Equal weight strategy
• Weighted averaging strategy

24
One Reason strategy

• (aka pick the best)
• Step 1 select most important feature
• Step 2 pick product best on that feature
• (no need to use numbers)
• Non-Compensatory in that other features can not
  make up for not being best on the most important
  feature

25
Elimination by aspect strategy

• (aka pick the last one standing)
• Step 1 set the requirements for each feature
  the minimum criteria
• Step 2 eliminate options one at a time if any
  feature does not meet the minimum feature
  requirement
• (does not necessarily require numbers)
• Non-Compensatory in that once a product is
  rejected on any feature it is eliminated
• This strategy does not always select only one
  best option

26
Satisficing strategy

• (aka Bounded Rationality Model)
• Step 1 setting cutoff levels for each of the
  features or criteria
• Step 2 an option is examined until it fails on
  any criteria and then is eliminated
• Step 3 The first option that passes all feature
  criteria is the one selected
• Step 4 (optional) if none of the products pass
  then the cutoff requirements are reduced and the
  process is repeated
• Non-Compensatory and is effected by the order in
  which the options are considered

27
Equal weight strategy

• (aka scoring strategy)
• Step 1 Set pass/fail criteria for each feature
• Step 2 Assign suitability scores (1,0) to every
  feature on all of the products
• Step 3 Total the suitability scores for all
  products
• Step 4 The option with the highest total is
  selected (there may be a tie when only a few
  features are considered)
• Strategy is Compensatory in that some feature
  suitability scores can compensate for other
  missing or failing features

28
Weighted averaging strategy

• (aka weighted adding strategy, grading model)
• Step 1 Set importance weightings for each
  feature
• Step 2 Assign suitability scores for every
  feature
• Step 3 Multiply the scores times their weights
• Step 4 Sum the weighted feature subscores into a
  total score for each option
• Step 5 Select the option with the highest
  weighted average score or highest sum
• This is a compensatory strategy both in terms of
  the weights and in terms of the suitability
  scores
• This method is considered normatively rational
  decision process because it uses all of the
  information in a consistent manner

29
Screening tools on the web

• Finding products with specified features using
  www.edutools.info/landonline/
• Selecting specified features to find products
  supporting those features.

30
Making a Short List of Application Options

• The optional applications are more like business
  partners in a continuing arrangement so there are
  usually additional considerations required than
  mere product features
• In the end there is a trade-off in how many to
  consider and how much time is available for the
  decision

31
Evaluating product suitability

• The recommended approach is to use the most
  rational decision strategy even though it is
  more work
• An crucial part of the work is to assign the
  relative importance weighting to each of the
  decision criteria
• This can be done in a way that facilitates
  political endorsements of the decision process
  involving stakeholders

32
Inviting the shortlist of vendors to make
competitive proposals (RFP model)

• This ensures that the most recent product
  information will be available in spite of rapid
  change in product evolution
• The RFP process shifts much of the information
  gathering task to the vendor rather than the
  buyer
• This process provides a preview of what it would
  be like to work with different vendors before
  making a expensive commitments
• Often vendors will provide on-site presentations
  that can be the basis for product comparison and
  suitability ratings a product competition

33
Judging product feature suitability to local
situation

• The same judges do not have to judge all features
  it is better to use judges with expertise in
  the specific feature to comparatively judge the
  products
• Persons with disabilities should be among the
  ranks of the judges or the accessibility of the
  products will likely be overestimated
• Consider the inclusion of new users whenever
  the system will become the front door to the
  institution for new students and faculty.

34
Demonstration model

• of the Comparative Analysis Decision Table with
  Three Options
• Using features
• Discussion Forums
• Course Layout Templates
• Course Management
• Using the edutools.info tools

35
Rechecking by doing Sensitivity Analyses

• You can change the weights and rescore
• You can change the ratings and rescore
• It is often comforting to know that the winner
  would not change even if the weights were a
  little off or if some of the suitability ratings
  had been a little different

36
Thank You for your Attention

• http//www.edutools.info
• Bruce_Landon_at_douglas.bc.ca