Seven Dangerous Questions Using IT and Assessment to Transform Academic Programs

1
Seven Dangerous QuestionsUsing IT and
Assessment to Transform Academic Programs

• Stephen C. Ehrmann, Ph.D.
• March 15, 2002

2
Thanks!

• 250 institutional subscribers to TLT Group
  services
• Annenberg/CPB and AAHE
• Washington State Univ., St. Edwards University,
  Indiana University
• TLT Group Founding Sponsors
• Blackboard, Compaq, Microsoft, SCT, WebCT
• EDUCAUSE/NLII, CNI, AACU, League, AASCU
• TLT Group Program Funders
• FIPSE, Mellon Foundation, NSF, APS, PT3, LAAP...

3
Outline

• Whats at Stake- 7 Dangerous Questions
• Transformative Assessment? Alignment?
• What are you trying to improve
• Criteria
• Examples
• What power could assessment have?
• Answer the 7 dangerous questions What sorts of
  data would be most useful
• Assessment Infrastructure

4
The Vision (1960 present)

• Technology, properly used, can improve education
  and its outcomes (the Triple Challenge who
  learns, what they learn, what it costs)
• Technology improves continually (Moores Law,
  etc.)
• Therefore with every passing decade we ought to
  be able to make further improvements in education
  and its outcomes.

5
Assessment isnt for Wimps

• Dangerous questions are
• Risky to ask!
• Even riskier not to ask! (especially if other
  institutions or your constituents are beginning
  to ask) hypothesis

6
Seven Dangerous Questions

1. As educators, how are we trying to improve
   results (from what baseline)?
2. Early in our effort to improve, is our behavior
   changing enough?
3. Where it isnt, whats standing in the way?
4. Is technology proving useful in changing what
   were doing?
5. Where it hasnt been used that way, why not?
6. Costs and stresses what are they? Can they be
   controlled?
7. Now that years have passed, have we moved enough
   (compare with 1)

7
TA Alignment
8
What You Can Study Triads
Activity structure in use
Goal - Outcome
I.T.

• The smallest number of elements to study if
  youre trying to understand and shape the use of
  technology to improve outcomes

9
Defn of Transformation?

• Important, qualitative change in

Activity structure in use
Goal - Outcome
I.T.
10
Criteria for Picking Triad

• Worth pursuing long enough to improve outcomes (5
  years? More?)
• Some aspect of this triad is already a strength
  and perhaps also a worry
• Improving IT can play a cumulative role in
  improving activity in this triad
• Caution IT is just the yeast, not the whole
  recipe

11
Example of a Triad
Outcome/Goal Activity-Structure Technology(s) and other needed elements
Better Learning outcomes 7 Principles (e.g., student-faculty interaction, student collaboration, active learning, time on task) Communications Productivity tools Skilled faculty, spread of low threshold activities (e.g.,, managing collaboration)
12
Not This
Outcome/Goal Activity-Structure Technology(s) and other needed elements
Whatever Infrastructure used a lot, with great satisfaction Infrastructure Tech support
13
Another Triad
Outcome/Goal Activity-Structure Technology(s) and other needed elements
Community, during after college Plentiful interaction, formal and informal, including outsiders Communications directories, etc. Staff who are good at collaboration outside partnerships
14
And Another
Outcome/Goal Activity-Structure Technology(s) and other needed elements
Speed to degree access Hybrid courses Altered recruiting and student info CMS, e-mail, Web, etc. Help in redesigning courses so they take more advantage of online less need for f2f
15
And Another
Outcome/Goal Activity-Structure Technology(s) and other needed elements
Skills of design in distributed environments Make part of education into such an environment practice feedback Networks, design tools Skilled faculty, corporate linkages problems
16
Triad Yours?
Outcome/Goal Activity-Structure Technology(s)




17
How to Focus (A Triad and Transformative
Assessment)

• Top down commission, strategic plan,
  accreditation self-study
• Bottom up- build momentum, coalition
• Voluntary effort aimed at silent start, slow
  growth
• Brown bags
• Workshops and talks debates of the issues, warts
  and all
• Voluntary studies
• Grants and gifts
• Community surveys, meetings
• Recognize the political side of the process (in
  best, worse senses of the term)

18
Assessments Powers

1. The discussion of how to collect data can help to
   make discussion of goals and strategies less
   abstract, agreements more meaningful
2. Discussion of alignment can help, too.
3. Periodic findings can refocus attention
4. Spotlight problems as well as achievements in
   order to attract resources
5. Steering, e.g., grab hidden opportunities, reduce
   or remove barriers

19
Seven Dangerous Questions

1. (Sooner) baseline of outcome
2. Is the activity on the increase?
3. Where it isnt whats standing in the way?
4. Technology helping?
5. Barriers to IT effectiveness?
6. Costs and stresses from the effort?
7. (Later) Outcome improving?

20
Assessment Costs, Too!

• People to do programmatic studies? Offices and
  budgets? Missions?
• Current activities (helping others do studies?
  Training others?)
• Assessment linked to mission and strategy?
  Autonomous? Adrift?
• Reward system for people who take the risk of
  spending time, money on studies?
• Training for them? Expert assistance? Tools for
  doing studies? Dogwork assistance?

21
Were Done!

• Questions or comments?
• Give me a card or send me e-mail if you want
  these slides, information about how Flashlight
  tools or Network membership might help your
  institution move on these issues
  Ehrmann_at_tltgroup.org