Shining a Flashlight on Education: Evaluations that Guide Improvement

1
Shining a Flashlight on Education Evaluations
that Guide Improvement

• Norm Vaughan, PhD
• Teaching Learning Centre
• nvaughan_at_ucalgary.ca

2
Purpose of Classroom Assessment

• The central purpose of classroomassessment is
  to empower both teachers and their students to
  improve the quality of learning in the
  classroom.
• Angelo and Cross, 1993

3
Purpose of Service Assessment

• The central purpose of service assessment is to
  empower both staff and their clients to improve
  the quality of services in the institution.
• Steve Ehrmann, 2005

4
Purpose of Flashlight Online

• Originally designed to help assess the impact of
  technology on education
• Flashlight Online provides a carefullydesigned
  set of tools to create, distribute, collect, and
  collate survey information via the Internet to
  assess almost anything

5
Triad Model
6
Resources in Flashlight Online

• Templates - Pre-made surveys that can be copied
  and used just as they are.
• Sample Surveys -Pre-made surveys that contain
  sample questions you should edit before using
  with your students.
• Flashlight Question Bank More than
  60categories of pre-written and tested survey
  items
• Custom Surveys - You can design and deliveryour
  own custom surveys with the software. Write all
  your own items or mix yours with questions from
  the system.

7
Example Templates

• Classroom response systems (clickers)
• Deep versus surface approaches to learning
• Diagnosing barriers to online collaboration among
  students
• Feedback on PowerPoint use in course
• Evaluating Educational Uses of the Web in Nursing
• More on templates

8
What do you want to investigate?

• It is very important to narrow the scope of the
  survey when thinking about the items to choose.
• Begin with a central research question. Think
  about what you want to investigate and try to
  capture your thoughts in one central question.
• Example What barriers exist that would keep my
  online students from workingcollaboratively with
  each other?

9
What should you ask?

• Think about some specifics you would need to know
  to answer your general question. Make a list.
  Share the list with someone sitting close to you.
  Check the following
• Are all the topics NECESSARY to answer the
  general question? Remove any that are not.
• Will the topics provide SUFFICIENT information to
  answer the general question? Add any that are
  needed.
• BEWARE of the It would be nice to
  knowphenomenon!

10
Flashlight Online System
Login ftdays Password - ftdays
http//ctlsilhouette.ctlt.wsu.edu/CTLSilhouette2_5
/Default.asp
11
Important Dos and Donts

• Do
• Ask questions thatmatter
• Have someone elseread the survey beforeyou use
  it
• Tell students what youlearned from anysurvey
  you ask them totake
• Act on what you learn!

• Dont
• Ask the question if youdont really want toknow
• Start a survey inFlashlight that youhave not
  proof read

12
Questions, Comments, Discussion
PowerPoint Slides and Resources Available
at http//www.ucalgary.ca/nvaughan/norm/presenta
tions.htm
13
Resources

• Asking the Right Question What Does Research
  Tell Us About Technology and Higher Learning?
• http//www.tltgroup.org/resources/Flashlight/Askin
  gRightQuestion.htm
• The Flashlight Program Home Page
• http//www.tltgroup.org/programs/flashlight.html
• Flashlight Resources for the University of
  Calgary
• http//www.tltgroup.org/calgary.htm
• Flashlight Online System
• http//ctlsilhouette.ctlt.wsu.edu/ctlsilhouette2_5
  /
• Tip Sheet for Users of Flashlight Online
• http//www.tltgroup.org/resources/Flashlight/Helps
  heet.htm