Research in Education

1
Research in Education A Perspective from an
Outside Firm
Ted Hodge Compustat Consultants Inc. CIRPA
Conference October 26, 2004
2
Agenda
Introduction to Compustat Survey
Methodology Paper-Based Surveys Telephone
Surveys On-line Surveys Reports and
Analysis Working Together Conclusion/ QA
3
Introduction to Compustat
Survey research since 1989 with clients in
education, business, and government. Full-service
research firm, experienced in - survey
design, - data collection, - analysis and
report writing, - presentations and
workshops. Extensive surveying with educational
institutions
4
Types of Education Surveys
Students Residents/Tenants Graduates Employers Em
ployees Faculty Program Advisors Public
Opinion Satisfaction Key Performance
indicators Engagement Demographics Marketing Reten
tion Employment
5
Currently
General objectives of surveys in
education Measure performance Foster
improvement Build competitive advantages Wider
variety of survey types Advantage of multiple
data collection methodologies, new technologies
and specialized expertise More collaboration
between institutions and standardized survey tools
6
Data Collection Methodologies

• (1) Paper-based Surveys
• scannable and data entry
• (2) Telephone Surveys
• (3) On-line Surveys
• (4) Other Methodologies
• interviews, focus groups

7
Factors in Choosing a Data Collection Methodology
Target population Response rate
Labour/technology requirements Cost
effectiveness
8
Statistical Accuracy
Response rate affects statistical accuracy. For
example Population Sample Size Accuracy
10,000 100 /- 9.8 1000
100 /- 9.3 10,000 300 /- 5.6
1000 300 /- 4.7 10,000 700
/- 3.6 1000 700 /- 2.0 -
Worst-case scenario i.e., accuracy of a
proportion at 50. - Accuracy increases as
proportions move towards 0 or 100.
Remember representation is important as well!
9
Paper-Based Surveys
Course/Teacher Questionnaire
Speed, accuracy, and quality assurance through
advanced Optical Mark Reading (OMR) technology
Based on your experience in this course, would
you recommend this course to a friend with the
same interests?
Yes
No
Not Sure
Page 3
10
Paper-Based Surveys
Benefits of paper-based surveys High response
rate in controlled settings(e.g. classroom) Low
cost set-up no special technology
required However . . . Lower response rate in
uncontrolled settings(e.g. mail-outs) Considerabl
e cost and effort to collect the data Longer time
before reports and analysis is available
11
College KPI in Ontario
Annual student surveys used by the Ontario
Ministry of Training Colleges and Universities
(MTCU) for Key Performance Indicators
(KPI). Standardized scannable survey tool with 5
college specific questions Applied in all 24
Ontario colleges Surveying of over 150,000
students, in both English and French Used to
produce key performance indicators and to track
improvement progress Process and results audited
12
College KPI in Ontario
Surveys administered in classrooms in a
controlled environment with standardized
procedures Results available by province,
college, school, campus, program and common
program averages across the province Results
available within 2-4 weeks of the end of the data
collection period Colleges perform more in-depth
analysis following the release of the data
13
Continuing Education in Ontario
Bi-annual surveys (since 1996) of over 200,000
Continuing Education students for Ontario Heads
of Continuing Education Scannable surveys
administered in classrooms Shared development,
process and costs Results available for province,
large colleges, medium colleges, small colleges,
Metro colleges, colleges, campuses, schools and
courses Development of performance benchmarks
14
Telephone Surveys
Appropriate for public and widely-distributed
target populations as well as less willing
participants. Generally, higher response
rates Specialized technology Computer-Assisted
Telephone Interview (CATI) technology. More
costly (approximate per completed survey) 9 5
min. survey public15 10 min. survey
public8 5 min. survey graduate13 10 min
survey graduate
15
Telephone Surveys
Benefits of phone surveys Timely and efficient
data collection Separate data entry not required
Can probe and clarify responses Use of simple or
complex logic Ease of tracking progress
16
Alberta Learning
Phone interviews are used by Albertas education
ministry (Alberta Learning) Total of 6,500
completed telephone interviews with - high
school students, - parents, - teachers, -
school board members, - adult learners, and -
members of the general public Survey items on
each survey are designed to assess satisfaction
with the learning system in Alberta Administered
annually
17
On-line Surveys
Respondent receives request to participate in an
on-line survey and is provided with a URL.
Using their web browser they locate the survey
and complete the survey on-line.
18
On-line Surveys

• When using Web URL, there are 2 methods of login
• Anonymous login
• Pros - Complete anonymity and
  confidentiality
• - Ease of use no usernames or passwords
• Cons - Cannot link additional data to
  respondents
• - No tracking for completions
• (2) Username and password login
• Pros - Link additional data to respondents
• - Tracking completions
• - E-mail reminders
• Cons - Respondents may feel they
• are being identified

19
On-line Surveys
Benefits of on-line surveys Timely and efficient
data collection Low cost data collection Separate
data entry not required Use of simple or complex
logic Ease of tracking progress However . .
. Lower response rate unless there is a
committed group of respondents (e.g., employees,
students in a class)
20
Sheridan Course/Teacher Feedback
On-line survey of approximately 30,000 Sheridan
College student surveys. 15-item survey of
courses and teachers. Multi-method data
collection On-line surveys for students with
computer access Paper surveys for students with
no access On-line survey results seamlessly
integrated with other data from scanned paper
surveys producing a single dataset of
results. Results are provided by email in a
PDFto each teacher
21
Sheridan Course/Teacher Feedback
Example use of student ID and system username as
2 pieces of information to validate respondents.
22
Sheridan Course/Teacher Feedback
23
Reports and Analysis
Reports and analysis reflect project objectives.
- What will the data be used for? - Who will be
using it? Numerical reports for survey data
provide detailed results in easy-to-read formats.
Various Report Types Tabular Reports with
Benchmarks Tabular Reports with Trends Graphical
Summaries Comparative (Crosstab or Drill-Down)
Reports
24
Reports and Analysis
For example, Key Performance Indicator
Reports 1. Detailed Comparative Report 2. Simple
Graphical Report 3. Numerical Snapshot Report 4.
PowerPoint Summary Report 5. Written Executive
Summaries 6. High-to-Low Program and Question
Reports 7. Statistical Accuracy Tables 8. Excel
Table Reports 9. Custom reports to meet
institution needs.
25
Reports and Analysis
Sample Comparative Report
26
Reports and Analysis
Sample Numerical Summary Report
27
Reports and Analysis
Sample Graphical Report
28
Reports and Analysis
Sample Graphical Report
29
Reports and Analysis
Sample Quintile Report
30
Reports and Analysis
Written executive summaries to interpret the data
and answer specific questions
Analysis To Identify Strengths/Areas for
Improvement Resolutions Trends Correlations Highli
ghts and more...
Report Types Summary Reports Graphical
Presentations Statistical Analysis Written
Interpretation Executive Summaries
31
Working Together
Benefits of educational organizations working
together Sharing of - data - knowledge -
expertise - costs Use outside firm to do the
specialized components of the work, use them to
coordinate a standard process Highly
motivated, experienced, impartial, cost
effective
32
Contact Information
Compustat Consultants Inc. 67 First
Street Orangeville, Ontario L9W 2E6 Toll Free
1-888-246-8068 Fax [phone] Web-
site compustatconsultants.com Ted Hodge,
President thodge_at_compustatconsultants.com Rose
DAlimonte Customer Service/Sales
Manager rose_at_compustatconsultants.com