Assessments of Teachers and Technology: Exploring the Possibilities

1
Assessments of Teachers and Technology
Exploring the Possibilities

• Jim Julius
• SDSU-USD Joint Doctoral Program in Education
• Society for Information Technology in Teacher
  Education Conference 2004

2
To follow up

• Conference proceedings pp. 2303-2308
• Download this ppt http//www.sandiego.edu/jjuliu
  s/tech_assess.ppt
• Contact mejjulius_at_sandiego.edu

3
Overview

• Professional Development
• Needs Assessment
• Validation
• Assessment Tools

4
Professional Development

• Inservice technology training for teachers is the
  "single most critical factor associated with
  improving student achievement when using
  technology in instruction" (Molenda Sullivan,
  p.17, citing a 2001 CEO Forum study)

5
Professional Development

• Researchers on technology-oriented staff
  development cite importance of identifying
  teacher needs (Barkley Branco, Boyd, Bray,
  Moersch 2002)

6
Needs Assessment

• Assessment tools also useful for accountability
  and evaluation
• Common assessment tool survey Advantages
• Easy
• Inexpensive
• Flexible administration (Jordan Hoefer)

7
Needs Assessment

• Potential errors in the use and interpretation of
  surveys
• biased design which may predetermine results
• respondents giving more desirable answers
• careless responses
• inconsistent responses to similar questions
• misinterpretation of directions and/or questions
• poor population sample
• inconsistent administration or scoring
• answers that are acquiescent, extreme (too
  high/too low), or evasive (middle option/no
  opinion)

8
Needs Assessment

• Embracing an empirically tested set of measures
  to ascertain teacher growth with technology use
  in the classroom will give policy makers, school
  administrators, and classroom practitioners the
  most consistent data to make informed decisions
  as to the real needs for improving the technology
  infrastructure beyond hardware and software
  issues. Furthermore, the findings stemming from
  these research studies can help shape the type of
  professional development interventions needed to
  maximize the level of technology implementation
  in the classroom. (Moersch 2001, p. 27)

9
Needs Assessment

• Using a standardized measure can take advantage
  of "the work that has gone into identifying and
  conceptualizing potentially important needs and
  of using a measure for which reliability and
  validity have been established" (Jordan Hoefer,
  p. 167)

10
Validation

• Reliability of an instrument Does it measure
  consistently?
• Reliability is important but not enough a
  reliable instrument may be invalid due to bias

11
Validation

• Validity of an instrument Does it measure what
  it is supposed to measure?
• Establishing validity a well-documented
  judgmental and empirical process (see Gable,
  Hittleman Simon, Jordan Hoefer, Smith
  Glass)

12
Assessments from the Literature

• Categories of technology assessments used with
  teachers in published studies which addressed
  validation in some way
• Self-efficacy belief theories
• Concerns-Based Adoption Model
• Basic technical skills
• Comprehensive assessments

13
Self-Efficacy Beliefs Microcomputer Utilization
in Teaching Efficacy Beliefs Instrument (MUTEBI)
14
Self-Efficacy BeliefsBeliefs About Teaching
with Technology (BATT)
15
CBAM Stages of Concern Questionnaire (SoCQ)
16
Basic Tech Skills Basic Technology Competencies
for Educators Inventory (BTCEI)
17
Basic Tech Skills Mankato survey
18
Basic Tech Skills CTAP2/iAssessment
19
Comprehensive Assessments Levels of Technology
Implementation Questionnaire (LoTiQ)
20
Comprehensive Assessments Taking a Good Look at
Instructional Technology (TAGLIT )
4-point scale from "I don't know how to do this"
to "I can teach others how to do this" How far
along are you in learning how to use a
spreadsheet to create graphs?
4-point scale from "I do not use it in teaching
and learning" to "I naturally include it in
teaching and learning and use it in powerful
ways" How far along are you in enhancing teaching
and learning using video production?
4-point scale from "No" to "Yes, Very Much As a
result of your use of technology in teaching and
learning, are you more inclined to serve as
coach, not lecturer or whole-group discussion
leader?
21
Comprehensive Assessments enGauge

• 4-point scale from "No experience" to "Expert"
• How much experience OR how much knowledge do you
  have with assessing student products created with
  technology such as multimedia projects,
  student-created Web pages, etc.
• Rate your skill level for Searching for
  resources on the World Wide Web

• How well is classroom technology use aligned with
  your districts standards, generally, throughout
  your school? (Not at all, Slightly, Fairly well,
  Very well, Don't know/No technology)

Interpret results of the educator survey within
the context of a specific school.
Designed to give a systemic view of a school or
district's use of technology
22
Conclusion

• Possibilities for tech-oriented teacher
  education
• None
• Done without needs assessment
• Done from homegrown needs assessment
• Done from existing needs assessment
• Done from existing, previously validated needs
  assessment
• Same as previous, with validity asserted for the
  context/use of the assessment

23
Conclusion

• Using existing validated assessments
• Avoid reinventing the wheel
• Precisely target tech-oriented teacher education
• Allow accumulation of data for research

24
Web References for Assessments

• BATT insight.southcentralrtec.org/ilib/batt.html
  
• BTCEI insight.southcentralrtec.org/ilib/btcei.htm
  l
• Mankatowww.isd77.k12.mn.us/resources/surveydatab
  ase.html
• CTAP2/iAssessment ctap2.iassessment.org
• LoTiQ lotilounge.com/LoTiQuestionnaire.html
• TAGLIT www.taglit.org/taglit/login.aspx
• enGauge www.ncrel.org/engauge
• SoCQ www.ao.uiuc.edu/ijet/v3n1/rakes/

25
Web References for Other Assessments

• Kentucky Implementation of Technology Survey -
  teacher oriented
• http//profiler.hprtec.org/smf/survey.jsp?survey_i
  d174
• Technology Surveys for School Stakeholders -
  comprehensive, community-oriented set of surveys
  
• http//www.serve.org/seir-tec/surveys/
• ISTEs Technology Support Index for school
  districts to assess technology support
• http//tsi.iste.org
• CEO Forums StaR Chart Benchmarking for schools
  to assess progress in technology integration
• http//www.iste.org/starchart/index.cfm
• Websites with Links to Additional Teacher
  Assessment Resources
• http//insight.southcentralrtec.org/ilib/masterlis
  t.html
• http//www.vismt.org/programs/infotech/planning/or
  esources/assessmt_resources.html
• http//www.kn.pacbell.com/wired/fil/pages/listtech
  nolosu.htmlcat2
• http//www.techtamers.com/free_resources/education
  al_technology/assessteach.htm

26
References

• Barkley, S., Bianco, T. (2001). Online and
  Onsite Training When To Mix, When To Match.
  Educational Technology, 41(4), 60-62.
• Boyd, E. G. (1997). "Training-on-Demand" A Model
  for Technology Staff Development. Educational
  Technology, 37(4), 46-49.
• Bray, B. (1999). Eight Steps to Success
  Technology Staff Development That Works. Learning
  and Leading with Technology, 27(3), 14-20.
• Cheung, D., Hattie, J., Ng, D. (2001).
  Reexamining the Stages of concern questionnaire
  a test of alternative models. Journal of
  Educational Research, 94(4), 226-236.
• Enochs, L. G., Riggs, Ellis. (1993). The
  Development and Partial Validation of
  Microcomputer Utilization in Teaching Efficacy
  Beliefs Instrument in a Science Setting. School
  Science and Mathematics, 93(5), 257-263.
• Flowers, C. P., Algozzine, R. F. (2000).
  Development and Validation of Scores on the Basic
  Technology Competencies for Educators Inventory.
  Educational and Psychological Measurement, 60(3),
  411-418.
• Gable, R. K. (1986). Instrument development in
  the affective domain. Boston Kluwer-Nijhoff.

27
References

• Hall, G., George, A.A., and Rutherford, W.L.
  (1978). Stages of concern about the innovation
  The concept, verification, and implications.
  Austin, TX Southwest Educational Development
  Laboratory.
• Hittleman, D. R., Simon, A. J. (1992).
  Interpreting educational research an
  introduction for consumers of research. New York
  Merrill.
• Jordan, C., Hoefer, R. A. (2001). Reliability
  and validity in quantitative measurement. In B.
  A. Thyer (Ed.), The handbook of social work
  research methods (pp. 546). Thousand Oaks,
  Calif. Sage Publications.
• Kaufman, R. A., English, F. W. (1979). Needs
  assessment concept and application. Englewood
  Cliffs, N.J. Educational Technology
  Publications.
• Lumpe, A. T., Chambers, E. (2001). Assessing
  Teachers' Context Beliefs about Technology Use.
  Journal of Research on Technology in Education,
  34(1), 93-107.
• Moersch, C. (1995). Levels of Technology
  Implementation (LoTi) A Framework for Measuring
  Classroom Technology Use. Learning and Leading
  with Technology, 23(3), 40-42.
• Moersch, C. (1999). Assessing Current Technology
  Use in the Classroom A Key to Efficient Staff
  Development and Technology Planning. Learning and
  Leading with Technology, 26(8), 40-43.

28
References

• Moersch, C. (2001). Next Steps Using LoTi as a
  Research Tool. Learning and Leading with
  Technology, 29(3), 22-24,26-27.
• Moersch, C. (2002). Measurers of Success Six
  Instruments to Assess Teachers' Use of
  Technology. Learning and Leading with Technology,
  30(3), 10-13.
• Molenda, M., Sullivan, M. (2001). Technology in
  Education The Boom Is Behind Us. Technos, 10(3),
  14-19.
• Smith, M. L., Glass, G. V. (1987). Research and
  evaluation in education and the social sciences.
  Englewood Cliffs, N.J. Prentice-Hall.

29
To follow up

• Conference proceedings pp. 2303-2308
• Download this ppt http//www.sandiego.edu/jjuliu
  s/tech_assess.ppt
• Contact mejjulius_at_sandiego.edu