OATS: Georgia Techs Online Assessment Tracking System

1
OATS Georgia Techs Online Assessment Tracking
System

• SAIR 2005 Conference
• Charleston, SC
• October 24, 2005
• Joseph Hoey
• Jon Gordon
• Office of Assessment, Georgia Tech

2
OATS Presentation Outline

• Development of OATS to document Institutional
  Effectiveness
• Description of OATS features
• OATS structure and elements
• Issues encountered, solutions generated
• Path forward
• Questions and discussion

3
Documenting Assessment With a Web-Based System

• Web-based documentation of assessment for
  demonstrating progress towards Institutional
  Effectiveness is an option, not a requirement
  we have found it very useful at Georgia Tech.
• Georgia Techs approach to documenting systematic
  assessment processes is our Online Assessment
  Tracking System (OATS).
• Similar systems have been developed by other
  institutions and commercial vendors.

4
OATS Background

• Annual Assessment Update concept was generated by
  GT unit coordinators in 1998 to document Georgia
  Techs responsiveness to SACS recommendations re
  assessment of student learning.
• Requests received to move to online environment
• Online process provides structure, formalizes
  best practices in assessment, serves as a
  cross-campus communication vehicle, and thus
  facilitates demonstration of compliance.
• Having an online documentation system proved to
  be a key feature in Georgia Techs compliance
  review with the Principles of Accreditation in
  2004.

5
Deciding What to Document and At What Level

• We document and track assessment at the degree
  program level
• For each degree program, we document
• Faculty expectations for student learning
• How these expectations are defined in practice
• How these expectations are assessed
• What results are gained (summary level)
• Based on results gained, what actions are being
  taken

6
Old Paper-Based Annual Assessment Update Structure

• Three open-ended sections
• What did you look at?
• What did you find?
• What did you do about it?
• Submission as MS Word or .pdf documents

7
Web-Based Annual Assessment Update Structure
OATS
8
Feature Comparison

• Paper-Based Method
• Many different formats
• Hard copy only
• Difficult to track progress over time
• Flexibility (but no consistency across Institute)
• Difficult to provide feedback internally and to
  facilitate institutional sharing of good practices

• Online Method
• Consistent format
• Database storage
• Ability to track progress over time
• Flexibility maintained
• Process facilitates accreditation e-review
• Easier to provide feedback facilitates
  institutional sharing

9
OATS Integrates Assessment and Annual Reporting

• Annual Assessment Update now done online via
  OATS.
• Becomes a longitudinal record of assessment and
  changes to each instructional program.
• Ideally, can be used to satisfy multiple internal
  and external information and accreditation needs.

10
OATS Application Features

• Includes user id/password login and Georgia Tech
  authentication
• Web accessible from any location software
  plug-in required for editing capability
• Defined format structureObjectives/Outcomes,
  Methods, Results, and Actions/Impact
• Allows posting of formatted text (tables, charts,
  etc.)
• Default annual rollover feature (except for
  assessment results and actions taken) saves time
• Assessment Updates due November 1 each year

11
OATS Application Features

• Archiving of all past submissions by degree
  program,
• Print-preview feature that produces a laddered
  report of objectives, methods, results, and
  actions taken,
• Notes feature that permits the storage of
  submission notes and commentary that wont be
  seen officially could be notes to colleagues
• Feedback feature is built in that permits units
  to obtain formative feedback on their update
  prior to final submission.
• College and Institute-level views, whereby deans
  are able at a click to view the online assessment
  update submissions of all units within their
  colleges.

12
Description of OATS Elements

• Main Menu
• Program Menu
• Unit Summary Section
• Objectives/Outcomes
• Methods
• Results
• Actions
• Print Preview

13
OATS Main Menu

• Includes links to instructions and changes in
  OATS
• Link to administrative page for those with
  administrative access
• Selections for
• editing current updates,
• viewing current updates (print preview format
  only), and
• viewing updates from previous years (print
  preview format for 2003 .pdf for earlier years)

14
Main Menu Current Year and History
15
OATS Program Menu

• Searchable by College
• Includes enumeration of objectives/outcomes,
  methods, results, actions/impacts and date last
  modified
• Indicates status/progress towards completion
• Options to combine programs, view notes, edit
  individual update, and delete update (Are you
  sure? screen appears)
• Notes feature can be used as information storage
  area by user (heres why we did this last
  year) or for feedback from Office of Assessment

16
College Level Ivan Allen College
17
Unit Summary Section

• Summary section of each OATS update includes
  portions for program purpose, responsibility for
  implementation, and operational (programmatic)
  objectives
• Programs use this section to describe the program
  purpose, set the context, describe operational
  objectives for the program, and to provide a
  brief summary of overall direction.

18
(No Transcript)
19
Objectives/Outcomes - 1

• We ask programs to limit OATS updates to 3-8 most
  important student learning objectives/outcomes
  faculty expect from program graduates.
• Engineering undergraduate programs prioritize
  and develop a rotational timeline for assessing
  ABET EC 2000 program outcomes dont try to
  assess all of them each year.

20
Objectives/Outcomes - 2

• For each objective/outcome, we ask programs to
  operationally define the activities and
  competencies to be assessed. What will students
  actually demonstrate? Describe what is to be
  achieved.
• State in terms of expected behaviors.
• Measurable.
• Aggregate (group) level not individual.

21
School Level History, Technology Society
22
Assessment Methods - 1

• We ask programs to specify the method by which
  student competencies will be assessed
  (presentations, embedded exam questions,
  projects, etc.)
• Optional Indicate location in curriculum.
• Optional Specify performance standards to be
  achieved.

23
Assessment Methods - 2

• We remind programs that the same method may be
  used to cover more than one learning
  objective/outcome.
• e.g., rubrics or checklists developed to assess
  senior design projects might be used to assess
  technical skills, teamwork, and communication
  skills.
• Consider triangulation (multiple methods) to
  increase reliability of measurement.
• Consider the reliability, validity, and quality
  of instrumentation. Will we be willing to trust
  the results?

24
School Level History, Technology Society
25
Assessment Results

• Succinctly summarize assessment results found
• e.g., 95 achieved passing scores on relevant
  portion of Fundamentals of Engineering Exam.
• Provide brief analysis and interpretation of
  results.

26
School Level History, Technology Society
27
Action Use of Assessment Results

• Include a discussion of actions taken or pending
  in context of the instructional program based on
  consideration of results obtained.
• Refer to instrumentation used if URL is
  available.
• Give timetable for completion of actions.
• e.g., The school has redesigned our
  undergraduate structures sequence and hired an
  additional faculty to teach the additional course
  beginning next semester.

28
(No Transcript)
29
Print Preview

• OATS includes an option to create a formatted
  report for each assessment update
• Summary section of update appears as distinct
  from learning outcomes section
• Update report is laddered such that methods,
  results, and actions are indented
• Print preview may be saved as an html document
  and email to others for review

30
Degree Program Level BS in HTS
31
(No Transcript)
32
Issues Encountered

• Who is really the customer?
• Multiple operating systems and levels of security
• Issues with plug-in software, supported
  applications
• Training needs
• Operational roles vary across units
• Varying levels of administrative enthusiasm and
  involvement
• Tree vs. multidimensional structure

33
Other Issues Encountered

• Multiple evaluations/multiple cultures problem
• Many external reviews and accountability demands,
  all with unique features SACS, Board of Regents
  program review specialized accreditation
  (especially ABET)
• Culture of disciplines demands custom solutions
  (especially ABET)

34
Path Forward - 1

• Process to move towards integrated, customizable,
  culturally acceptable solution
• Interviews with key players and users to assess
  needs during Summer 2005
• Task force assigned to develop conceptual
  recommendations by November 2005
• Technical working group will write specifications
  by March 2006
• Programming and testing completed by August 2006
• Next OATS version operational for November 2006
  OATS

35
Path Forward - 2

• Desirable features identified so far
• Customized shell views selected at login for
  each relevant specialized accreditation body
  (ABET, CAC, AACSB, NAAB, NASAD, etc.)
• Ability to link to representative courses in
  curricula
• Ability to view OATS entries by
  objectives/outcomes, by assessment methods used,
  and by courses in major
• Ability to import IR data into unit summary
• Ability to import assessment data from GT online
  assessment data query tool directly into updates
• Ability to link across disciplines for project or
  program evaluations
• Ability to do all of the above by yesterday with
  no faculty effort

36
Questions?

• Contacts
• Joseph Hoey, joseph.hoey_at_oars.gatech.edu or
  telephone [phone]
• Jon Gordon, jon.gordon_at_oars.gatech.edu or
  telephone [phone]