Assessment Matrix Workshop

1
Assessment Matrix Workshop

• Katherine Perez
• Jacqueline Peña
• Office of Planning and Institutional
  Effectiveness
• January 2009

2
Overview of workshop

• SLO matrix
• SLOs and Corresponding Assessment Methods
• Workshop Revising SLOs and Assessment Methods
• Reporting Results and Use of Results
• Workshop Writing the Results and Use of Results
• PO matrix
• POs and Corresponding Assessment Methods
• Reporting Results and Use of Results
• Workshop Revising POs and Assessment Methods
  Writing Results and Use of Results
• Additional individualized assistance

3
Why Focus on Assessment?

• SACS Accreditation
• Key SACS Deadlines
• September 10, 2009Report due
• March 8-12, 2010Onsite Review
• Key IE Deadlines
• February 1, 2009Fall 2008 SLO matrices due
• June 1, 2009Spring 2009 SLO and PO matrices due
• Continuous improvement
• Institutional Effectiveness
• Sound, research-based and assessment-based
  decisions

4
Blank SLO Matrix 08-09
Link to the Mission
5
Writing the slo

• Criteria
• Can be observed and measured
• Relates to student learning towards the end of
  the program
• Reflects an important concept
• Formula Who Action Verb What
• Students will identify and apply the key
  components of a learning system.
• Students will identify and apply the five key
  components of a learning system.

6
Technology SLO

• Principle 3.4.12
• The institutions use of technology enhances
  student learning and is appropriate for meeting
  the objectives of its programs. Students have
  access to and training in the use of technology.
  (Technology use)
• Required for all undergraduate programs
• Optional for 2007-2008 SLO matrices
• Required for 2008-2009 SLO matrices

7
Sample Technology outcomes

• General (not appropriate for SACS)
• Students will use information technology to
  gather and disseminate information.
• More Specific
• Graduates of the program will demonstrate the
  ability to use appropriate computer technology,
  software, and the Internet to complete their
  capstone research papers.
• Much more Specific (appropriate for SACS)
• Students will be able to effectively demonstrate
  information technology skills by locating and
  retrieving information on economic topics and
  issues, published research in Economics and
  related fields, and/or by finding information
  about the generation, construction, and meaning
  of economic data for their final research
  projects.
• Students will write and present a capstone
  project that requires the use of Word, Excel,
  PowerPoint, and information technology.

8
Action Verbs Who Action Verb What
Note This table reflects the original Blooms
Taxonomy Table. The modified version places
evaluation as the fifth column and synthesis
(creating) as the sixth column.
9
Assessing the SLO

• Artifact
• Collection of the artifact
• Where, when, and how
• Census versus sampling
• Criteria
• Minimum standards on a rubric/scale or the
  percentage of correct items
• Evaluation of the artifact
• Faculty panel or external evaluators
  (reliability)
• Rubrics, embedded questions, etc.

10
Direct and Indirect Measures
11
Sample SLO Assessment
12
Reporting the results

• Summary of Results
• Direct measures
• Test items
• Performance as determined by rubrics
• Indirect measures
• Surveys and questionnaires
• Interview and focus group data
• Format
• Narrative
• Tables or charts
• Analysis of results
• If pertinent, explain results in a narrative form
  by interpreting results or using qualitative
  analysis of the data.
• Every student learning outcome must have at
  least
• One set of results
• One student learning improvement strategy (use of
  results)

13
Results Activity

• All the students passed the assessment.
• 75 of the students met the criteria for success.
• Our students passed the dissertation defense on
  the first attempt.
• 75 of the students (n15) achieved a 3 or better
  on all the rubric categories for the capstone
  course research paper.

14
Masters Thesis Results Table
Criteria Students will achieve a 3 or better on
a 4-point rubric on all five sections of the
masters level thesis.
15
Sample SLO Assessment Results
16
Explaining your use of results
17
Use of Results Activity

• Target met. Will continue to monitor.
• The faculty will meet and revise the three
  introductory courses.
• A larger sample will be obtained.
• We will revise the rubric and have a calibration
  session with the faculty prior to evaluating the
  student papers each semester.
• A capstone course will be created that emphasizes
  research and thesis writing methods.
• A student resource center will be created with
  computer equipment, writing tutors, and
  statisticians who can assist our students with
  course- and thesis-related writing and research.

18
Sample Completed SLO matrix
19
Sample Completed SLO matrix
20
The summary page

• Overview and Comparisons
• Give an overview or summary of all the outcomes
  together
• Discuss trends that you have seen over the years
• Explanations
• Provide qualitative explanations for poor results
  or exceptionally high results
• Notes and documentation affecting results
• Response rate (e.g. Only 50 of the students
  completed the project.)
• Inter-rater reliability (e.g. 2 faculty members
  reviewed the artifacts and the inter-rater
  reliability was only 60.)
• Assessment Commentary
• Revise or create instrument (e.g. artifact,
  rubric)
• Modify assessment methods (e.g. data collection,
  sampling, criteria, evaluation process)

21
Activity Results and Use of Results

• Write or revise your Results and Use of Results.
• Results
• Clearly state the results of each PO assessment
  (table or narrative).
• Use of Results
• Explicitly state how you will use the results to
  improve the program during the following
  assessment cycle/year.
• Summary Page
• Overview of results and use of results (looking
  at the big picture)
• Trends over the last year or several years
• Comparison to national or state trends or data
• Explanations for poor or unexpected results
• Notes or documentation that could explain the
  results
• Assessment improvement plans

22

• END OF THE SLO PORTION
• What Is Next?
• 10-minute overview of Program Outcomes
• PO writing assistance
• Continued SLO writing assistance

23
Writing the PO

• Program-level outcomes
• Focus on student success (not student learning)
• Formula Who Action Verb What
• Examples
• Graduates seeking employment in the field will
  find such employment within 6 months of
  graduation.
• Graduates will be satisfied with advising and
  mentoring services.
• Faculty will publish at least one journal article
  or book chapter every two years.

24
Sample Completed po matrix
25
Activity Program outcomes

• Take 15 minutes to review or write at least one
  program outcome with corresponding assessment
  methods, results, and use of results.
• PO Formula
• Who Action Verb What
• Assessment Methods Formula
• State the instrument/method/tool (e.g. survey,
  national exam results).
• Explain the data collection steps (e.g. when,
  where, how).
• State the success criteria (minimum standards for
  success).
• Results
• Clearly state the results of each PO assessment
  (table or narrative).
• Use of Results
• Explicitly state how you will use the results to
  improve the program during the following
  assessment cycle/year.

26
The Ie Team

• Marta Perez
• Director
• perezma_at_fiu.edu
• 305-348-2733
• Maria Corrales
• Coordinator
• corrales_at_fiu.edu
• 305-348-0459
• Katherine Perez
• Coordinator
• katherine.perez_at_fiu.edu
• 305-348-1418
• Jacqueline Peña
• Coordinator
• jpena_at_fiu.edu.
• 305-348-1367

• Mayelin Felipe
• Computer Specialist
• mfelip01_at_fiu.edu
• 305-348-0115
• Karla Felipe
• Computer Specialist
• kgarcia_at_fiu.edu
• 305-348-0115
• Amanda Berhaupt
• Graduate Assistant
• aberh001_at_fiu.edu
• 305-348-2731
• Randhir Kaur
• Graduate Assistant
• rkaur001_at_fiu.edu
• 305-348-2731