Effective Results and Actions for Course Assessment Plans

1
Effective Results and Actions for Course
Assessment Plans

• Kim Anderson
• ASLO Subcommittee Chair
• Eva Bagg
• Associate Dean, Institutional Effectiveness
• John Hetts
• Research Consultant
• Fall 2010

2
Why an Outcomes Assessment Process?

• Follows a cycle of planning, implementation,
  analysis and action to improve student learning.
• Facilitates periodic assessment
• Guides planning management guideline
• Improves student learning, retention, program
  completion
• Enables faculty to play central role
• Encourages collaboration within and among
  departments
• Supports practice of professional educators

3
Course Outcomes Assessment Cycle

• Course Review Cycle - at least every 6 years
• Starting up may require shortened cycle
• The Three Ms
• Meaningful
• Measureable
• Manageable

4
Outcomes Assessment Cycle Management

• Internal Management Structure
• Who (SLO Officer, lead faculty)
• When analyze and report (noted in assessment
  plan)
• How (implement by course, not SLO)
• Examples
• Gantt Chart/Excel spreadsheet
• Web based tool Open Source
• TracDat

5
Outcomes Assessment Loop
6
Components of theOutcomes Assessment Cycle

1. State desired outcomes clearly
2. Identify assessment tools
3. Establish criteria and level of expectation for
   success
4. Gather and analyze results
5. Take action to improve and re-evaluate

7
Sample Layout
Intended Outcome Assessment Task Criteria /Expected Level of Achievement Results of Assessment Actions Taken
Write intended outcome in this column. Student Learning Outcomes (SLOs) are identified in the course outline of record. What Describe assessment tool or task How Explain implementation of task Who Identify persons responsible for conducting assessment When Identify timeframe of assessment task commencement and for how long (semesters/years) Expected Achievement Establish minimum expected benchmark for achieving outcome Success Level Determine acceptable success standard based on identified assessment task Students Included Describe students to be included (e.g. successful completers, all queried, random sample) Begin collection of evidence for this course at this point. Results Describe sample and summarize results on outcome measure Key Findings Highlight notable results, good and bad Conclusions Describe implications for curriculum development and assessment improvement Action Explain (in past tense) what you have done in response to results Re-evaluation Date Identify timeframe for reevaluation
8
Outcomes Assessment Plan Process Format
Intended Outcome Assessment Task Criteria/Expected Level of Achievement Results of Assessment Actions Taken
SLO What How Who When Expected Achievement Success Level Students Included Begin collection of evidence for this course at this point. Results Key Findings Conclusions Action Re-evaluation Date
Ongoing Assessment
Assessment Plan
9
Assessment Results and ActionsOngoing Part of
Outcomes Assessment Cycle

• Step 4 Results, Findings, and Conclusions
• Meet and discuss assessment evidence collected
• Analyze evidence making sure to
• highlight key findings from assessment evidence
• draw conclusions about needs/issues revealed
• formulate actions to improve student learning
• identify decisions that people can live
  with/implement
• Step 5
• Take action based on conclusions
• (Note this step requires immediate action)
• Record/document actions already taken

10
Assessment Results and ActionThings to Consider

• Time
• balancing competing responsibilities and
  priorities
• Management
• aligning with review cycle and resources
  availableneeds planning
• Discussion
• collaborating with stakeholders in various
  venues
• staff development, department meetings, small
  groups

11
Assessment Results and ActionThings to Consider
(cont.)

• Key questions to consider when examining results
  of assessment
• Evaluation of the process
• How consistent is our evaluation?
• How robust is our final assessment information?
• Application of the process
• How can we use this information to improve
  student learning?
• How can we use this process to support and
  integrate with the colleges planning process?
• How effectively do our results and conclusions
  provide guidance to our stakeholders?

12
Step 4. Gather results of assessment

• Gather and organize
• Process protocol and whos involved internal
  deadlines regular cycle in conjunction with
  course review cycle dates
• Discussions at various meetings need time
• Do not become a statistician
• Aggregate (combined)
• Within the course/department/program
• Office of Institutional Effectiveness
• Protect identities and confidentiality
• Disaggregate
• Determine more discrete information
• Address components more effectively

13
Step 4 Simple strategies for analysis

• Right now (where available/if applicable)
• Compare
• results to expected level of achievement
• results to previous results
• results to pretest
• results across demographic categories
• outcome to outcome
• outcomes across sections
• In the future
• Compare year to year
• Evaluative effectiveness of actions taken

14
Step 4 Describing your results

• Describe your sample
• Your sample is just the people who were assessed
• Indicate how many people were assessed
• N total number of students
• Indicate how they were selected from population
• Population entire group of people you are using
  the sample to understand/draw inferences about
• Clearly describe procedure used to identify and
  select participants
• Key issues
• Is sample (N) large enough?
• Challenging with small populations like courses
• Is sample representative of population?

15
Sampling issues

• So how big a sample should you aim for?
• For sample needed for estimate of meeting
  standard see table
• for small populations, you need to assess almost
  everyone
• In the absence of comprehensive assessment,
  sampling needs to be rigorous
• What about representativeness?
• Selection biases
• You
• Students
• Identify population and select randomly
• Must try to get everyone selected

Interval size Interval size Interval size
Pop. Size 15 10 5
20 14 17 19
30 18 23 28
50 23 33 44
100 30 49 80
150 33 59 108
200 35 65 132
500 39 78 217
US Pop 43 96 384
Calculations made using http//www.surveysystem.c
om/sscalc.htm
16
Step 4 Describing your results (cont)

• Key distinction is outcome categorical
  information or more quantitative information
• Categorical example
• Common educational rubric for outcome assessment
• 4 - Advanced (98 and up) exceed standard
  substantially on all criteria
• 3 - Proficient (70-97) met standard on all
  criteria
• 2 - Basic (50-70) met standard or close on all
  criteria
• 1 - Below Basic (below 50) did not meet
  standard on multiple criteria
• Categories are different widths intervals are
  not equivalent, i.e., a point difference
  doesn't always mean the same thing
• Provide frequency
• Simple count of number of students in category
• Supplement with percentage
• Ratio of frequency to total number ( 100)
• 5 (frequency) out of 100 (N) would yield 5

17
Step 4 Describing your results (cont)

• Describe your results quantitative info
• E.g., number of items answered correctly
• Describe central tendency
• executive summary of your data
• two key types
• Mean add all scores then divide by N
• Note Be careful about summing (or averaging)
  across outcomes
• Imagine two standardized assessments of outcomes
  in Astrology
• Number of astrological signs correctly identified
• Estimate quality of astrological reading on a
  1-10 scale
• Median score at which ½ sample is below, ½ above
• Use with mean when extreme scores make mean
  ambiguous
• E.g., 10 item test with 10 scores 0,
  8,8,8,9,9,9,10,10,10
• Mean 8.1, Median 9_

18
Step 4 - Analyze Key Findings

• Analysis and evaluation
• Through discussion with vested individuals
• Evaluate the validity of the assessment
• Compare with goals, and, eventually, previous
  results
• Discuss any discipline-specific issues
• Engage data with critical eye to interpret and
  apply appropriately
• Highlight strong indicators
• Significant patterns or trends in the data
• Met, not met, or exceeded stated SLOs expected
  level of achievement
• Focus on accomplishments and examine difficulties
  or needed improvements

19
Step 4 Develop supportable conclusions

• Focus on actionable evidence
• Keep information manageable
• Identify what
• worked well
• could be improved
• Relate conclusions back to course SLO
• Reflect facultys discussion
• Make appropriate decisions for the data level
• Note unexpected results, flaws in study, and
  faculty observations based on the evidence

20
Step 5. Document Actions Taken

• Based on Step 4, developed and implemented steps
  to improve student learning and its assessment
• Include what has been done to improve student
  learning
• What still thinking about goes in conclusion in
  step 4
• Curriculum (content emphasis, sequencing,
  formative assignments, more time with a topic of
  concern)
• Pedagogy (varied delivery methods, technology,
  learner styles, tutors, Supplemental Instruction,
  prompt feedback)
• Assessment (SLO, assessment tool,
  internal/external factors, criteria, sampling)

21
Step 5. Document Actions Taken (cont)

• Key issues to consider when documenting actions
  taken
• Clarify extent of change incidental modification
  or significant change
• Specify definite actions taken and identify
  faculty responsible
• Outline meaningful time frame for implementation
  and re-evaluation
• Describe any follow-up process
• Teaching Learning Center

22
ESL Sample
Intended Outcome Assessment Task Criteria /Expected Level of Achievement Results of Assessment Actions Taken
Write a focused thesis statement and maintain unity throughout an essay. What Write a compare/contrast essay based on short readings of fiction and non-fiction. How The Test of Written English (TWE) scoring guide (rubric) items 1 and 4 as criteria. Who Instructors of ESL 33 and 33X. When Collect and report in Spring 2010 Expected Achievement 70 of students should achieve passing score. Success Level passing score is 4 or higher (on 6 pt scale) Students Included random sample of students enrolled in all sections of ESL 33 and 33X. 5/31/10 Results 55 (55 out of 100 total) of the randomly sample essays obtained a score of 4 or higher with 100 participation from all ESL 33 ESL 33X 10 received a 6 15 received a 5 30 received a 4 25 received a 3 25 received a 2 5 received a 1 Key Findings We were surprised by the low percentage, but noted a large number of students were close to meeting the expected level of achievement. 8/24/10 Actions taken 1) Modified scoring guide to include salient feature for each score at each level. 2) Clarified task we are asking students to complete. 3) Created guidelines for selection of readings. Re-evaluation Date Spring 2011.
23
ESL Sample
Intended Outcome Assessment Task Criteria /Expected Level of Achievement Results of Assessment Actions Taken
Conclusions Students were exposed to six writing modalities (twice as many as in ESL 34/34X) giving little time to understand the essay structure and apply skills to targeted writing tasks. Students were nearly able to achieve expected level of completion , but students' focus on the essay caused a drop in their use of standard academic English (see SLO 3). Also, the reading given to complete the task compared two groups as well as two time periods. Students struggled to organize their essays since they didn't know which task to complete to contrast the groups or the time periods.
24
Questions?