William Carey University Faculty Development

1
William Carey UniversityFaculty Development

• Staying ahead in an era of assessment
• Robert G. Cuzzolino, Ed.D.Vice President for
  Graduate Programs and Planning, PCOM

2

• Osteopathic Medical Education in an environment
  of planning and assessment

3
Objectives

• Be able to distinguish among mission, goals and
  outcomes
• Describe how a continuous quality improvement
  program works.
• Organize goals or competencies with related
  outcomes data in an understandable way
• Describe the conditions necessary for strategic
  planning to yield organizational or instructional
  change.

4
Objectives, continued

• Describe methods of longitudinal analysis
• Differentiate between the measurement of student
  achievement and the assessment of institutional
  or program effectiveness

5
Mission, Goals, Competencies, Outcomes

• External reviewers focus on mission as the
  foundation for institutional assessment and
  planning.
• Mission statements are just a starting point in
  the assessment/planning process chain.

6

• Mission
• Goals
• Competencies
• Student Outcomes
• Program Assessment
• Planning
  

7

• Mission demands goals
• Goals demand learning objectives
• Objectives demand measurement
• Measurement demands analysis
• Analysis demands planning
• Planning demands change
• Change demands review in context of
  mission

8

• Be kind to your outside reviewers.
• The linkages among institutional goals, program
  competencies and outcomes are not intuitive to
  most visitors, so why not map them out?

9
Institutional Learning Goal Academic Program Competency Group Outcomes Data
II. Communication The student acquires interpersonal and communications skills that result in effective information exchange and teaming with patients, their families, and other health professionals. Postgraduate assessment ( above comparative low quartile) Patient Instruction Skills 98 (x3.22/4) Standardized Patient Communications Assessment (end of year) 1-9 point assessment scale 1 poor, low level/ 9 superior skill 1st Year Standardized Patient Communications Assessment (end of year) Rapport 6.8 Empathy 7.2 Confidence 5.3 Info 7.0 Listen 6.6 Feedback 6.1 PE 7.3 2nd Year Standardized Patient Communications Assessment (end of year) Rapport 7.4 Empathy 7.3 Confidence 7.9 Info 8.0 Listen 7.4 Feedback 7.3 PE 8.1 Seniors rating of instruction in physician/patient relationship 89 appropriate 3.7 inadequate 7.2 excessive
10
A look at a hypothetical WCUCOM GCO Grid
11
On Change

• 1.3. The COM must connect its strategic plan
  to learning outcomes assessment in order to
  continuously improve the educational quality of
  its osteopathic medical education program.
• In other words, How does my COM make
  changes?
• Have the changes weve made been
  based on outcomes data?

12

• Assessment Data / SWOT Analysis /Departmental
  Plans /Community Input / Committee Actions/
  Accreditor Recommendations/ Budget Requests
  
• Planning Resource Allocation Responsibility
  for Change
• Progress Reporting
• Annual Plan Renewal

13
Outcomes, outcomes and more outcomes

• Constant testing will no more address the
  problems with our educational system than
  constantly putting an overweight person on the
  scale will cure obesity. -Anna Quindlen

14
Agreeing on the terminology

• There is a difference between the measurement of
  individual student achievement and the assessment
  of institutional or program effectiveness.
• Unfortunately, accreditation standards often
  refer to both of these aslearning assessment
  or outcomes.

15

• COCA Standard 6.6The COM must define,
  publish and implement educational outcomes, based
  on its own educational objectives that will
  prepare students for osteopathic graduate medical
  education.

16

• Your COM can not measure learning outcomes if it
  has not articulated its learning objectives.

17

• Standard 6.5 Documentation guideline for proving
  that your COM has stated learning objectives and
  assessment methodsCourse objectives and
  course syllabi, evaluation process for all
  courses, and outcomes from the evaluation
  process.
• Standard 6.6.1The COM must establish clinical
  competencies and a methodology to ensure they are
  being met.

18
Learning Assessment Plans

• Most institutions have or are developing learning
  assessment plans.
• These plans tend to focus on the individual
  student level and define the expected
  competencies, assessment methods, timing and
  (sometimes) explain how these outcomes are
  reported back to the college.

19

• Learning Assessment Plans are valuable at the
  individual student assessment level.
• They document the measurement of learning up
  front for students, and also compel faculty to
  clearly define objectives and develop measures
  of student achievement - - as a system.
• While they are a good basis for developing
  objectives and corresponding measures of
  learning, assessment plans do not produce program
  effectiveness data. They illustrate how you
  evaluate students not the program.

20
Question for the group

• What have you learned from Comlex I results?

21
200 Comlex scores mean nothing in and of
themselves. But if the lowest 50 scores are your
PBL track students, you have a program
effectiveness issue
22
When does a set of learning assessment data help
in planning?

• When it is aggregated
• When it is augmented by other data
• When it illustrates a trend
• When it is benchmarked or comparative
• When it suggests a course of action, a change, or
  the need for a resource.

23
What learning outcomes data sources can help
inform institutional change?

• Comlex scores are a mandated element.
• Others?

24

• Student clerkship ratings
• Course grade trends
• Standardized patient encounters
• Student survey data
• End of clerkship exams
• Alumni surveys
• AOA, state board data
• Freshman surveys Graduation Surveys
• Match data and residency selection data
• Certification

25
Council for Higher Education Accreditation

• Effectiveness is a broad concept that refers to
  the overall attainment of the mission and goals
  of a particular institution or program. As such,
  it may embrace various kinds of behavioral
  outcomes for students that go beyond student
  learning, such as employment, economic mobility,
  and contributions to personal and civic life

26
Summary on outcomes data

• In the self study, we have two needs for academic
  performance data
• First Student Assessment
• WCUOM must show that individual student
  achievement is regularly measured against the
  expected competencies in a planned manner. It is
  also important that we indicate when and how
  these expectations and their assessments are
  communicated.

27

• Second Program Evaluation
• The self study must coalesce individual
  student achievement results into meaningful
  program assessment data sets, and then show that
  the COM analyzed the outcomes data and took the
  appropriate action. Link curricular,
  instructional, organizational, and resource
  changes to data.

28
Example of an institutional effectiveness process
COMLEX and DO student success

• This brief presentation will share information
  that was developed by the PCOM learning resource
  committee and shared with the DO Curriculum
  Committee and the DO Admissions Committee.
• It departs from analysis of grades, clinical
  observation ratings and mean Comlex scores and
  treats retrospective data to inferential
  analysis, with conclusions as to meaning,
• but leaves the action plan to the faculty.

29
Tracking Change

• This is most difficult aspect of continuous
  quality improvement through strategic planning.

30
Longitudinal data

• What do you (will you) know about practice
  characteristics (geographic distribution,
  specialty, residency match, certifications) of
  your graduates ?
• Think ahead define your questions before you
  graduate a class.

31
Please rate the graduate, FIRST_NAME
LAST_NAME, on each of the following items by
circling the appropriate number. In making the
ratings please compare the intern with all
interns you have supervised, not just those in
your recent group
Top Upper
Lower Bottom Unable to


Quarter Middle Middle Quarter Judge

Quarter Quarter
Osteopathic Principles and Practice, which are
established and incorporated in the development
of skills. ____ ____ ____ ____
____
32
Question 17 Diagnostic Skill
33
Question 13 Understanding the Payor/
Reimbursement System
34
Strategic Plan

• Goal 2 Enhance academic quality, faculty
  development, scholarly activity and clinical
  service.
• k) Pursue state approval of interdisciplinary PhD
  research degree or pursue partnership with
  PhD-granting institution (see Goal 5 a)
• l) DO curriculum committee to assess the adequacy
  of business in medicine content and encourage
  co-curricular presentations on business,
  management and medical economics.
• m) Convert geriatric lecture content to on-line
  format reformat course to focus on interactive
  discussion

35
Summary on assessment

• Demonstrating the connection of learning outcomes
  to strategic planning is important.
• Student assessment and program assessment are not
  the same thing, but both are important to the
  self study, and both run on their own respective
  form of learning outcomes data.
• The COCA standards carry an expectation of
  longitudinal data.

36

• A look at the PCOM strategic plan