Assessment at the TTUHSC

1
Assessment at the TTUHSC

• Or.What does assessment have to do with me?

2
Assessment

• Process of collecting and reviewing evidence
  about the TTUHSCs academic and administrative
  programs and services and using it to evaluate
  these programs and services to improve their
  quality

3
Institutional Effectiveness

• The process by which the TTUHSC demonstrates how
  well it is succeeding in accomplishing its
  mission and meeting its goals

4
SACS (Southern Association of Colleges and
Schools) Principles of Accreditation

• 2.5 The institution engages in ongoing,
  integrated, and institution-wide research-based
  planning and evaluation processes that
  incorporate a systematic review of programs and
  services that (a) results in continuing
  improvement and (b) demonstrates that the
  institution is effectively accomplishing its
  mission.
• 3.3.1 The institution identifies expected
  outcomes for its educational programs and its
  administrative and educational support services
  assesses whether it achieves those outcomes and
  provides evidence of improvement based on
  analysis of those results.

5
TTUHSC Mission

• To improve the health of people by providing
  educational opportunities for students and health
  care professionals, advancing knowledge through
  scholarship and research, and providing patient
  care and service

6
TTUHSC GOALS

• Prepare competent health professionals and
  scientists
• Increase externally funded, peer-reviewed
  research, especially NIH funded research, and
  research focused on aging, cancer, and rural
  health
• Improve access to quality health care for the
  TTUHSCs target populations
• Prepare health professions students for an
  increasingly diverse workforce and patient
  population
• Provide leadership in the development of
  partnerships and collaborations to improve
  community health
• Operate TTUHSC as an efficient and effective
  institution

7
Relationship of Types of Planning at an
Institution
Institutional Effectiveness is focused on
Results Improvement
Strategic Planning is focused on Process Action
Institutional Mission, Goals Strategies
Answers Question What actions should we take to
support the mission goals of the institution?
Answers Question How well are our students
learning our administrative services (AES)
functioning?
Unit Mission/Functions
Academic Planning
Budget Planning
Administrative Planning
Facilities Planning
Means of Assessment
Use of Results
Assessment Results
Outcomes
Inform the Planning Process
Adapted from Nichols, J.O Karen W., A Road Map
for Improvement of Student Learning and Support
Services Through Assessment
8
What is WEAVEonline?

• A data-based tool for tracking assessment at the
  unit or program level
• Tracks both student learning outcomes for degree
  programs and outcomes for administrative
  educational support units
• Demonstrates links between units/programs and the
  TTUHSC mission, goals, strategies
• Reporting capabilities

9
WEAVE
10
WEAVE

• Write expected outcomes
• Establish criteria for success
• Assess performance
• View assessment results
• Effect improvements

11
Whats the Assessment Timeline?

• During our 2006-2007 ramp up year
• Assessment plans entered during the Fall Semester
  (ideally by Oct. 31, 2006)
• Assessment activities Sept. 2006 through June
  2007
• Assessment reports due July 15, 2007
• Includes findings or results
• Includes a brief action plan on how findings will
  be used for improvement
• Revised assessment plans due Sept. 2, 2007

12
Who will use WEAVE?

• All Academic degree programs (for student
  learning outcomes)
• Administrative and educational support
  programs/units
• Includes academic program units, i.e. Dept. of
  Pediatrics, Office of the Dean for SON, etc.
• Administrative units, i.e. Traffic Parking,
  IT-El Paso
• Educational Support Units, i.e. Outreach Service,
  TTUHSC Libraries SOAHS Office of Admissions
  Student Affairs

13
Help!

• Office of Institutional Planning Effectiveness
  (OIPE)
• Add new office number
• 743-2918
• http//www.ttuhsc.edu/admin/oipe/
• Staff
• Sharon Kohout, Director
• H.D. Stearman, Associate Director
• TeRee Reese, Senior Business Assistant

14
Assessment 101

15
Assessment

• Assessment is a systematic process of gathering
  and interpreting information to discover if your
  program is meeting its outcomes and then using
  that information to enhance your program.
• Assessment is a process designed to answer the
  question Are our efforts bringing forth the
  desired results?

16
Who needs to do assessment?

• All academic degree programs must conduct ongoing
  assessment related to student learning outcomes
• Administrative educational support programs or
  units
• Possible criteria Does your program or unit..
• have a purpose (or a mission) statement?
• provide a unique service?
• have a box on the organizational chart?
• have a separate budget?
• There are no rules
• At what level does assessment makes sense if your
  focus is improvement of your program or services?

17
(No Transcript)
18
Where do you start?

• Mission or Purpose Statement
• A brief description of the unit or programs
  mission and purpose stated in broad terms. Be
  sure to mention how the work of the unit supports
  some part of the Universitys mission and/or
  goals.
• Keep the mission statement to 2 3 sentences
  (for WEAVE).
• Should be revisited annually in case services are
  eliminated or added.

19
Core Functions/Services

• Four to seven functions/areas of responsibility
  that ensure that the mission is being
  accomplished
• Helpful for determining outcomes/objectives and
  potential areas for improvement

20
Ask Yourself These Questions

• What decision did you make about your unit or
  program last year?
• What evidence did you use to inform that
  decision?
• What was it that you were trying to influence or
  change about your unit/program when you made that
  decision?

21
Outcomes-Based Assessment

• Most people
• Do capitalize on their innate intellectually
  curiosity to find out what works
• Dont articulate their intended end results
  (e.g., outcomes) ahead of time
• Dont document the decisions made based on their
  results
• Dont follow up later to see if their decisions
  made the intended improvement

22
The Assessment Cycle(Bresciani, 2003)

• The key questions
• What are we trying to do and why? or
• What is my unit or program supposed to
  accomplish?
• How well are we doing it?
• How do we know?
• How do we use the information to improve or
  celebrate successes?
• Do the improvements we make work?

ASSESS
PLAN
IMPROVE
23
Assessment( Bresciani, 2006)

• Most importantly, it should be
• Understood
• Inclusive
• Meaningful
• Manageable
• Flexible
• Truth-seeking/objective/ethical
• Systematic
• Inform decisions for continuous improvement or
  provide evidence of proof
• Promote a culture of accountability, of learning,
  and of improvement

24
The Purpose of Assessment

• Outcomes-Based assessment does not exist for
  assessments sake
• It is taking what most of us already do, and
  making it systematic
• It is NOT personnel evaluation
• Its purpose is to reflect on the end result of
  doingthe outcome. Are we accomplishing that
  which we say we are?

Bresciani, 2002
25
Purpose of Assessment (continued)

• Reinforce or emphasize the mission of your unit
• Improve programs and/or performance
• Inform planning
• Inform decision making
• Evaluate programs, not personnel

Bresciani, 2002
26
Purpose of Assessment (continued)

• Assist in the request for additional funds from
  the University and external community
• Assist in the re-allocation of resources
• Assist in meeting accreditation requirements,
  models of best practices, and national benchmarks
• Celebrate successes
• Create a culture of continuous improvement a
  culture of accountability, of learning, and of
  improvement

Bresciani, 2002
27
Components of a TTUHSC Assessment Plan

• Program/Unit Name
• Program/Unit Mission or Purpose Statement
• Outcomes
• Student Learning or Administrative
• Link to the TTUHSC Strategic Plan
• Measure (Method of Assessment)
• Target Level (Criteria for Success)
• Findings (Assessment Results)
• Summarize the results for each outcome
• Summarize the process to verify/validate the
  results
• Action Plan (Use of Results)
• Summarize the decisions/recommendations made for
  each outcome

28
Mission Statement

• A mission statement needs to communicate the
  essence or purpose of your unit or program.
• Your mission statement answers the question
• What do we do and how do we support the TTUHSC
  mission?

29
Outcomes

• Outcomes are statements that describe results
  that you expect to see.
• in the way you deliver your services
• related to internal goals you might have set
  within your unit
• as you carry out your functions or
  responsibilities
• Outcomes are specifically about what you want the
  end results of your activities to be.

30
Guidelines for Formulating Outcomes

• Must be related to something under the control of
  the unit
• Should be worded in terms of what the unit will
  accomplish or what its clients should think,
  know, or do following the provision of services
• Should lead to improved service
• Must be linked to a TTUHSC strategy and/or goal
  (WEAVE will make the linking simple)
• YOU decide which strategies and/or goals your
  unit best links to (can be more than one, but
  must link to a goal at the very least see Goal
  5)

31
Ideas for Deciding on Outcomes

• Review your core services/ functions/responsibilit
  ies
• Brainstorm with other staff about potential areas
  for improvement
• Look at data/trends to determine issues or areas
  that need time and attention
• What are areas in which you are held accountable
  by senior management or external audiences?
• Remember outcomes-based assessment is about
  improvement not just doing
• Start small with just 2 or 3 outcomes

32
Examples of Intended Outcomes

• All news media inquiries concerning the
  University will receive appropriate and timely
  responses.
• An increasing number of undergraduate students
  will engage in research with faculty and graduate
  students.
• Seminars and conferences offered by this program
  will provide enhanced opportunities for faculty
  to engage in interdisciplinary discussion and
  develop networks for future research
  collaborations.
• Customers will receive prompt assistance in
  effectively resolving technical problems related
  to systems, networks, and desktop applications.

33
Questions to Ask Yourself About Outcomes

• Is it measurable/identifiable?
• Is it meaningful?
• Is it manageable?
• Who is the target audience of my outcome?
• Who would know if my outcome has been met?
• How will I know if it has been met?
• Will it provide me with evidence that will lead
  me to make a decision for continuous improvement?

34
Before Choosing A Method or Measure

• Think about what meeting the outcome looks like
• For example
• All news media inquiries concerning the
  University will receive appropriate and timely
  responses.
• What does it look like when your outcome has been
  met?
• How do you currently deliver this outcome?
• There may be clues in the delivery of the outcome
  that help you determine how to assess it

35
Before Choosing a Method or Measure (continued)

• Think about collecting data
• from different sources to make more meaningful
  and informed decisions for continuous improvement
  (e.g., surveys, observations, self-assessment)
• that you believe will be useful in answering the
  important questions you have raised
• that will appeal to your primary audience or to
  those with whom you are trying to influence

36
Measurement Methods

• Evidence of learning- basically two types
• Direct-
• methods of collecting information that require
  the customers/students to display their knowledge
  and skills
• Indirect
• methods that ask customers/students or some one
  else to reflect on the customer/student learning
  rather than to demonstrate it

(Palomba and Banta, 1999)
37
Some Methods That Provide Direct Evidence

• Observations of customer/student behavior
• Tracking the use of a service (i.e. hits on a
  website)
• Tracking processing time
• Counts of participants/customers served
• Tracking complaints and how they are resolved

38
Some Methods That Provide Direct Evidence
(continued)

• Document analysis (e.g., meeting minutes,
  policies, handbooks)
• Observations of work performance
• Web tracking program analysis
• Other document tracking analysis

39
Some Methods That Provide Indirect Evidence

• Client satisfaction surveys (students, staff,
  faculty, alumni, other customers)
• Interviews and focus groups with participants,
  customers, or other stakeholders
• Benchmarks set by national, state, or peer
  organizations or institutions
• Retention rates
• Enrollment trends

40
Examples of Measures

• Surveys of customer satisfaction
• Counts of program participants growth in
  participation
• Focus groups, individual interviews, phone
  surveys
• Feedback from advisory groups or committees
• Analysis of service usage

41
Choosing an Instrument

• What outcome(s) are you measuring?
• Who is being assessed? How often do I have access
  to them? Do I know who they are?
• What is my budget?
• What is my timeline?
• What type of data is most meaningful to me
  direct/indirect and qualitative/quantitative
• Who will have responsibility for data collection?

42
Target Levels (Criteria for Success)

• What target will you set for yourself during the
  coming year in order to accomplish your outcome?
• What level of accomplishment do you hope to see?
• Target levels should specific, measurable, and
  attainable.
• You are encouraged to set more than one target
  level or criteria for success for each outcome.

43
Examples Outcomes with Criteria for
Success/Target Level

• Students will be satisfied with the health care
  provider on their campus.
• Target On Student Satisfaction Survey, 90 will
  respond that they are satisfied or very satisfied
  with health care provider on their campus.
• Instructional support services will be provided
  to meet the needs of TechLink faculty.
• Target By Dec. 2006, faculty using TechLink
  will be surveyed to determine three most critical
  areas of need
• Target By June 2007, a workplan for addressing
  the major areas of need will be presented to IT
  leadership.

44
Conduct Assessment Activities

• Be sure you have a plan for conducting the
  activities that will lead to accomplishment of
  your target level

45
Findings (Results of Assessments Conducted)

• Summarize briefly the major findings from your
  assessments
• 70 of respondents indicated that they had
  received a response to their request for service
  within 24 hours.
• Customers expressed frustration with the wait
  time for help desk requests.
• Feedback gathered from our advisory group
  indicated that department staff desire additional
  assistance with web design.

46
Closing the Loop

• Briefly report method of assessment for each
  outcome
• Document where the customers/students are meeting
  the intended outcome
• Document where they are not meeting the outcome
• Document decisions made to improve the program
  and assessment plan
• Refine assessment method and repeat process after
  proper time for implementation

47
Action Plans (Use of Results for Improvement)

• How do you plan to use the results of your
  assessments to improve the units programs or
  services?
• Briefly describe any improvements, changes, or
  plans that you will undertake
• If you meet your target levels, review your
  outcomes to see if other improvements or
  activities are needed

48
Examples of Improvement Statements

• A new campaign emphasizing to new graduates the
  benefits of annual giving to the TTUHSC was
  developed and implemented in July 2004. We
  continue to monitor our progress in encouraging
  this group to contribute.
• Since our service demands have increased,
  additional staff are needed to provide adequate
  coverage, and will be requested in the next
  fiscal year.
• Our focus groups interviews with users indicated
  a preference for electronic reports, so the hard
  copy version was discontinued.

49
Demonstrate Document Use of Results for
Improvements

• Most important step data-based decision making
• Data collected from doing assessment often will
  inform Strategic Planning

50
Relationship of Types of Planning at an
Institution
Institutional Effectiveness is focused on
Results Improvement
Strategic Planning is focused on Process Action
Institutional Mission, Goals Strategies
Answers Question What actions should we take to
support the mission goals of the institution?
Answers Question How well are our students
learning administrative services (AES)
functioning?
Unit Mission/Functions
Academic Planning
Budget Planning
Administrative Planning
Facilities Planning
Means of Assessment
Use of Results
Assessment Results
Outcomes
Inform the Planning Process
Adapted from Nichols, J.O Karen W., A Road Map
for Improvement of Student Learning and Support
Services Through Assessment
51
Take-Home Messages

• You do not have to assess everything you do every
  year.
• You dont have to do everything at once-start
  with a reasonable number of intended outcomes
• Think baby steps
• Be flexible
• Acknowledge and use what you have already done.

52

More Take-Home Messages

• Assessment expertise is available to help you
  evaluate your program better not to evaluate
  your program for you
• Borrow examples from other institutions to modify
  as appropriate
• Time for this must be re-allocated
• We allocate time according to our values and
  priorities

53
Resources

• Each Other
• Web Resources
• http//www2.acs.ncsu.edu/UPA/assmt/
• http//www2.acs.ncsu.edu/UPA/assmt/resource.htmhb
  ooks
• http//www.unt.edu/SACS/ie_nonacademic/index.htm

54
Help!

• Office of Institutional Planning Effectiveness
  (OIPE)
• 743-2918
• http//www.ttuhsc.edu/admin/oipe/
• Staff
• Sharon Kohout, Director
• H.D. Stearman, Associate Director
• TeRee Reese, Senior Business Assistant