WTCS Encore Technical Skill Attainment September 18, 2008

1
WTCS -Encore Technical Skill AttainmentSeptember
18, 2008

• WTCS
• Annette Severson
• Sandy Schmit
• Roe Parker
• Julie Tyznik
• WTCS Foundation WIDS
• Judy Neill

2
Agenda

• What are the basics of assessment of Technical
  Skill Attainment (TSA)?
• Why are we placing emphasis on assessment of TSA?
• What does Carl Perkins 2006 (Perkins IV) require?
• How will WTCS accomplish assessment of TSA?
• What are the open issues and challenges?
• Whats next?

3
Welcome to the WTCS Technical Skills Attainment
Webinar!

• Weve organized the Webinar so that you will
  receive as much information about Assessment of
  Technical Skill Attainment as possible. To that
  end we are making the following features
  available to you
• Audio listen only via phone
• Video via your computer screen when signed into
  Webinar
• Question Answer written using question
  feature
• Attendee List visible on the tool bar
• Polls available as presenters activate them
• Recording of Session available online to
  participants online the next day
• Follow-up Survey available for you to complete
  the day after the session. Please help us by
  giving feedback for continual improvement!

4
Welcome to the WTCS Technical Skills Attainment
Webinar!

• Presenting
• Roe Parker WTCS Grant Development Coordinator
• Sandy Schmit WTCS Education Director
• Annette Severson WTCS Associate Vice President
• Judy Neill WTCS Foundation WIDS Director
• Answering/Monitoring Questions
• Julie Tyznik WTCS Coordinator Performance
  Management

5
What are the basics of assessment of Technical
Skill Attainment (TSA)?Roe Parker
6
Technical Skill Attainment

• An assessment that objectively measures a
  student's attainment of industry recognized
  skills upon graduation.

7
Big Picture Perspective

• Focuses on industry aligned skill sets.
• Students know they are acquiring skills that have
  value portability.
• Increases odds that students will be successful
  in their employment.
• Ensures that WTCS is educating students to meet
  the needs of employers.

8
Why are we placing emphasis on assessment of
TSA?Roe Parker
9
Drivers to Process

• Increased accountability to improve teaching and
  learning
• Accreditation (NCA, AQIP, industry)
• Carl D. Perkins IV legislation and funding

10
Benefits

• Students
• Feedback about skills needed on the job
• Added value for education
•  Colleges
• Improvement of teaching and learning
• Supports overall mission
•  Employers
• Assurance that WTCS graduates have attained
  technical skills needed on the job

11
What does Carl Perkins 2006 (Perkins IV)
require?Roe Parker
12
Carl D. Perkins 2006

• Newly revised Career and Technical Education
  (CTE) Law
• Key Concept - Demonstrate that program
  graduates have attained the knowledge and skills
  that are industry validated career and technical
  skills

13
Perkins IV Areas of  Emphasis

• Preparation for postsecondary education and
  employment
• Preparation not for jobpreparation but
  academic and technical preparation
• Increased emphasis on achievement of a degree,
  certificate or credential

14
Increased Accountability

• New accountability provisions are a reminder
  that
• Funds are not an entitlement
• Use of funds is flexible responsive to the
  accountability data

15
Increased Accountability

• Separate secondary and postsecondary measures
• Performance levels at state and local levels
• Negotiations and redirection of funds

16
Increased Accountability for Direct Measurement
of Learning

• Indirect Measures

• Direct Measures

• Data that support inferences about the quality of
  teaching and learning and give insight into why
  achievement is high or low
• Enrollment
• Retention rates
• Course completion
• Student/graduate/employer satisfaction surveys
• Placement statistics

• Data that provide evidence that students have
  achieved the intended outcomes
• Performance assessments with rubrics
• Portfolios
• Artifacts
• Performances
• Outcome referenced tests

17
Increased Accountability for Direct Measurement
of Learning
Prior accountability required only indirect
measurement
Numerator Number of CTE concentrators who passed
80 of the technical courses they attempted
during the three year reporting period
Measure

• Denominator number of CTE concentrators who took
  technical courses during the three year reporting
  period

18
Increased Accountability for Direct Measurement
of Learning
New OVAE Guidance for Perkins IV Measurement
requires direct measures of skill attainment
Numerator Number of CTE concentrators who passed
technical skill assessments that are aligned with
industry-recognized standards, if available and
appropriate, during the reporting year
Measure

• Denominator Number of CTE concentrators who took
  technical skill assessments during the reporting
  year

19
Increased Accountability for Direct Measurement
of Learning

• Department of Education goal is to have end of
  program outcomes assessments that are
• Direct measures of outcomes
• Independent
• 3rd-Party
• Employer validated
• By 2015 most of our assessment should be external
  or WTCS assessments.

20
Increased Accountability for Direct Measurement
of Learning
Wisconsin Standards for Assessment - WTCS

• WTCS TSA Assessments
• External TSA Assessment
• (ex. NCLEX, Barber/Cosmetology, etc.)
• Indirect Assessments

21
Increased Accountability for Direct Measurement
of Learning
Wisconsin Standards for Assessment - WTCS

• WTCS Assessment Standards
• WTCS approved assessment that objectively
  measures student attainment of industry
  recognized skills upon graduation.
• May be developed and/or implemented by the state
  system.
• May be developed and/or implemented locally
  according to system guidelines and with industry
  (advisory committee) approval.
• Must be valid and reliable.
• External Assessment Standards
• Indirect Assessment Standards

22
Increased Accountability for Direct Measurement
of Learning
Wisconsin Standards for Assessment - WTCS

• WTCS Assessment Standards
• External Assessment Standards (ex. NCLEX, ASE,
  Barber/Cosmetology)
• External, third-party assessment that objectively
  measures student attainment of industry
  recognized skills upon graduation.
• Indirect Assessment Standards

23
Increased Accountability for Direct Measurement
of Learning
Wisconsin Standards for Assessment - WTCS

• WTCS Assessment Standards
• External Assessment Standards
• Indirect Assessment Standards
• Non-assessment related indicators such as
  indirect measures.
• GPA and/or Course Completion
• Program Completion Graduation
• Teacher-developed exams that do not meet WTCS
  standard (not WTCS approved)

24
New Accountability for Direct Measurement of
Learning
WTCS will move toward a balance of WTCS TSA
Assessments and External TSA Assessments
WTCS Assessment
External Assessment
Indirect
2012-2013
25
Perkins Reserve Grants

• Allocation 1,015,300
• Purpose
• Strengthen CTE Programs through assessment of
  Technical Skills Attainment (TSA)
• Programs of Study-high school to tech college
  transition

26
Perkins Reserve Grant Areas

• Technical Skills Attainment
• Identify model
• Identify common outcomes, protocols
• Describe measures for a statewide approach
• Programs of Study
• Identify key elements
• Align curriculum postsecondary/secondary
• Opportunities for dual/concurrent enrollment
• Identify degrees or industry credentials

27
FY09 Reserve Grants-TSA

• Technical Skill Attainment
• FVTC-Accounting TSA
• LTC-Welding Industrial TSA
• MATC-MILW-Automotive Tech TSAs
• NWTC-Early Childhood Education TSA
• NATC-Criminal Justice TSA
• Programs of Study
• MPTC-CORD training
• FVTC-Web site feasibility study
• SWTC-Electro-mechanical

28
How will WTCS accomplish assessment of TSA?Judy
Neill
29
Assessment of TSA - WTCS Strategies

• Ensure that programs are based on industry-driven
  program outcomes.
• Review existing program outcomes, checking for
  industry relevance and measurability revise as
  needed.
• Align core technical program outcomes for
  programs offered by multiple colleges.
• Determine which programs have third
  party/external assessments that meet standards
  for validity, reliability, fairness, and
  feasibility.
• Develop or identify existing summative
  assessments that meet standards for WTCS
  Assessment of TSAs.
• Develop strategies for collecting and reporting
  data on assessment of TSAs.

30
WTCS WIDS Model

• Who are the learners?

• What do they need to be able to achieve?

• How will we know and show when theyve achieved
  it?

• How will they get there?

31
WTCS Program Outcomes

• Measurable, observable, and field-specific skills
  major outcomes
• Number 5-7 per program (guideline not rule)
• Originate from
• Current DACUMs
• Accrediting Agencies
• National (or other) Skill Standards
• Advisory Committees
• Threaded through courses
• Performance verified with summative assessment of
  skill performance

32
Integrating Outcomes
33
Dental Hygiene Example
ADA Dental Hygiene Education Standard
Graduates are competent in applying ethical,
legal and regulatory concepts to the provision
and/or support of oral health care services.
External
Dental Hygiene Program Outcome
Incorporate into dental hygiene practice
professional laws, regulations and policies
established by the licensing state and regulatory
agencies
Program-Level
Dental Hygiene Competency
Respond to a request to perform a task that is
not legally permitted to be delegated to a dental
hygienist.
Course-Level
34
Sound Assessment

• Fair
• Valid
• Reliable
• Learners informed of expectations up front!
• Feedback to learners

• Reliable
• Performance assessment based on consistent
  rubrics, scoring guides, and rating scales
• Consistent process - each learner is assessed in
  same way as other learners

• Valid
• Outcomes based on standards (industry)
• Measures intended outcomes
• Measures application and critical thinking

35
WTCS Standards for3rd-Party External
Assessments Sandy Schmit
36
3rd Party TSA

• Also referred to as
• External
• Direct Method
• Credential
• License
• Generally controls entrance into an occupation

37
Considerations for selecting 3rd Party External
Assessments

• Validity / Reliability
• Alignment to Standards
• Availability of data
• Value to the Student
• Development / Administrative Costs
• Pooling / Sharing Resources

38
Considerations for selecting 3rd Party External
Assessments

• Can we obtain the results/data?
• Are cost and administration feasible?
• Does it add value or control entry to the
  student/occupation?
• Is it valid, reliable, reputable, recognized?
• Is it summative, cumulative or a partial
  measurement of skills?

39
Considerations for selecting 3rd Party External
Assessments

• Does the assessment align with program outcomes?
• Does the assessment measure what needs to be
  measured?
• Does the data help improve and strengthen our
  programs?

40
WTCS 3rd Party Assessments Currently in Place

• Occupations regulated by the Department of
  Regulation and Licensing
• Barber/Cosmetology
• Funeral Director
• LPN
• Registered Nurse
• Dental Hygienist
• Massage Therapist
• Occupational Therapy Assistant
• Physical Therapy Assistant
• Respiratory Care Practitioner
• Veterinary Technician
• Wisconsin Registered Interior Designer

41
Guidelines for Selecting 3rd Party Assessments

• This is a guiding document to assist districts in
  determining direct technical skill assessment
  appropriateness.
• Districts will work with the state office staff
  in the determination.
• Keep in mind this is for a small number of
  programs.

42
Guidelines for Selecting 3rd Party Assessments
43
Guidelines for Selecting 3rd Party Assessments

• Concerns
• Only measures part of the program outcomes-All
  Code-No Application
• Does not indicate job performance
• Does it help employers?
• Focused only on technical skills, what about
  academic skills and soft skills?

44
3rd Party Assessment - Example

• Automotive Service Excellence
• Long-standing credential of automotive
  technicians
• A series of 8 certification areas/assessments
• Would it work for Perkins?

45
3rd Party Assessment - Example

• Logistics Professional
• CPL/DL-International Society of Logistics
• CPIM/CSCP-Association for Operations Management
• PLS/CTL/DLP conferred by the American Society of
  Transportation Logistics

46
3rd Party Assessment - Example

• Radiotelephone Operator's License
• Administered by the Federal Communications
  Commission
• Required to adjust, maintain or repair
  radiotelephone transmitters in aeronautic
  services

47
WTCS Standards for TSA AssessmentsJudy Neill
48
WTCS Assessment Standards

• WTCS approved assessment of learning outcomes
  that objectively measures student attainment of
  industry-recognized technical skills upon
  graduation

49
WTCS Assessment Standards

• WTCS approved assessment objectively measures
  student attainment of industry recognized skills
  upon graduation and
• is valid and reliable
• meets WTCS standards/guidelines
• is approved by industry (advisory committee)
• is developed and/or implemented collaboratively
  by the WTCS colleges or
• developed and/or implemented locally according to
  system guidelines and with industry (advisory
  committee) approval.

50
Available on WTCS CurriculumBank (http//WTCS
CurriculumBank Curriculum Projects Assessment
of Technical Skill Attainment)
51
How can WIDS help?

• Use WIDS Model and Software Tools to
• Create/maintain Program Outcome Summaries for all
  new and existing programs.
• Develop/maintain Course Outcome Summaries for all
  courses.
• Design Performance Assessment Tasks for course
  level (formative) and summative assessments.
• Use WIDS Analyzer to build Learning Outcome
  Matrices that document how outcomes are linked
  with programs and courses.
• Present WIDS documentation as a central feature
  in accreditation.

52
How can WIDS help?

• WIDS Team
• Facilitate collaborative development of
• DACUM or Industry Review Focus Groups
• Program Outcomes
• Program Outcome Summaries
• Summative Performance Assessments
• Produce
• Industry Approval Reports
• Analyzer Reports
• Updated WIDS files and reports
• Post WIDS files and reports to CurriculumBank

53
What is the WTCS Implementation Strategy?Annette
Severson
54
Does TSA require WTCS Program Alignment?

• Statewide Aligned Programs
• program outcomes
• course titles
• course numbers
• course credits
• course competencies
• performance standards

• TSA
• common core program outcomes
• common assessment strategies
• common data collection strategies (TBD)

55
Major Communication Documents

• 5-Year Perkins State Plan
• July 1, 2008 June 30, 2013
• Annual Carl D. Perkins Grant Guidelines
• Annual College Perkins Plans
• WTC System Office Policy Directives

56
High Level Concepts

• TSA applies to 349 approved programs.
• WTCS will work with districts to classify
  programs in one of the three categories.
• Districts will collaborate to determine if a
  third party assessment is available and useful or
  develop internal skill attainment standards for
  programs with the same program number.
• Districts with unique programs will work with
  WTCS to identify or develop an appropriate TSA.
• WTC System Office will approve all assessments,
  including 3rd Party assessments or certifications.

57
High Level Concepts

• All WTCS programs are currently being assessed at
  the indirect level (or higher).
• Initially priority will be placed upon programs
  with the largest student enrollment along with
  other criteria.
• WTC colleges will strive to phase out course
  completion as primary assessment over the next 7
  years.
• WTCS and Districts will work to implement WTCS
  approved or 3rd-party assessment by June 30,
  2013.
• WIDS and the WIDS model will be used as core
  tools for WTCS approved assessment standards.

58
General Timeline
59
TSA Project Assumptions

• System will use a multiple-year phased in
  approach.
• Process will include
• faculty input
• industry input
• a role for college advisory committees
• input from college advance/research staff
• Assessments are or will be industry aligned.
• Assessments are or will be based on program
  outcomes.
• The same or similar program at multiple districts
  will define a core set of programs outcomes used
  to measure TSA.

60
TSA Project Assumptions

• Each college will develop a WTCS Assessment of
  Technical Skill plan for WTC System Office
  approval.
• Colleges will collaboratively create a plan to
  update the assessment.
• Colleges will internally document the outcome
  assessment results for each program student.
• Colleges will report student data via the Client
  Reporting System.
• WIDS will be used to document program outcome
  assessment for WTCS approved assessments.

61
WTC System Office Support

• Reserve Grants
• Grant Guideline Meetings
• State Called Meetings
• Special Topic Meetings
• Webinars
• WTCS Systemattic Web Site
• http//systemattic.wtcsystem.edu/Grants/Perkins-4/
  perkins4.htm
• Written Guides

62
WTC System Office Resources

• Staff
• Julie Tyznik
• Coordinator Performance Management
• [phone]
• julie.tyznik_at_wtcsystem.edu
• Web Sites
• CurriculumBank - Assessment of Technical Skills
• (http//curriculumbank.org Curriculum Projects
  Assessment of Technical Skill Attainment)
• WTCS Perkins IV Page (coming soon)