Examiner Training

1
Examiner Training

• Process comments
• Results comments
• Scoring

2
Agenda

• Classroom training (Examination focus) Day 3
• Key Factors
• Process Items
• Strength Comments
• Opportunity for Improvement Comments
• Results Items
• Strength Comments
• Opportunity for Improvement Comments

3
Agenda

• Classroom Training (Examination focus) Day 4
• Scoring process for Process and Results Items
• Overview of what to expect as a member of the
  Board of Examiners
• Graduation

4
General information

• There was one trophy winner in 2008. The winner
  was Minneapolis VA Medical Center, Minneapolis
  Minnesota. Their 2008 application will be used
  as the case study for this section of the
  training
• Sections of the Minneapolis VAMC application to
  be used in training include
• Organizational Profile
• Responses to Items 1.2, 2.1, and 7.6

5
What have we already learned?

• A great deal of the information examiners need to
  know has already been covered in this class.
  This includes
• Understanding Approach, Deployment, Learning, and
  Integration
• Understanding Levels, Trends, Comparisons,
  Integration, and projections
• Developing and using a Key Factors Worksheet

6
We also learned

• The following are NOT considered when assessing
  an application
• Organization of the application. As an examiner,
  if you can find the information needed then you
  give the applicant credit
• Typos
• Incorrect figure numbers
• Poor quality binding

7
General Examination Process

• Review application to determine if you have a
  conflict of interest
• Conflicts include
• You examined this applicant within the past three
  years
• You worked for this applicant within the past
  three years
• Your close friend or relative either works at the
  applicant site or is a customer of the site.
• This includes the recent past. For example, my
  dad was a patient at X. He passed away two
  months ago.
• NOT a conflict
• You examined the applicant five years ago
• You worked for the applicant 8 years ago
• You participated in a SOARS or OAI site visit to
  the applicant last year

8
General Examination Process

• Read through the entire application (a quick
  read)
• Develop a Key Factors Worksheet
• Associate Key Factors with a Criteria Item
• Examine the Item responses and compare to the
  Criteria
• Take notes in the application book
• Draft Strength and OFI comments
• Repeat steps 3 through 5 for all Items
• Score all Items

9
Developing a Key Factors Worksheet

• Read Organizational Profile questions in the
  Criteria book
• Review Organizational Profile responses in the
  application and underline key facts
• Draft a Key Factors Worksheet by recording the
  key facts about the applicant
• This Key Factors Worksheet is a key tool in
  examining an application

10
Developing a Key Factors Worksheet

• Exercise 20 minutes
• Review Organizational Profile responses in the
  application
• Underline key facts
• We will use this marked-up Organizational Profile
  as our Key Factors Worksheet for purposes of the
  training class

11
Associate Key Factors with an Applicants Item
Responses

• Exercise 10 minutes
• Read the Overall Item Requirements for 1.2
• Review the applicants Organizational Profile
• Select the 4 6 underlined facts that you
  believe are relevant when evaluating the
  applicants responses to this Item
• Record these facts at the topic of a piece of
  paper

12
Taking Notes in an Application

• Exercise 20 minutes
• Evaluate 1.2b(12) and 1.2c(12)
• Read the applicants responses to Item 1.2
• For each Multiple Requirement within the Criteria
• Read the criteria question
• Read the applicants response
• Write A or A-, D or D-, L, or L-, I or I- in
  the margin next to the applicants response to
  indicate if the scoring factor was positively
  addressed or not

13
Process Strength Comments

• Strength comments
• Are used when the applicant provides evidence of
  ADLI in reference to the criteria
• Strength comments contain two parts
• An initial phase or sentence linking the strength
  to the criteria
• A final phrase or sentence providing evidence
  (examples) from the application to support the
  strength

14
Process Strength Comments, Part One

• Subject from the Criteria or application
• The senior leaders
• The organization
• The school
• The medical center
• The cemetery
• The applicant

• Verb(s) and specific Requirements from the
  Criteria
• review organizational performance
• Communicates, listens, and learns
• Makes needed data and information available

15
Process Strength Comments, Part One

• EXAMPLE
• 1.1.a(1) Senior leaders set and deploy the
  organizations vision and values through the
  leadership system to the workforce...
• Note When you pull from the Criteria to write
  your comment, you must be accurate. The Criteria
  says to the workforce, to key suppliers and
  partners, to patients and to other customers.
  In this case there is the link to the Criteria
  and that is what is evidenced in the application.

16
Process Strength Comments, Part One

• EXAMPLE
• 1.1.a(1) Senior leaders set and deploy the
  organizations vision and values through the
  leadership system to the workforce...
• Note
• Address Item 1.1, Area a, multiple requirement
  (1)
• Question Which aspects are addressed in this
  comment? A, D, L, and/or I

17
Process Strength Comments, Part Two

• Transition
• By/through/using
• This is demonstrated by

• Example(s) from the application
• Provide the activities or processes that led you
  to the conclusion that this was a strength the
  facts, the WHAT, the HOW

18
Process Strength Comments, Part Two

• by using an annual Strategic Planning process
  and numerous communication methods, such as
  newsletters, meetings, and stakeholder sessions
  to communicate to all faculty, staff, students,
  and key stakeholders and partners. The
  leadership system involves many participants and
  decision-making levels.
• Note This information comes from the
  application.

19
Process Strength Comment - Tips

• Your comment can be about one or more of the four
  aspects (ADLI)
• Only comment on the aspects that are particularly
  strong
• Support your findings with examples from the
  application
• Review the scoring guidelines does your comment
  align with the performance level that you feel is
  representative of the applicant?

20
Process Strength Comments

• Exercise 30 minutes
• Review your 1.2b(12) notes
• Draft two Strength comments
• For each comment
• Underline the link to the criteria
• Circle the examples from the application
• Identify a speaker for the group

21
Process Opportunities for Improvement (OFI)
Comments

• The second part of an Item level examination is
  the identification and preparation of OFIs
• An OFI identifies an area (usually one factor) in
  the applicants response that in some way does
  not fully respond to a criteria question and may
  be of value to the applicant in its efforts to
  achieve its business objectives
• How does an examiner determine if a possible OFI
  is of value to the applicant?

22
Process Opportunities for Improvement Comments

• OFIs contain two parts
• A statement containing the Criteria and examples
  from the application
• A statement describing the possible impact on the
  organization

23
Process Opportunities for Improvement Comments
Part One

• Introductory Statement
• It is not clear how/what
• It is unclear how/what
• It is not evident how/what
• It is not apparent how/what

• Subject from the Criteria or Application
• The senior leaders
• The organizations
• The medical center
• The MSN
• The regional office

24
Process Opportunities for Improvement Comments
Part One

• Verb(s) and specific requirements from the
  Criteria
• Reviews organizational performance
• Communicates, listens, and learns
• Makes needed data and information available
• Manages the complaint process

25
Process Opportunities for Improvement Comments
Part Two

• The second part of an Opportunity for Improvement
  comment is a statement describing the possible
  impact on the organization. This is often
  referred to as a so what statement
• Why dont we use so what statements in Strength
  comments?
• Why do we use so what statements in an
  Opportunity for Improvement comments?

26
Process Opportunities for Improvement Comments
Part Two

• The so what statement describes the benefit or
  consequence associated with the Opportunity for
  Improvement.
• If you know the part of the criteria that is not
  addressed in the applicants response you could
  look up a standard OFI response in a Baldrige
  reference book
• For example, an applicant does not gather
  information from customers of competitors. The
  so what might be,
• Without information from customers of competitors
  leaders may be hindered in their ability to win
  over these customers and gain market share.

27
Process Opportunities for Improvement Comments
Part Two

• If a reference book can provide a standard OFI
  comment why doesnt the Carey program provide
  teams with a reference book?

28
Process Opportunities for Improvement Comments
Part Two

• The applicant puts a great deal of time and
  effort into their application
• Examiners want to provide value-added information
  to the applicant
• Value-added OFIs are linked to areas that have
  been identified as important to the applicant
• So what statements need to be specific enough
  and important enough to motivate applicant
  leadership to act
• Answer A generic OFI from a reference book may
  be a start but it is not specific enough to
  provide the level of importance needed to
  facilitate action
• ALSO, MOST APPLICANTS ARE SMART ENOUGH TO
  IDENTIFY GENERAL OFI COMMENTS

29
Process Opportunities for Improvement Comments
Part Two

• So, where does the information to support a good
  so what statement come from?
• Answer The Organizational Profile - the Key
  Factors Worksheet

30
Process Opportunities for Improvement Comments
Part Two

• Revised OFI comment
• Without information from customers of competitors
  leaders may be hindered in their ability to
  identify new products and services that may be
  critical in keeping current customers that have a
  choice in health care providers.
• Note This would be appropriate if the applicant
  identified keeping current customers who have a
  choice as a key success factor relative to
  competitors, a strategic advantage, or strategic
  challenge in their Organizational Profile.

31
Examining an Application

• Exercise 20 minutes
• Review the criteria, the applicants response,
  and your notes for Item 1.2.c(12)
• Identify one potential OFI
• Draft an OFI comment
• Use the standard two-part comment structure
• Underline the first part of the comment
• Circle the so what portion of the OFI
• Identify a speaker for the group

32
Process Examination

• Exercise 30 minutes
• Read Criteria Overall Requirements for Item 2.1
• Select 4 6 Key Factors
• Take notes on 2.1a(1)
• Draft 1 Strength comment
• Draft one OFI comment

33
Results Examination

• Examining the Results Items
• Similar to evaluating process Items except the
  scoring factors change. The examination scoring
  factors are
• Levels (Le)
• Trends (T)
• Comparisons (C)
• Integration (I)
• Projections, when appropriate

34
Example showing Level
Satisfied or Extremely Satisfied
Figure 7.1-1 Customer Satisfaction with Product
Quality
35
Example showing Levels with Segmentation
Satisfied or Extremely Satisfied
Figure 7.1-1 Customer Satisfaction with Product
Quality
36
Example showing Levels, Segments, and Trends
Satisfied or Extremely Satisfied
Figure 7.1-1 Customer Satisfaction with Product
Quality
37
Example showing Levels, Segments, Trends, and
Comparisons
Benchmark
Satisfied or Extremely Satisfied
Competitor
Industry Average
Figure 7.1-1 Customer Satisfaction with Product
Quality
38
How can this chart be improved?
Benchmark
Satisfied or Extremely Satisfied
Competitor
Industry Average
Figure 7.1-1 Customer Satisfaction with Product
Quality
39
It can be improved by adding projections that
link back to long range goals, strategies, etc.
identified in the Organizational Profile or Item
responses
Benchmark
Satisfied or Extremely Satisfied
Competitor
Industry Average
Figure 7.1-1 Customer Satisfaction with Product
Quality
40
Identifying Results - Category 7 notes
41
Category 7Example of Strength Comment

• Class Exercise
• What scoring aspects are addressed in this
  strength comment?
• 7.6a(3) Over the last three years, the
  applicant has increased the percentage of
  external board members from 25 to 60 (Figure
  6.7-1) and has appointed an external director as
  the head of its audit committee. These results
  are related to the applicants strategy of
  achieving greater independence in oversight of
  governance and financial audits.

42
Results Strength Comments

• Results Strength comments are similar in
  structure to Process Strength comments.
• There are two parts to a Results Strength
  comment.
• The first part references the scoring aspect that
  is the focus of the comment (Levels, trends,
  Comparisons, and/or Integration).
• The second part of the comment provides examples
  (evidence) from the application to support the
  finding.

43
Category 7Examination of Results

• The keys to successful examination of Items in
  Category 7 are
• Identifying Expected Results and Key factors
• Assessing performance levels in relation to
  scoring aspects
• What results do you expect to find in Category 7?
• For results shown, assess in terms of (Le, T, C,
  and I)

44
Category 7Results Comments

• Comment Structure
• Elements subject (applicant or result),
  timeframe (in, to/from), numbers (be specific),
  figure reference number (always cited), and so
  what statement (if applicable)
• All key areas of the Item responses should be
  reflected in the Item feedback comments
• Scoring Aspects all scoring aspects (Le, T, C,
  I) should be addressed in comments

45
Category 7Example of OFI Comment

• Exercise
• What scoring aspects are addressed in this OFI
  comment?
• Although overall satisfaction levels are shown,
  no segmented results are provided. The applicant
  identified inpatient, elderly women, and
  long-term care as key customer segments. Without
  results for these segments it may be difficult
  for leaders to assess progress in meeting the
  unique needs identified as important to
  satisfying these customers and meeting business
  objectives.

46
Category 7Examination of Results

• The keys to successful examination of Items in
  Category 7 are
• Identifying Expected Results and Key factors
• Assessing performance levels in relation to
  scoring aspects.
• What results do you expect to find in Category 7?
• For results shown, assess in terms of (Le, T, C,
  I, and projections)

47
Category 7Example of Strength Comment

• Exercise
• What scoring aspects are addressed in this
  strength comment?
• Overall satisfaction levels are consistently
  higher than VHA average in each of the three key
  customer requirements (access, courtesy, and
  quality of care) as shown in Figure 7.2-3. The
  Overall Customer Satisfaction Access levels are
  in the mid-90s for the past three years,
  consistently above VHA average (see Figure
  7.2-4). The Overall Customer Satisfaction
  Courtesy levels show an increase from 80 in 2003
  to 88 in 2005 (see Figure 7.2-5). The Overall
  Customer Satisfaction Quality of Care levels
  are sustained at high levels of satisfaction, 97
  each of the past three years (see Figure 7.2-6).
  This places the applicant in the top 10 VHA
  facilities each year for quality of care.

48
Category 7Example of OFI Comment

• Exercise
• What scoring aspects are addressed in this OFI
  comment?
• 7.1a While the applicant shows overall
  customer satisfaction with product quality for
  2002, no trend data are presented to allow it to
  track the rate of improvement (Figure 7.1-1). In
  addition, the data are not segmented by key
  customer group (i.e., industrial or commercial).
  Thus, it is not clear how the applicant
  identifies variation in its customer satisfaction
  over time or from sector to sector to target
  opportunities for improvement.

49
Category 7Example of OFI Comment

• Exercise
• What scoring aspects are addressed in this OFI
  comment?
• Although overall satisfaction levels are shown,
  no segmented results are provided. The applicant
  identified inpatient, elderly women, and
  long-term care as key customer segments. Without
  results for these segments it may be difficult
  for leaders to assess progress in meeting the
  unique needs identified as important to
  satisfying these customers and meeting business
  objectives.

50
Results Comments

• Exercise 20 minutes
• Review Overall Requirements for Item 7.6
• Identify Expected Results
• Identify key Factors
• Evaluate information provided using Le, T, C, and
  I
• Consider projections when appropriate
• Draft one Strength comment
• Draft one OFI comment

51
Scoring Items

• Scoring can easy or difficult
• If you have well-written comments that cover the
  Item criteria and areas most important to the
  applicant, then scoring is easy.
• If comments are general focused on a narrow
  portion of the Item Criteria concentrated on one
  or two examination aspects and not aligned with
  areas of importance to the applicant then scoring
  is difficult.

52
Scoring Process

• Individual
• Review all comments (Strengths and Opportunities
  for Improvement) associated with an Item
• Turn to the appropriate set of scoring guidelines
  in the Baldrige criteria book or use scoring
  sheets
• Begin with Approach (A) at the 50 65 score
  range
• Read the score range description

53
Scoring Process

• Individual
• Is this description consistent with comments in
  your analysis of the Item?
• If yes, read the range above and below to ensure
  you are comfortable with the range selected.
  Indicate if the applicant is in the high, middle,
  or low end of the selected range
• If no, move up or down until you read a range
  that is consistent with your Item comments.
  Then, read the range above and below to ensure
  you are comfortable with the range selected.
  Indicate if applicant is in the high, middle, or
  low end of the selected range

54
Scoring Process

• Individual
• Repeat this process for each of the scoring
  aspects (A, D, L, I for Process Items and Le, T,
  C, I or Results Items)
• Once score ranges have been selected for all four
  scoring aspects, select an overall score range
  that you believe represents the applicants
  demonstrated performance in this area.
• Review comments and scores.
• Select a percent score (score in increments of
  5). For example 30, 40, or 45.

55
Scoring Process

• Individual
• Repeat this process for each Item
• Team
• Place Item level scores in a table displaying the
  scores of everyone on the team.
• Calculate/estimate the average score for the
  Item.
• If individual scores are within plus or minus 10
  of the average, average all scores and move on to
  the next Item.

56
Scoring Process

• If scores vary by more than 10 from the average
  score the team needs to discuss, understand, and
  reach consensus
• Have the person with high score present facts
  supporting the higher score (linked to the four
  scoring aspects)
• Have the person with the low score present facts
  supporting the lower score (linked to the four
  scoring aspects)
• The entire team listens, considers, and rescores

57
Scoring Process

• Things to consider
• 10 averaged with 4 other people on the team
  amounts to about a 2 change in score
• 2 of an Item is about 1.5 points
• There are a total of 1000 possible points
• How long do you want to spend trying to reach
  consensus on 2 points?
• Carey specific procedure
• If individual scores are more than 15 apart for
  any individual Item, that Item will be flagged
  for the judge. A detailed review of the
  applicant responses, team comments, and score
  will be conducted and the judge will determine
  what changes are needed

58
Scoring

• Exercise Part I 15 minutes
• Case study comments for Item 1.1
• Review all comments for Item 1.1
• Individually score using process discussed
• Share individual scores with people at your table
• Record individual scores
• Discuss
• Rescore
• Record consensus score

59
Scoring

• Exercise Part II 15 minutes
• Case study comments for Item 7.6
• Review all comments for Item 7.6
• Individually score using process discussed
• Share individual scores with people at your table
• Record individual scores
• Discuss
• Rescore
• Record consensus score

60
Carey Examination Process Flow

• Examiners
• Selected and trained
• Assigned to teams (4-7 on a team)
• Receive application, conflict of interest form,
  reporting instructions in the mail
• 2 3 days of independent work prior to team work
  activities face-to-face
• Read application
• Key Factors Worksheet
• Notes in Application
• Draft comments for assigned Items

61
Carey Examination Process Flow

• 3.5 days in Washington area for face-to-face
  meeting
• Examine application - consensus on comments and
  scores
• Produce a draft feedback report
• Report to Carey process oversight group (program
  manager and representative from each of the three
  administrations)
• Oversight group reviews all work and decides
• Which applications are recommended for site visit
• Which applications are not recommended for site
  visit
• Which teams may have had an issue during the week
• Which teams did an unacceptable job and require
  another examination
• EVERY MEMBER IS EMPOWERED TO FLAG AN APPLICATION
  FOR ANOTHER REVIEW IF NEEDED

62
Carey Examination Process Flow

• All applications and team analyses sent to an
  objective, external, Baldrige expert, serving as
  a Carey judge. Judges review the feedback report
  and scores. Judges then calibrate scores and
  comments to ensure a fair and objective
  assessment of each applicant
• Site visit conducted at facilities that have the
  potential of being recommended for the trophy
  award
• All applicants scoring over 330 are evaluated to
  ensure that there are no issues that might
  otherwise exclude them from being recognized by
  the Secretary. This could eliminate a fair
  number of potential award recipients
• Feedback reports prepared and sent to applicant
  organizations

63
Examining Process Flow

• A list of those remaining applicants with scores
  over 330 is sent to the Secretary with
  performance (score information), a summary of
  applicant best practices, and general
  recommendations (330 430 Achievement, 431 and
  over Excellence or Trophy).
• Recommendations are considered, then the
  Secretary selects Achievement, Excellence, and
  Trophy winners
• Symposium and Awards Ceremony in fall

64
THANK YOU

• Have nice evening and a safe trip home