Examining Small Facilities

1
Examining Small Facilities

• What to include in an application
• How to evaluate as an Examiner
• Joe Muzikowski
• joe.muzikowski_at_gmail.com

2
A Simplified Look at the Carey Criteria
3
The Assessment Process
A well-written application and supporting
documentation results in an informative
feedback report
Use your feedback for improvement!
4
Small is in the Eyes of the Beholder
500 People
Same evaluation Criteria, but different emphasis,
based on the Key Factors
250 People
100 People
50 People
The smaller the organization, the more informal
the approach
25 People
5
In a small organization, size may be the most
critical Key Factor

• Size affects
• Resource Availability
• Organizational Reach
• Level of Formality

6
Considerations for Small OrganizationsLimited
Reach
Implications for Applicants
Implications for Examiners

• Must consider the scope of the organizations
  mission

• Often not able to demonstrate impact beyond a
  small customer base and geographic area

• Remember the relative size and geographic scope
  of the organization

• Usually not able to demonstrate leverage with
  suppliers and other stakeholders

• Take into account relative customer/market/workfo
  rce complexity and the facility's ability and
  need to slice and dice data

• Segmentation may be difficult to analyze or not
  even necessary

7
Considerations for Small OrganizationsLimited
Resources
Implications for Applicants
Implications for Examiners

• Must adapt to more limited information provided
  for the assessment (but still expect responses
  for all applicable sections of the Criteria)
  expect to review a lot of information on-site

• Difficult to devote people and time to write a
  full application (but applicant still must be
  able to demonstrate what it does against all
  applicable sections of the Criteria)

• Comparisons are still expected, but may be from
  within VA or from local or free external
  sources

• Cost may prohibit using some benchmark sources

8
Considerations for Small OrganizationsLimited
Resources
Implications for Applicants
Implications for Examiners

• While these approaches in themselves may be
  systematic, Examiners must ensure they are
  deployed and used effectively by the applicant.
  Examiners also may check for how improvement
  opportunities are identified

• Approaches and services may be provided by the
  parent organization

• Leveraging all available resources is a critical
  issue for a small facility

• Must check how volunteers, suppliers and
  partners are involved

9
Considerations for Small OrganizationsInformal
Approaches
Implications for Applicants
Implications for Examiners

• Expect to see little formal documentation of
  procedures

• Approaches may rely heavily on verbal
  communication, rather than written documentation

• Still must ensure approaches are repeatable, use
  data, and drive results

• Must check deployment as fully as possible

• Data management, documentation and IT support
  may be limited

• Data may be managed by individuals via PCs and
  spreadsheets

10
Informal Can Still Be Systematic

• Informal
• Approaches may be verbal
• Approaches may not be documented
• No records might exist
• Verification may be done through interviews of
  participants

• Systematic
• Is repeatable
• Uses data and information
• Can be reviewed and improved
• Regular, two-way communication therefore can be
  systematic

11
Considerations for Small OrganizationsWhats
Expected
Implications for Applicants
Implications for Examiners

• Keep asking how do you to get beyond the
  tendency for the applicant to answer with what
  or provide examples

• Despite size and informality, you still need to
  demonstrate effective approaches in all six
  process Categories of the Criteria

• Still need to demonstrate results that derive
  from your approaches

• Seek to understand why the organization measures
  what it measures, and why the results are what
  they are

Always be aware that there are huge differences
between facilities with 1,000, 100 and 20
people. In small organizations, people wear many
hats.
12
The Organizational Profile

• Whether Youre an Applicant or an Examiner, It
  All Starts Here

13
The Organizational Profile (OP)

• What is it?
• A 5-page description of the basic facts and
  figures that makes your organization unique
• How is it used?
• Examiners use it as a Rosetta Stone to
  translate the generic Criteria into a meaningful
  evaluation of an applicant, taking into account
  its unique aspects and specific important factors
• They develop a short, bullet point list of Key
  Factors and build their feedback comments and
  scoring based on the applications response to
  the Criteria questions, using the Key Factors as
  a filter to ensure significance and relevance of
  their comments and accuracy of their scoring

14
Organizational Profile
P.2 Organizational Challenges

• Competitive Environment
• Competitive position
• Principal success factors
• Key changes taking place
• Sources of competitive/comparative data
• Strategic Context
• Strategic challenges and advantages
• Performance Improvement System

• Organizational Environment
• Main products / services
• Mission, vision, values
• Workforce profile and requirements
• Facilities
• Legal/regulatory requirements
• Organizational Relationships
• Structure/governance
• Stakeholders and their requirements
• Role of suppliers and partners
• Relationship/communication mechanisms

Regardless of size, any organization should be
able to develop an Organizational Profile
15
Organizational Description Examples
P.1a(3) What is your workforce profile?
From Mesa Industries, a 2006 Baldrige Recipient
75 employees
P.1a(5) ..What are the applicable occupational
health and safety regulations?
From Baptist Health, a 2003 Baldrige Recipient
16
Organizational Challenges Examples
P.2b What are your strategic
challenges?
From Boeing Aerospace Support, a 2003 Baldrige
Recipient
P.2a(3) What are your available sources of
comparative and competitive data?
From Baptist Health, a 2003 Baldrige Recipient
17
Understand How and What

• The Criteria are a series of two types of
  questions
• How questions ask for descriptions of
  systematic approaches
• Asked in Categories 1 6
• What questions ask for
• A list of specific details (in the Organizational
  Profile and Categories 1 6)
• See the Organizational Profile also 1.1b(2)
  1.2b(1) 1.2b(2) 1.2c 2.1a(1) 2.1b(1)
  2.2a(4) 2.2a(5) 2.2a(6) 2.2b 3.2a(2)
  4.1a(1) 4.1b(1) 5.1c(1) 5.2b(1) 6.1a(1)
  6.1b(1) 6.1b(2) 6.2a(1)
• Performance results (in Category 7)

18
How Questions

• How questions are looking for detailed
  descriptions of the processes you use. Respond
  with either
• A narrative
• A graphic, or
• A flowchart
• Example 3.1.a(1) - How do you identify
  patients, other customers, customer groups, and
  health care market segments?

19
What Questions

• What questions are looking for specific
  information, without many supporting details.
• Respond with
• A bullet point list, or
• A table
• A figure
• Examples
• 1.1b(2) What performance measures do senior
  leaders regularly review to inform them on needed
  actions?
• 7.1a(1) What are your current levels and trends
  in key measures or indicators of product and
  service performance?

20
Process Descriptions

• A process can be described like this
• Or like this
• Step A
• Step B
• Step N

21
Applicants Should Tell Your "STORIE"

• S Describe the steps in the process
• T Describe the timing of steps or activities
• O Mention who the process owner is
• R Describe the related results (A cross-table
  of process approaches to results in Category 7
  might be a useful preparation tool)
• I Describe the inputs to the process
• E Describe how you make sure the process is
  effective and how you review and improve it

22
Criteria Hierarchy for Small Organizations
7 Categories, (e.g., 1.0 Leadership)
sub-divided into
Respond at this level of detail whenever
possible
31 Items (e.g. 1.1a Vision and Values
23
Process Evaluation Factors ? A-D-L-I

• Approach The process (method or methods) used
  by the organization to address the Criteria
  requirements
• Deployment The extent to which the
  organizations process is followed in all
  appropriate work units, and used consistently
• Learning How the organization evaluates and
  improves its processes how the organization
  learns
• Integration How the individual approaches of
  the system operate as a fully interconnected unit

24
Approach What the Criteria Look For

• Is there a process? (Not an anecdote)
• Is it appropriate to the Item requirements?
• Is it systematic (i.e., repeatable, managed using
  data)?
• NOTE A process can be informal, but may still
  be systematic
• Does it appear to be effective?
• Is the approach a key organizational process
  (i.e., important to the organizations overall
  performance)?

25
Deployment What the Criteria Look For

• Is it used by all appropriate personnel, work
  units, departments as needed?
• Is it used consistently?
• Small organization should have an advantage over
  larger organizations with regard to deployment

26
Learning What the Criteria Look For

• Has the approach been evaluated and improved?
• In a systematic manner?
• Is there evidence of breakthrough change, or
  innovation?
• Is there evidence that learning has been shared
  with other personnel, departments and/or
  partners, as appropriate?
• Is there evidence that learning is embedded in
  how the organization operates?

27
Integration What the Criteria Look For

• How well is the approach aligned with
  organizational needs (i.e., those identified in
  the Organizational Profile and Process Items)?
• Are the measures, information, and improvement
  systems complementary across processes and work
  units?
• How well does the approach align and integrate
  with other approaches to support achievement of
  organizational goals?

28
1 - Leadership

• How do your organizations senior leaders
• Set and deploy its organizational vision and
  values?
• Foster and require legal and ethical behavior?
• Create an environment for performance
  improvement, accomplishment of strategic
  objectives and learning?
• Communicate with and motivate the entire
  workforce?
• Create a focus on action?
• Address accountability for managements actions?
• Address any adverse impacts of your services and
  operations?
• Promote and ensure ethical behavior in all your
  interactions?
• Actively support and strengthen your key
  communities?

29
Mesa Industries Leadership System
2006 Baldrige Recipient 75 employees
30
Mesas Governance Approaches
2006 Baldrige Recipient 75 employees
31
Mesas Community Support Process
2006 Baldrige Recipient 75 employees
32
2 - Strategic Planning

• How does your organization
• Conduct its strategic planning?
• Collect and use data and information for your
  strategic planning process?
• Address the challenges identified in response to
  P.2 in your Organizational Profile?
• Develop and deploy action plans to achieve your
  strategic objectives?
• Allocate resources to ensure accomplishment of
  your action plans?

33
Branch Smiths Strategic Planning Process
2002 Baldrige Recipient 68 employees
34
Mesas Planning Process
2006 Baldrige Recipient 75 employees
35
SSMs Response in Item 2.2
2002 Baldrige Recipient
2.2a(1) How do you develop and deploy action
plans throughout the organization to achieve your
key strategic objectives?
Entity Level
Patient Loyalty
Drivers (or KCRs)
Pain Mgmt.
Response
Department Level
Call light response
Meds. Del. W/in X min. of order
Pain assessed per pain scale
Individual Level
I will respond in X min.
I will assess 1X/shift/pt
I will respond in X minutes
I will ask if there is anything else 100 of
time
36
3 Patient, Other Customer and Market Focus

• How does your organization
• Listen and learn to determine customer
  requirements and changing expectations?
• Use relevant information from current and former
  customers?
• Build relationships to acquire customers and
  increase loyalty and repeat business?
• Enable customers to seek information, conduct
  business, and make complaints?
• Manage customer complaints?
• Determine customer satisfaction, dissatisfaction
  and loyalty?
• Use satisfaction and dissatisfaction information
  for improvement?

37
Mesas Complaint Process
2006 Baldrige Recipient 75 employees
38
Mesas Customer Satisfaction Process
2006 Baldrige Recipient 75 employees
39
4 - Measurement, Analysis and Knowledge
Management

• How does your organization
• Select and collect data and information for
  tracking daily operations and overall
  organizational performance?
• Review organizational performance and
  capabilities?
• Make needed data and information available to
  your workforce, suppliers, partners,
  collaborators and customers, as appropriate?
• Ensure your data, information and organizational
  knowledge are accurate, reliable, timely and
  secure?
• Manage organizational knowledge?

40
PRO-TECs Communication Approaches
2007 Baldrige Recipient 231 employees
41
Chugachs Response for Item 4.1
4.1b(1) How do you review organizational
performance?
2002 Baldrige Recipient 20 employees
42
PRO-TECs Data Quality Approaches
2007 Baldrige Recipient 231 employees
43
Texas Nameplates On-Line Dashboard
2004 Baldrige Recipient 43 employees
44
5 - Workforce Focus

• How does your organization
• Determine the factors that affect workforce
  engagement and satisfaction?
• Foster an organizational culture conducive to
  high performance and a motivated workforce?
• Support high-performance work and workforce
  engagement?
• Address the needs and desires for learning and
  development identified by your workforce?
• Assess workforce engagement?
• Ensure and improve workplace health, safety and
  security?
• Support your workforce via policies, services,
  and benefits?

45
Chugachs Response for Item 5.2
5.2b(1) How do you ensure and improve workplace
health, safety and security? What are your
performance measures and improvement goals?
2002 Baldrige Recipient
46
Stoners Training Approaches
2003 Baldrige Recipient 48 employees
47
6 - Process Management

• How does your organization
• Determine its core competencies?
• Decide which processes will be internal to your
  organization and which will use external
  resources?
• Determine key work process requirements?
• Design your work processes to meet all the key
  requirements?
• Ensure workforce and workplace preparedness for
  disasters or emergencies?
• Implement your work processes to ensure that they
  meet design requirements?
• Improve your work processes to achieve better
  performance?

48
A Visual Workplace is Possible Even in Small
Organizations
Pals Sudden Service, a 2001 Baldrige Recipient
49
Sharps Process Management Approach
2007 Baldrige Recipient
50
Stoners Production Process
2003 Baldrige Recipient 48 employees
51
Branch Smiths Key Processes
2002 Baldrige Recipient 68 employees
52
Chugachs Support Processes
2002 Baldrige Recipient 20 employees
53
Mesas Process Improvement Approach
2006 Baldrige Recipient 75 employees
54
7 - Results

• How does your organization
• Perform (i.e., current levels and trends) in the
  following areas
• Product and service results?
• Patient/Customer satisfaction?
• Financial and marketplace performance?
• Workforce engagement and workforce environment?
• Process effectiveness results?
• Leadership and social responsibility results?

• NOTE Where possible, provide results against
  appropriate external comparisons, either within
  or external to VA
• Link results to Key Factors (from the OP),
  important processes, and action plan requirements

55
Results Items Point Values
Remember Results are a mere 45 of the total
point value!
56
Results Descriptions

• Results may be presented in
• Graphic format
• Tabular format
• Text, or
• A combination of these

Our financial performance has demonstrated
continued improvement over the past three
years
57
Results Descriptions
58
Results Evaluation Factors ? Le-T-C-I

• Levels Numerical data that place an
  organizations performance on a meaningful
  measurement scale
• Trends Numerical data that show the direction
  and rate of improvements
• Comparisons Numerical data that show the
  relationship of an organizations performance
  relative to others (or the performance of units
  within an organization to one another)
• Integration Connections to important customer,
  product/service, process, or action plan
  performance requirements
• and Segmentation as appropriate

59
Levels What the Criteria Look For

• Are results provided for all key measures
  mentioned in the OP and Process Categories? For
  example, note linkages to
• P.1a(5) Performance against regulatory
  standards (report results in Item 7.6)
• 1.2b(1) What are your key compliance processes
  measures and goals? 1.2b(2) What are
  yourmeasuresformonitoring ethical behavior?
  (report results in Item 7.6)
• 2.2a(6) What are your key performance
  measuresfor tracking progress on your action
  plans?
• 3.2b(1) and (3) How do you determine customer
  satisfaction and dissatisfaction? (report
  results in 7.2)
• 4.1a(1) What are your key organizational
  performance measures?
• 5.1c(1) How do you assessworkforce engagement
  and workforce satisfaction? (report results in
  Item 7.4)
• 5.2b(1) What are yourmeasures for (workplace
  health, safety and security)? (report results
  in Item 7.5)
• 6.2a(1) What are your key performance
  measuresused for the control and improvement of
  your work processes? (report results in Item 7.5)

60
Results - Current Levels
Agreed or Strongly Agreed
Employee survey question I am satisfied with my
job.
61
Results - Current Levels, Key Segments
Good
Agreed or Strongly Agreed
Employee survey question I am satisfied with my
job.
Data also could be segmented by type of response
(agree strongly agree, etc.)
62
Trends What the Criteria Look For

• Are results tracked over time? Is performance
  consistent?
• If there is no trend, is performance sustained at
  a high level?
• Do the trends make sense?
• Has the applicant explained what accounts for the
  trends (whether favorable or unfavorable) and for
  specific negative occurrences?
• Whats the direction of goodness?
• NOTE To be considered a valid trend, at least
  three data points are required
• If you do not have three years worth of data,
  report data by month or quarter to demonstrate a
  trend

63
Results - Trends
Good
Agreed or Strongly Agreed
Employee survey question I am satisfied with my
job.
64
Comparisons What the Criteria Look For

• Do the comparisons come from recognized sources?
• Are the comparisons presented for high
  performance levels or just averages?
• Are comparisons provided for a few, some, many or
  most measures?
• Are comparisons provided for the most important
  results?
• Are the comparisons appropriate to the
  organizations needs?
• Are the comparisons consistent across related
  measures?
• Against average, top quartile or top decile?

65
Results - Comparisons
Good
Benchmark
Main Competitor
Agreed or Strongly Agreed
Industry Trend
Employee survey question I am satisfied with my
job.
66
Integration What the Criteria Look For

• Do the results link to responses in the OP and
  Process Item Criteria?
• To what extent are results provided to show
  performance in areas of importance?
• Do the results include appropriate segmentation?
  If not, is it clear why?

67
Bronsons Response in Item 7.1
2005 Baldrige Recipient
68
Stoners Customer Satisfaction Results
2003 Baldrige Recipient 48 employees
69
Texas Nameplates Results
Note the benchmark source
2004 Baldrige Recipient 43 employees
70
Bronsons Response to Item 7.6
2005 Baldrige Recipient
71
Linking Results and Processes
72
Application Development

• Telling Your Story as Effectively as Possible
• Interpreting the Application for a Small Facility

73
The First Rule of Application Writing
74
Writing a Carey Application in 3 Easy Steps

• Read the question
• Answer the question
• Repeat steps 1 and 2 until youve answered all of
  the questions

75
A Classic Mistake The Seven Buckets
76
How to Get Useful Feedback

• Do
• Understand what the Criteria ask for
• Understand the evaluation factors
• Be truthful
• Be accurate
• Be as detailed as possible (keeping in mind the
  page limitation)
• Tell your story with pride
• Keep in mind youre writing a story with 7
  chapters

• Dont
• Forget the importance of the Organizational
  Profile
• Stray far from the Criteria questions
• Embellish or exaggerate
• Forget youre also writing for non-sector people
• Write empty prose to fill up space
• Forget linkages
• Write 7 short stories

77
What Examiners Should Expect in a Small
Application

• Do Expect
• Responses to at least the overall Criteria
  requirements
• A clear indication of who does what
• Many approaches and services come from the parent
  organization
• Some consistency in approaches with similar VA
  organizations

• Dont Expect
• Detailed descriptions of processes and approaches
• Extensive evidence of deployment and integration
  (its taken for granted in many small
  organizations)
• A full 50-page document

78
Resources

• Carey website http//www1.va.gov/op3/page.cfm?pg1
  1
• Carey Application and New Examiner Course
• Planning to Prepare a Carey Application
• Carey Application template
• Carey Application Writing class
• Baldrige web site http//www.quality.nist.gov
• Award recipient contacts and application
  summaries
• Case study materials and other publications
• Your own organizations communications
• Key facts and figures about your organization

79
Examining Small Facilities

• What to include in an application
• How to evaluate as an Examiner
• Joe Muzikowski
• joe.muzikowski_at_gmail.com