Training Evaluation Report The Flexible Thinker Program

1
Training Evaluation Report -The Flexible
Thinker Program

• Dr. Dusya Vera
• Richard Ivey School of Business
• The University of Western Ontario

2
Kirkpatricks Levels of Training Evaluation
LEVELS EVALUATED IN THIS STUDY
Level 4
Results Did it make a difference?
Level 3
Behavior Did they use it?
Level 2
Learning Did they learn it?
Level 1
Reaction Did they like it?

• Each level of evaluation builds on the previous
  one. If people like the training, they are likely
  to learn it. If people learn it, they are likely
  to use it on the job. If people use it on the
  job, they are likely to achieve results.

3
First Level of Evaluation Did they like it?
Very satisfied
Very dissatisfied

• 156 people were trained.
• A frequent comment from participants was that
  although I was told to come to this course and
  didnt know what to expect, I have enjoyed it and
  I think it
• is important for my job
• The conclusion from this level
• of evaluation is that participants
• liked the training.

Comparison to other courses
Recommend to others?
Much worse
Much better
No
Definitely
4
Second Level of Evaluation Did they learn it?

• At the start and the end of each workshop,
  participants completed a Flexible Thinker Test,
  where they were exposed to a problem and had 3
  minutes to generate ideas on how to solve it.
• The number of ideas generated at the end of the
  workshop was significantly higher than the number
  of ideas generated at the start of the workshop.
• The conclusion from this level of evaluation is
  that during the workshop participants acquired
  new knowledge about Flexible Thinking tools
  that allowed them to be more creative under
  pressure.

5
Third Level of Evaluation Did they use it on
the job?
Large change for the better
No change
Large change for the worse

• Six weeks after the training, participants
  received a Feedback form with questions about
  their experience using The Flexible Thinker on
  the job. 96 forms were returned (61).
• The conclusion from this level of evaluation is
  that participants perceived a positive change in
  themselves and in their teams. Participants
  observed a larger change at the individual level
  than they did at the team level.

6
Fourth Level of Evaluation Did it make a
difference?

• I.T. systems directly affect roughly 1,100 city
  employees.
• Average wage at the city is roughly 17 an hour.
• It is impossible to say precisely the level of
  productivity would be lost in the case of a total
  system failure and how much preventative
  maintenance can prevent that. However, not doing
  preventative maintenance significantly increases
  the likelihood of a system failure and
  significant loss in productivity.

Average Hourly Manpower costs (in 10K)
7
Some of the answers to the open-ended questions
what changes have you noticed in yourself /
your team?
YOURSELF
YOUR TEAM

• I am more open-minded / I look for more options
  / I look at things from different perspectives
  (35 people)
• I have not noticed any changes / very little
  changes (14 people)
• I am calmer in situations of decision making (7
  people)
• I am more receptive to ideas from others / I use
  yes-and-because (7 people)
• I have a better approach to problems / better
  assessment of situation (4 people)
• I am more positive (4 people)

• I have not noticed any changes / very little
  changes (28 people)
• We are more open-minded / we look for more
  options / we look at things from different
  perspectives (14 people)
• More positive attitude in the team (11 people)
• Not everybody in the team was trained (8
  people)
• We work together more / we work towards the same
  goals (5 people)
• Some in the team have changed, others not (5
  people)

8
Some examples of ocassions when training was used
and suggestions to make the program more helpful
EXAMPLES
SUGGESTIONS

• yes because dealing with customers yes
  because in the boardroom (5 people)
• used tools in project to create standard
  operating procedures (2 people)
• hats at a meeting made members more relaxed
  receptive to new ideas
• we change the way we respond to the public
  (concern with roadway conditions)
• productivity staff tries new things customer
  satisfaction the party we accomodated in short
  notice
• use library partnerning in developing new
  processes

• All employees, management council members
  should attend training (15 people)
• Refresher courses / refresher courses with team
  members / type of follow up / emails to keep
  things in mind (8 people)
• All mgt. to take the course / training efforts
  need senior managers buy in / include
  supervisors (7 people)
• Training great the way it is (4 people)
• Communicate with other trained members, identify
  how people use the skills and benefit from
  training (3 people)
• Teams that work together should train together
  (3 people)

9
Barriers to using the skills
How eager were you to change your behavior?
Have you been able to apply the skills in your
job?

• Participants were asked about barriers to using
  the skills. No barrier achieved an overall high
  level.
• When asked about other barriers to using the
  skills, participants mentioned factors such as
  high workload / lack of time and have not
  encountered situation yet

Not Eager
Very Eager
Not at all
To a large extent
To some extent
Eager
Strongly agree
Strongly disagree
10
Possible reasons for a greater change in behavior
at the individual level than at the team level

• Although efforts were pursued to train all
  members of the team, including its supervisor,
  this objective could not be achieved for all of
  the teams. If not all team members and the
  supervisor are trained, it is hard to achieve
  high levels of change at the team level.
• Teams were not trained together in order to not
  disrupt the Citys operations. However, training
  a team is more than individually training the
  members of a team. There are synergistic benefits
  of training at team together.
• Some teams that went through the training may not
  be a team, but a group. That is, while team
  members need to work in a coordinated and
  mutually dependent way, group members perform
  similar jobs in a pretty independent way. Group
  members may not report a high level of change at
  the team level if they do not need to work in
  coordination with others.
• Some teams may have had high levels of
  responsiveness and creativity before the
  training, thus, couldnt report high levels of
  change achieved.
• Possible existence of a I am OK, but the rest of
  the team arent syndrome.

11
Evaluation through a Control Group Perceptions
of Creativity and Responsiveness/Spontaneity
Strongly agree
Strongly disagree

• A survey was developed to capture employees
  perceptions about their day-to-day job. The suvey
  didnt make any reference to the Flexible
  Thinker training. The first survey was completed
  before the training started. The second survey
  was completed two months after the training
  finished. 175 surveys were sent to the training
  teams. 164 people answered the first survey
  (94), 100 answered the second survey (57), and
  95 answered both (54). 173 surveys were sent to
  the control teams. 99 people answered the first
  survey (57), 83 answered the second survey (48)
  and 72 answered both (42).
• All things being equal, the training group
  experienced a significant positive change in
  individual creativity and responsiveness/spontanei
  ty, while the control group didnt. There was no
  significant change in team creativity and team
  responsiveness/spontaneity neither in the
  training group nor in the control group.

12
Evaluation through a Control GroupPerceptions
of Team Performance
Values to think about
Strongly agree
Strongly disagree

• Before the training, the training and control
  groups were not homogeneous in terms of
  performance. The control group had a higher
  perception of its performance than the training
  group had.
• Performance variables didnt experience a
  significant change in the period of study neither
  in the training group nor in the control group.
• To think about While teams perceive their
  customer satisfaction and productivity levels to
  be around 5, they perceive their innovation and
  employee satisfaction levels to be around 4.

13
Assessment of Contextual Variables
Values to think about
Strongly agree
Strongly disagree

• Real-time information communication were low
  across teams. Participants do not feel updated
  about what is happening in other teams and in the
  organization. This is an obstacle to creativity,
  since in creative processes real-time information
  plays a coordinating role and permits groups to
  get feedback on their actions.
• The level of experimentation in the teams
  culture was low across teams. Participants do not
  feel that the firms culture supports competent
  mistakes when trying new approaches to things.
  This is an obstacle to creativity, since in
  creative processes experimentationa and errors
  are indispensable.
• Employees perceive medium levels of change and
  turbulence in the environment. This is an
  obstacle to creativity, since the perception of
  external change helps to motivate teams to be
  more creative at work.

14
Explanation of Fourth Level Approach

• The focus group choosen was information
  techology.
• A specific challenge was decided within the group
  that had thus far and could have major
  consequences for the entire organization.
• Because Brampton is groing so quickly, the i.t.
  infrastructure has not been able to keep up and
  much of the technology is becoming outdated.
  Also, the system was originally designed as a 9
  to 5 system and the demands have become 24-7.

15
The 4th Level Approach

• All members of I.T. and pilot program
  participants were required to attend a 2 day
  workshop
• A plan for the launch and how to proceed in
  solving our challenge was then developed
• A walkabout was organized to talk to participants
  give them a full information package. This
  information package included information on what
  we were doing and why as well as a review of the
  course, was given to all participants.

16
The 4th Level Approach

• All participants were e-mailed the final
  challenge wording, based on the 15 word or less
  exercise that they filled out. The final wording
  on the challenge was How can systems
  maintenance be executed with minimum service
  disruption? This was done to allow participants
  a chance for people to give feedback to the issue
  we were facing and to ensure that everybody was
  on-board.
• People were divided into one of teams (Teams
  Spectacular, Genius, Creative, Flexible,
  Brilliant and Innovative).

17
Level 4 Solution

• A person was put given full ownership over all
  maintenance.
• Formed teams of end users and i.t. personnel to
  discuss critical windows and defining systems
  that those are done on.

18
Level 4 Solution

• Created a matrix in consultation with both end
  users and i.t. personnel which was presented to
  the organization for further feedback.
• Using theme days (i.e. emergency preparedness
  day, meeting day, etc.) to tie-in maintenance
  to organizational initiatives and therefore limit
  lost productivity when/if maintenance needs to be
  performed during peak user hours.

19
Conclusions

• Participants were highly satisfied by the
  Flexible Thinker workshop and learned
  creative-thinking tools during it (slides 3 and
  4).
• At the individual level, participants on-the-job
  behavior has been positively impacted by the
  workshop. Participants perceive a positive change
  in their ability to be creative under pressure
  and responsive to new situations (slides 5
  through 8, and 11).
• The Flexible Thinker made a measureable impact on
  the bottom line (slide 6 and slides 14 through
  18) and solved a problem that could have had
  far-reaching impact on the organization.

20
Conclusions

• The impact of the workshop at the team level has
  been smaller than the impact achieved at the
  individual level. The reasons for this were
  discussed in slide 11.
• As part of the study, employees perceptions of
  their work environment were collected. Some
  variables that may represent obstacles to
  creativity and responsiveness have been detected
  in slides 12 and 13.