Presentation to Alberta Innovation and Science

1
Presentation to Alberta Innovation and Science

• Complete Results of Flexible Thinker Program to
  Help Foster an Environment of Innovation

2
Part 1

• Summary of Individual, Team and Organization
  Effects of Flexible Thinker Program for Alberta
  Innovation and Science

3
Kirkpatricks Levels of Training Evaluation
LEVELS EVALUATED IN THIS STUDY
Level 4
Results Did it make a difference?
Level 3
Behavior Did they use it?
Level 2
Learning Did they learn it?
Level 1
Reaction Did they like it?

• Each level of evaluation builds on the previous
  one. If people like the training, they are likely
  to learn it. If people learn it, they are likely
  to use it on the job. If people use it on the
  job, they are likely to achieve results.

4
First Level of Evaluation Did they like it?
Very satisfied
Very dissatisfied

• 79 people were trained.
• A frequent comment from participants was that
  Excellent tools that were learned in one day and
  simply to apply.
• The conclusion from this level
• of evaluation is that participants
• liked the training.

Comparison to other courses
Recommend to others?
Much worse
Much better
No
Definitely
5
Second Level of Evaluation Did they learn it?
Average per person

• At the start, participants completed a Flexible
  Thinker Test, where they were exposed to a
  problem and had 3 minutes to generate ideas on
  how to solve it. At the end they were given the
  exact same test and told to come up with new and
  different answers than they thought of before.
• The number of ideas generated at the end of the
  workshop was significantly higher than the number
  of ideas generated at the start of the workshop.
• The conclusion from this level of evaluation is
  that during the workshop participants acquired
  new knowledge about Flexible Thinking tools
  that allowed them to be more creative under
  pressure.

6
Second Level of Evaluation Did they learn it?

• At the beginning of the workshop, participants
  were asked to identify all of the obstacles that
  they face in building the perfect workplace.
  These obstacles included external obstacles such
  as time, money, etc. and internal obstacles such
  as preconceived ideas, false ego, complacency,
  negativity and fear.
• They were then told to apply the tools to come up
  with specific ideas that they could apply the
  next day to help overcome them.
• When they had at least 1 option to overcome that
  obstacle, they were told to check it off.
• The conclusion from this level of evaluation is
  that during the workshop participants were able
  to apply the Flexible Thinking tools to the
  real issues that they face.

7
Third Level of Evaluation Did they use it on
the job?
Ind. Questions
Team Questions

• 1 week prior to training, an online survey was
  conducted asking the pariticpants a series of
  questions regarding both individual and
  team/organizational performance.
• 37 participants filled out the Before survey
  and 32 completed the After. This represented
  41 and 36 of the number of course participants.
• The survey represented a series of statements
  that participants were given and asked to rate on
  a 1 to 5 scale with 1 being STRONGLY DISAGREE to
  5 STRONGLY AGREE.
• The questions contained both positive (where a
  move toward 5 represents positive change) and
  negative (where 1 represents positive change)
  questions.
• The conclusion from this level of evaluation is
  that participants perceived a positive change in
  themselves and in their teams.

8
Fourth Level of Evaluation Did it affect the
organization?

• This was assessed through the use of open ended
  questions such as Please describe a situation
  where you applied the Flexible Thinker tools and
  what was the result? and What have been your
  accomplishments over the past 3 months?
• Some accomplishments include ability to think
  much more clearly and results were more
  productive addressed the concerns of a non-union
  employee into a union position in a way that
  there was buy-in engaging a person in a meeting
  who was being cut-off and using their ideas to
  find a solution to a problem and designing a way
  to introduce a new approach to doing our business
  increasing interest.
• The average of people who agreed with the
  statement that they have been able to use the
  Flexible Thinker concepts at work was 3.74. If
  averaging is applied, the level is higher as
  there was 1 2.
• The conclusion from this level of evaluation is
  that the program did have a positive affect on
  the organization.

9
Conclusions

• Based on the Before responses, Innovation and
  Science Alberta was already a positive
  environment where people are engaged and the
  environment lives up to its name. This is
  signified by the average response to positive
  questions being above 3 and the negative
  questions being below that number.

10
Conclusions

• The training did, however, help to improve that
  atmosphere by creating a common language that
  helped encourage people to apply their creativity.

11
Conclusions

• There are two very important elements that were
  indicated that had a very positive effect on
  ensuring that the training would have a positive
  effect on the culture
• There was internal follow-up. Comments were I
  like what you are doing with the follow up
  e-mails, will not let us forget, will you.
  Bonnie is also really good and keeps reinforcing
  stuff in her e-mails and I feel that most of the
  Flexible Thinker 'tools/keywords' can and have
  been applied in our workplace. Quit often at
  least one person will bring their orange with
  them to a meeting! Means that there was coaching
  and holding others accountable for application of
  the learning happening within the environment
  This is essential to ensure positive
  organizational and team change.

12
Conclusions

• Based on the difference between the Ivey study of
  the City of Brampton and Innovation and Science
  Alberta, it is important that, if the goal is
  team or organizational change, the entire team
  should attend the training either together or
  within a short timeframe of each other. This
  helps to create the common language and
  reinforcement noted in the point above.

13
Conclusions

• There is a perceived greater risk factor in being
  innovative in the public sector. Many jobs are
  restricted by legislation and there is an
  emphasis on following the rules. There is also
  pressure from political leaders who are in turn
  pressured by their constituents. In addition,
  there is a market for the mistakes of the public
  sector and public sector employees are asked to
  continually do more and more with less. As both
  this and the Ivey study demonstrate, there is not
  only room but it is essential to create an
  environment of innovation in order to meet the
  constantly growing demands on public sector
  employees. The greatest risk is no longer in
  action, but it is in inaction. As demands change
  and multiply, the ability to adapt and create an
  environment that allow that is essential.

14
Part 2

• Summary of the Study conducted by the University
  of Western Ontarios Ivey School of Business on
  the Programs Effects in the city of Brampton,
  Ontario

15
First Level of Evaluation Did they like it?
Very satisfied
Very dissatisfied

• 156 people were trained.
• A frequent comment from participants was that
  although I was told to come to this course and
  didnt know what to expect, I have enjoyed it and
  I think it
• is important for my job
• The conclusion from this level
• of evaluation is that participants
• liked the training.

Comparison to other courses
Recommend to others?
Much worse
Much better
No
Definitely
16
Second Level of Evaluation Did they learn it?

• At the start and the end of each workshop,
  participants completed a Flexible Thinker Test,
  where they were exposed to a problem and had 3
  minutes to generate ideas on how to solve it.
• The number of ideas generated at the end of the
  workshop was significantly higher than the number
  of ideas generated at the start of the workshop.
• The conclusion from this level of evaluation is
  that during the workshop participants acquired
  new knowledge about Flexible Thinking tools
  that allowed them to be more creative under
  pressure.

17
Third Level of Evaluation Did they use it on
the job?
Large change for the better
No change
Large change for the worse

• Six weeks after the training, participants
  received a Feedback form with questions about
  their experience using The Flexible Thinker on
  the job. 96 forms were returned (61).
• The conclusion from this level of evaluation is
  that participants perceived a positive change in
  themselves and in their teams. Participants
  observed a larger change at the individual level
  than they did at the team level.

18
Conclusions

• Participants were highly satisfied by the
  Flexible Thinker workshop and learned
  creative-thinking tools during it (slides 3 and
  4).
• At the individual level, participants on-the-job
  behavior has been positively impacted by the
  workshop. Participants perceive a positive change
  in their ability to be creative under pressure
  and responsive to new situations (slides 5
  through 8, and 10)
• The impact of the workshop at the team level has
  been smaller than the impact achieved at the
  individual level. The reasons for this were
  discussed in slide 9.
• As part of the study, employees perceptions of
  their work environment were collected. Some
  variables that may represent obstacles to
  creativity and responsiveness have been detected
  in slides 11 and 12.

19
Part 3

• Complete Data on Alberta Innovation and Science
  Flexible Thinker Pilot Program

20
Negative Questions 1 and 4

• These are questions where a change downward
  represents positive change (i.e. people feeling
  that it is not difficult to create options under
  pressure).
• 1 individual and 1 team question were inserted at
  different junctures to ensure that people were
  carefully reading the question and to use as a
  check for the rest of the survey.
• There was a significant change for both the
  individual (create options under pressure) and
  the team (resistance to new ideas).

21
Positive Questions 2-3, 5-18

• These are questions where a higher rating
  represents positive change.
• Overall there was significant change, with a
  couple of exceptions (Q11 and 18).

22
Team Questions

• There is a significant change in team performance
  as people report that there is less resistance to
  new ideas (Q4) and teams adapt to change (Q15).
  The only exception is Q11 (my team is open to new
  ideas).
• This is the greatest difference between the Ivey
  Study of the City of Brampton and this one as the
  effect on teams is quite significant.
• The major difference is that in the Ivey study
  the entire department and organization did not go
  through it and there were significant time lags
  between teams, whereas with IS the entire group
  went through in a short amount of time (3 days).

23
Individual Questions

• There is a significantly positive impact on
  individual performance and flexibility.
• This corresponds almost identically with the Ivey
  study of the City of Brampton.

24
1 I often find it difficult to create options to
many of the challenges I face

• There is over a quarter of a point difference
  between how people felt before and how they felt
  afterwards.
• Only 1 person rated this as a 4 (nobody put in
  5) on the After survey. If averaging is
  applied the difference is even more significant.

25
2 I am able to be creative in my job.
26
3 I am able to communicate my point of view
effectively.
27
4 There is resistance to new ideas in my team.
28
5 I am able to overcome the resistance of others
when working to solve a problem.
29
6 I am open to the input of others when looking
for options to address an issue.
30
7 This is an environment that promotes creative
solutions.
31
8 I am able to influence people and decisions
even when I have limited or no influence.
32
9 I can always create options to help address
difficult situations.
33
10 I know how to deal effectively with difficult
people.
34
11 My team is open to new ideas.

• This slide is the exemption with a slight
  negative from the Before to the After (.07
  points).
• This is contradicted by the results from Question
  4 which indicated significantly less (.25 point)
  team resistance.

35
12 My ideas are valued.
36
13 I am able to be creative within my job.
37
14 I have the tools to adapt to change
effectively.
38
15 My team adapts to change effectively.
39
16 My team communicates in a way that helps
create options and builds ideas.
40
17 My team values my ideas.
41
18 My job allows me to effectively utilize my
strengths and creativity.
42
Kirkpatrick Level 4

• Full data on organizational effect.

43
Please describe the situation where you applied
the Flexible Thinker tools and what was the
result?

• I was in a meeting when I noticed that one of my
  colleagues was cutting off another one
  consistently and starting each statement with
  'but'... Eventually the cut-off team member
  stopped contributing. I jumped in the 'and to
  build on what so and so was saying..' This lead
  the person to rejoin the conversation and
  contributed several good ideas which lead to a
  solution.

44
Please describe the situation where you applied
the Flexible Thinker tools and what was the
result?

• Innovation and Science is planning an open house
  to display and demonstrate some of the things
  which the Ministry is doing.  The name of the
  open house will be something like "Innovation
  Open House of Hats".  The open house is scheduled
  for October 31, November 1 and November 2. 

45
Please describe the situation where you applied
the Flexible Thinker tools and what was the
result?

• Many situations at work when we had to think of
  options. Also used this approach at home, when
  thinking of landscape possibilities or meal
  ideas, etc...

46
Please describe the situation where you applied
the Flexible Thinker tools and what was the
result?

• I try to use the Yes and because tool, I find
  this is more useful than using 'But'. Also try
  to get more ideas by adapting the different hats
  tool.
• I am a but-er, i.e., loved to use the word but.
  I still let it escape however, I am cognisant
  when I do and try to retract it. This is the
  only one I have used so far. I am also
  relaxing and enjoying myself.

47
Please describe the situation where you applied
the Flexible Thinker tools and what was the
result?

• I have used the relax tools often to reduce
  stress in my new position.
• Unfortunately, I haven't applied the tools as
  well as I should have. Thanks for this reminder,
  I am going to review the material and try to use
  it.

48
Please describe the situation where you applied
the Flexible Thinker tools and what was the
result?

• I have use the tools to get buy-in from my
  manager to change a decision he made that I
  strongly thought was not align with my groups
  activities.
• In designing a way to introduce a new approach to
  doing our business I looked at it from the point
  of view of the motivation and needs of the others
  involved. Interest is better, but a long way to
  go yet.

49
Please describe the situation where you applied
the Flexible Thinker tools and what was the
result?

• Yes and Because There is a development
  opportunity in my Unit but the opportunity is
  under the Union agreement and the individual who
  is being given the opportunity is under an '
  exclusion' group and currently is a non-union
  employee. That individual is adamant that the
  development position be changed into a non-union
  position. I applied the 'yes and because'
  technique and led the individual to come to the
  conclusion that the development position is a
  Union position and she is not being ripped off.

50
Please describe the situation where you applied
the Flexible Thinker tools and what was the
result?

• Used the relax tool. Able to think much more
  clearly and results were more productive.

51
Please describe the situation where you applied
the Flexible Thinker tools and what was the
result?

• Have used some of the tools to design activities
  for training courses - Orange, what if, and hats
  (other props than hats)
• Overall in my day to day situations I have been
  more flexible, a better listener and more
  creative not only at work but at home.

52
Please describe the situation where you applied
the Flexible Thinker tools and what was the
result?

• I was dealing with a difficult client and applied
  both 'I want' to understand what he really wanted
  with 'yes and because' and 'what if' to create a
  solution.
• I am able to apply the tools when I face many of
  my internal obstacles, especially fear.

53
Please describe the situation where you applied
the Flexible Thinker tools and what was the
result?

• I applied yes and because and orange to a very
  negative person and, in spite of their
  resistance, I was able to get them to help me
  develop a solution to a situation I was having.
• I have applied the hats tool when I get stuck.

54
Accomplishments over the past 3 months and how
the tools were applied.

• By using Yes and Because in situations and being
  open and trusting. Using other tools including
  Relax in situations that are stressful.
• I aggressively created many accomplishment
  statements recently in order to prepare for an
  upcoming interview. Through this process I gained
  confidence and was prepared for the interview and
  got the job.

55
Accomplishments over the past 3 months and how
the tools were applied.

• C - Obtain knowledge of company challenges and
  provide recommendations for programs to address
  needs. A - meet with AB Companies R -
  development and implementation of new programs.
  Tools used - Orange, Relax, Library

56
Accomplishments over the past 3 months and how
the tools were applied.

• The important challenges I identified have been
  taken out of the control, or influence, of staff
  and taken over almost totally by Executive. That
  is the new challenge, with no results yet, in
  part, because interaction between Executive and
  staff is quite limited. CAR is a good idea,
  but I still have trouble seeing a strong link
  with flexible thinking. It seems to be an add-on
  of a very different type of tool.

57
Accomplishments over the past 3 months and how
the tools were applied.

• The team will improve in accountability -Take
  time to include people in my plan- ask them for
  their input and really listen -Communicate my
  expectations clearly and identify my role -Walk
  the talk and remind my superiors to do the same.
  
• I haven't completed the projects I have been
  working on since the course. Have the challenge,
  in the midst of the action, results not in.

58
Accomplishments over the past 3 months and how
the tools were applied.

• Think outside the box - new perspective on old
  issues. Listen to others and their ideas.
  Measure the success of our tem and if we have met
  our goals. I feel that I have definitely
  improved on becoming more creative and becoming a
  better listener. Have not measured our teams
  success.

59
Accomplishments over the past 3 months and how
the tools were applied.

• I applied the Flexible Thinker tools to think
  differently and communicate more effectively and
  have been able to deal with both co-workers and
  clients more effectively.

60
Next Steps

• Suggestions for moving forward

61
Comments/suggestions on effectively applying
innovation tools in the workplace.

• Periodic follow ups of the tools and additional
  classes that utilize them in specific situations
  (i.e. with project management, stress, assertive
  communication, etc.)

62
Comments/suggestions on effectively applying
innovation tools in the workplace.

• I think being more open and realizing there is
  validity in opinions that are different than
  yours is very powerful - that alone has the
  potential to change the way you communicate, hold
  meetings and interact with others - leading to a
  more creative workplace, greater value and
  increased long term performance.

63
Comments/suggestions on effectively applying
innovation tools in the workplace.

• I missed the morning portion of the session and
  was therefore disoriented when I got
  there....after 1000 a.m. I wish I had attended
  the entire day to get the benefit. This is one
  thing that may be established at the outset of
  the session. Maybe this can be a fun agenda
  item in a branch meeting. Practice one tool each
  meeting. I like what you are doing with the
  follow up e-mails, will not let us forget, will
  you. Bonnie is also really good and keeps
  reinforcing stuff in her e-mails.

64
Comments/suggestions on effectively applying
innovation tools in the workplace.

• I think the concept of Me Inc. is very powerful.
  In Alberta right now we are experiencing an
  economic boom and anyone can have a job, but
  eventually the boom will end and those that have
  not managed their career and developed their
  product(themselves) will be at a disadvantage.

65
Comments/suggestions on effectively applying
innovation tools in the workplace.

• It is a good program, just need to review and
  apply more often.
• It is effective tool to promote and develop
  people to think differently and contribute to an
  organization culture that would encourage and be
  open to new concepts and ideas.

66
Comments/suggestions on effectively applying
innovation tools in the workplace.

• Re the accomplishments journal it seemed
  counterproductive in terms of morale as there are
  too few accomplishments and such a long time in
  between.
• People know but tend to forget- everyone is the
  CEO of his or her career and he/she can be a
  leader in their assigned duties.

67
Comments/suggestions on effectively applying
innovation tools in the workplace.

• C.A.R. can be applied to performance planning and
  assessment. Challenge goal action
  Strategy/action Results results
• I feel that most of the Flexible Thinker
  'tools/keywords' can and have been applied in our
  workplace. Quit often at least one person will
  bring their orange with them to a meeting!

68
Comments/suggestions on effectively applying
innovation tools in the workplace.

• It really helps that everybody go through the
  program as it holds people accountable to create
  options and solutions.
• I have found that the course has created a more
  positive and environment where I feel I can be
  more creative.

69
Comments/suggestions on effectively applying
innovation tools in the workplace.

• I feel a positive difference in the culture and I
  am more involved in my job. The concept of
  orange-ing yourself as a problem solver is very
  powerful.
• It was difficult to apply the tools initially as
  I tend to 'but' people. Now, by applying yes and
  because and what if I find that people respond to
  me differently and I can find solutions.

70
Comments/suggestions on effectively applying
innovation tools in the workplace.

• Internal follow up through either coaching or
  team follow ups.

71
Part 4

• Complete data from the Ivey Study

72
Some of the answers to the open-ended questions
what changes have you noticed in yourself /
your team?
YOURSELF
YOUR TEAM

• I am more open-minded / I look for more options
  / I look at things from different perspectives
  (35 people)
• I have not noticed any changes / very little
  changes (14 people)
• I am calmer in situations of decision making (7
  people)
• I am more receptive to ideas from others / I use
  yes-and-because (7 people)
• I have a better approach to problems / better
  assessment of situation (4 people)
• I am more positive (4 people)

• I have not noticed any changes / very little
  changes (28 people)
• We are more open-minded / we look for more
  options / we look at things from different
  perspectives (14 people)
• More positive attitude in the team (11 people)
• Not everybody in the team was trained (8
  people)
• We work together more / we work towards the same
  goals (5 people)
• Some in the team have changed, others not (5
  people)

73
Some examples of ocassions when training was used
and suggestions to make the program more helpful
EXAMPLES
SUGGESTIONS

• yes because dealing with customers yes
  because in the boardroom (5 people)
• used tools in project to create standard
  operating procedures (2 people)
• hats at a meeting made members more relaxed
  receptive to new ideas
• we change the way we respond to the public
  (concern with roadway conditions)
• productivity staff tries new things customer
  satisfaction the party we accomodated in short
  notice
• use library partnerning in developing new
  processes

• All employees, management council members
  should attend training (15 people)
• Refresher courses / refresher courses with team
  members / type of follow up / emails to keep
  things in mind (8 people)
• All mgt. to take the course / training efforts
  need senior managers buy in / include
  supervisors (7 people)
• Training great the way it is (4 people)
• Communicate with other trained members, identify
  how people use the skills and benefit from
  training (3 people)
• Teams that work together should train together
  (3 people)

74
Barriers to using the skills
How eager were you to change your behavior?
Have you been able to apply the skills in your
job?

• Participants were asked about barriers to using
  the skills. No barrier achieved an overall high
  level.
• When asked about other barriers to using the
  skills, participants mentioned factors such as
  high workload / lack of time and have not
  encountered situation yet

Not Eager
Very Eager
Not at all
To a large extent
To some extent
Eager
Strongly agree
Strongly disagree
75
Possible reasons for a greater change in behavior
at the individual level than at the team level

• Although efforts were pursued to train all
  members of the team, including its supervisor,
  this objective could not be achieved for all of
  the teams. If not all team members and the
  supervisor are trained, it is hard to achieve
  high levels of change at the team level.
• Teams were not trained together in order to not
  disrupt the Citys operations. However, training
  a team is more than individually training the
  members of a team. There are synergistic benefits
  of training at team together.
• Some teams that went through the training may not
  be a team, but a group. That is, while team
  members need to work in a coordinated and
  mutually dependent way, group members perform
  similar jobs in a pretty independent way. Group
  members may not report a high level of change at
  the team level if they do not need to work in
  coordination with others.
• Some teams may have had high levels of
  responsiveness and creativity before the
  training, thus, couldnt report high levels of
  change achieved.
• Possible existence of a I am OK, but the rest of
  the team arent syndrome.

76
Evaluation through a Control Group Perceptions
of Creativity and Responsiveness/Spontaneity
Strongly agree
Strongly disagree

• A survey was developed to capture employees
  perceptions about their day-to-day job. The suvey
  didnt make any reference to the Flexible
  Thinker training. The first survey was completed
  before the training started. The second survey
  was completed two months after the training
  finished. 175 surveys were sent to the training
  teams. 164 people answered the first survey
  (94), 100 answered the second survey (57), and
  95 answered both (54). 173 surveys were sent to
  the control teams. 99 people answered the first
  survey (57), 83 answered the second survey (48)
  and 72 answered both (42).
• All things being equal, the training group
  experienced a significant positive change in
  individual creativity and responsiveness/spontanei
  ty, while the control group didnt. There was no
  significant change in team creativity and team
  responsiveness/spontaneity neither in the
  training group nor in the control group.

77
Evaluation through a Control GroupPerceptions
of Team Performance
Values to think about
Strongly agree
Strongly disagree

• Before the training, the training and control
  groups were not homogeneous in terms of
  performance. The control group had a higher
  perception of its performance than the training
  group had.
• Performance variables didnt experience a
  significant change in the period of study neither
  in the training group nor in the control group.
• To think about While teams perceive their
  customer satisfaction and productivity levels to
  be around 5, they perceive their innovation and
  employee satisfaction levels to be around 4.

78
Assessment of Contextual Variables
Values to think about
Strongly agree
Strongly disagree

• Real-time information communication were low
  across teams. Participants do not feel updated
  about what is happening in other teams and in the
  organization. This is an obstacle to creativity,
  since in creative processes real-time information
  plays a coordinating role and permits groups to
  get feedback on their actions.
• The level of experimentation in the teams
  culture was low across teams. Participants do not
  feel that the firms culture supports competent
  mistakes when trying new approaches to things.
  This is an obstacle to creativity, since in
  creative processes experimentationa and errors
  are indispensable.
• Employees perceive medium levels of change and
  turbulence in the environment. This is an
  obstacle to creativity, since the perception of
  external change helps to motivate teams to be
  more creative at work.