Presentation to the City of Mississauga

1
Presentation to theCity of Mississauga

• Complete Results of Flexible Thinker Program

2
Kirkpatricks Levels of Training Evaluation
LEVELS EVALUATED IN THIS STUDY
Level 4
Results Did it make a difference?
Level 3
Behavior Did they use it?
Level 2
Learning Did they learn it?
Level 1
Reaction Did they like it?

• Each level of evaluation builds on the previous
  one. If people like the training, they are likely
  to learn it. If people learn it, they are likely
  to use it on the job. If people use it on the
  job, they are likely to achieve results.

3
First Level of Evaluation Did they like it?
Very satisfied
Very dissatisfied

• 32 people were trained.
• A frequent comment from participants was that
  Excellent tools presented and very practical for
  here.
• The conclusion from this level
• of evaluation is that participants
• liked the training.

Comparison to other courses
Recommend to others?
Much worse
Much better
No
Definitely
4
Second Level of Evaluation Did they learn it?
Average per person

• At the start, participants completed a Flexible
  Thinker Test, where they were exposed to a
  problem and had 3 minutes to generate ideas on
  how to solve it. At the end they were given the
  exact same test and told to come up with new and
  different answers than they thought of before.
• The number of ideas generated at the end of the
  workshop was significantly higher than the number
  of ideas generated at the start of the workshop.
• The conclusion from this level of evaluation is
  that during the workshop participants acquired
  new knowledge about Flexible Thinking tools
  that allowed them to be more creative under
  pressure.

5
Second Level of Evaluation Did they learn it?

• At the beginning of the workshop, participants
  were asked to identify all of the obstacles that
  they face in finding a solution to their personal
  challenge.
• They were then told to apply the tools to come up
  with specific ideas that they could apply the
  next day to help overcome them.
• When they had at least 1 option to overcome that
  obstacle, they were told to check it off.
• The conclusion from this level of evaluation is
  that during the workshop participants were able
  to apply the Flexible Thinking tools to the
  real issues that they face.
• This measurement was done without any assistance.
  At the end of the course, we were able to
  brainstorm options to the remaining obstacles to
  equal options to 100 of the obstacles listed.

6
Third Level of Evaluation Did they use it on
the job?
Ind. Questions
Team Questions

• 1 week prior to training, an online survey was
  conducted asking the pariticpants a series of
  questions regarding both individual and
  team/organizational performance. An after survey
  was sent to participants 10 weeks after the
  training event.
• 31 participants filled out the Before survey
  and 29 completed the After. This represented
  100 and 94 of the number of course
  participants.
• The survey represented a series of statements
  that participants were given and asked to rate on
  a 1 to 5 scale with 1 being STRONGLY DISAGREE to
  5 STRONGLY AGREE.
• The questions contained both positive (where a
  move toward 5 represents positive change) and
  negative (where 1 represents positive change)
  questions.
• The conclusion from this level of evaluation is
  that participants perceived a significantly
  positive change in themselves and in their teams
  (which is measured by more than a .20 point
  difference).

7
Positive Questions 2-3, 5-18

• These are questions where a higher rating
  represents positive change.
• Overall there was significant change, with some
  changes of over half a point.

8
Negative Questions 1 and 4

• These are questions where a change downward
  represents positive change (i.e. people feeling
  that it is not difficult to create options under
  pressure).
• 1 individual and 1 team question were inserted at
  different junctures to ensure that people were
  carefully reading the question and to use as a
  check for the rest of the survey.
• There was a significant change for both the
  individual (create options under pressure) and
  the team (resistance to new ideas), with the
  perception of the team being over quarter point
  difference.

9
Team Questions

• There is a significant change in team performance
  as people report that there is less resistance to
  new ideas (Q4) and teams adapt to change (Q15).
• This has been found by research to have a direct
  impact on the attraction and retention of talent
  because people who feel that they are networked
  in and have positive feelings about their
  co-workers are much more likely to stay.
• Somewhat higher, but in line with other studies
  conducted within the public sector.

10
Individual Questions

• There is a significantly positive impact on
  individual performance and flexibility.
• Individual questions are a method of measuring
  engagement.
• Impact in line with previous studies.

11
Fourth Level of Evaluation Did it affect the
organization?

• Assessed through 2 questions/statements.
• One a scale of 1 to 5, participants rated 3.93
  that they have been able to apply the tools from
  the course.
• 22 out of 29 noticed either a subtle or
  pronounced difference in the environment 10 weeks
  after the course. This represents 76 noticing a
  positive change.

12
Fourth Level of Evaluation Did it affect the
organization?

• Measurement also included 2 open ended questions
  to fully gauge the effect of the program on the
  organization.

13
Fourth Level of Evaluation Did it affect the
organization?

• The questions were
• If you answered yes (that you noticed a
  difference since the workshop), please describe
  as best you can the differences you have noticed.
  
• Please describe the situation where you applied
  the Flexible Thinker tools and what was the
  result?
• We received 21 responses asking to describe
  differences that they noticed and 17 responses to
  what happened when they applied the tools.

14
Fourth Level of Evaluation Did it affect the
organization?

• Responses to the changes that people noticed are
• In my smaller team, team I lead, we discuss other
  staffs different perspectives (change hats),
  where they are coming from, we take '5'(relax)
  and sit back and discuss before drawing any
  conclusions. Note, in my answers to the survey, I
  was answering with regards to the bigger team in
  which myself and the team I lead work with.

15
Fourth Level of Evaluation Did it affect the
organization?

• In some cases, more closed off staff appear to be
  more open, and other staff are vocalizing, 'it's
  not my job'. One common theme is, in meetings in
  which we have to resolve issues, staff prefix the
  meeting with 'Let's orange this!'. Sometimes this
  is genuine and other times it's telling other to
  Orange it so we agree with this person. It's a
  start.
• I do consider and utilize the concepts when
  examining how to address issues or to deal with
  others.

16
Fourth Level of Evaluation Did it affect the
organization?

• I try to think out of the box - ORANGE it and get
  more creative. I see others point better as well
• People are working much better together and
  seemed to be more focused on finding answers than
  blaming each other.

17
Fourth Level of Evaluation Did it affect the
organization?

• I seen the change mostly in how I respond to
  others and how I approach situations. I don't
  seem to feel as defeated as I am have in the past
  because I know that I will come up with a
  solution.

18
Fourth Level of Evaluation Did it affect the
organization?

• I have been able to use a number of the tools
  that I learned from the sessions most recently at
  a cross-departmental workshop/meeting for the
  migration and implementation of an advanced
  enterprise resource program(ERP) within the city.

19
Fourth Level of Evaluation Did it affect the
organization?

• Previously I felt a lot of resistance from a
  certain individual, and after the workshops I
  noticed that the same individual actually
  listened to what I said in a meeting and he/she
  went as far as drawing a diagram on a whiteboard
  and asked me if that was what I had in mind....I
  felt as though I had purpose O) If only everyone
  got the same out of this course.
• People have "mentioned" that we should use what
  we learned in the workshop.

20
Fourth Level of Evaluation Did it affect the
organization?

• How I interact with people has changed, I'm more
  motivated to speak with everyone regarding issues
  and find resolution with their involvement.
• Speaking only for myself, I think I have tried to
  determine why other persons may be behaving in a
  way I may not agree to get past the "issues"
  looked for and implemented more creative
  approaches and have tried to communicate those
  approaches in a more positive way.

21
Fourth Level of Evaluation Did it affect the
organization?

• Confidence in an eventual solution provides
  constructive work.
• I can better communicate my ideas. Redefining
  helps create a better point of view of the
  problem and people react more positive.

22
Fourth Level of Evaluation Did it affect the
organization?

• Extremely negativity has been replaced with
  logical dialogue and multiple solutions to
  resolve issues.
• Feel better every time I tell myself to relax
  whenever I am under pressure at work
• Not only am I now open to more ideas, suggestions
  and opinions but I have been able to share these
  tools with my team as well.

23
Fourth Level of Evaluation Did it affect the
organization?

• More productive brainstorming sessions which lead
  to better solutions. More buy in to the agreed
  upon solution.
• Differences have been subtle. I use the
  information from the course as another source for
  options to resolve problems or conflict.

24
Fourth Level of Evaluation Did it affect the
organization?

• Oranging situations - "Thinking outside the box"
  - Wearing the different hats and dealing more
  effectively with the "I Want Group". - Networking
  with different clients
• Using the "What if Concept".

25
Fourth Level of Evaluation Did it affect the
organization?

• There is more of an attitude that we can work
  together to solve problems. Problem solver vs
  problem creator very powerful concept.
• I think everyone seems a little more open to
  other's suggestions and ideas.

26
Fourth Level of Evaluation Did it affect the
organization?

• Responses to the question of describing a
  situation where they applied the Flexible Thinker
  tools and what was the result included

27
Fourth Level of Evaluation Did it affect the
organization?

• Relax and think - During some heated discussions,
  I will say I do not want to continue discussion
  until I've had time to research and reflect with
  my team.

28
Fourth Level of Evaluation Did it affect the
organization?

• Changing hats - In situations of conflict and my
  team members report to me that they are
  confronted with 'I'm right and you are wrong'.
  These team members have also attended the
  workshop. They do not respond in kind but instead
  say, that is your opinion but it is not up to
  either of us, but the people above.

29
Fourth Level of Evaluation Did it affect the
organization?

• Good answer I say. I took my team members
  off-site to discuss. I talked about why I think
  this person may be behaving this way -loss of
  control of how things were done before. I can't
  change that persons behavior but reviewed with my
  team the process I'm following to resolve these
  issues, democratically and at the management
  level.

30
Fourth Level of Evaluation Did it affect the
organization?

• I document requirements, team member documents
  specifications, developer programs according to
  specifications and does not deviate based on
  verbal instructions. Any concerns, go to me, I
  document the issue and our position with pros and
  cons, I discuss first with the other leader and
  the co-ordinator.

31
Fourth Level of Evaluation Did it affect the
organization?

• If the conclusion is not in our favour, I report
  how it negatively affects the project in which
  management will eventually see and we go on. At
  least we followed a process and did not react
  emotionally and unprofessionally. We let it go.
  Before the workshop, I would have followed this
  plan anyway but what is different is I'm
  communicating to my team more on my method so
  they are involved.

32
Fourth Level of Evaluation Did it affect the
organization?

• I have applied concepts at work to assisst in
  dealing with other units that tend to resist
  change or the review of new alternatives.
• I was having a difficult time with a person and
  was at the point where I was even thinking about
  quitting but after the course I began to
  understand where the person was coming from and
  the issues are being resolved.

33
Fourth Level of Evaluation Did it affect the
organization?

• 1. I Want - Assessed individuals and their "I
  Want's". This allowed me to understand what they
  wanted to get out of the project in view of the
  overall plan. 2. Hats - Tried to see things from
  different perspectives so that I could fully
  understand the vision for the project.

34
Fourth Level of Evaluation Did it affect the
organization?

• 3. Relax - When confronted with a situation where
  I knew that their may be some conflict, I relaxed
  and thought twice before proceeding. 4. Yes
  Because - When ideas/options began to be
  rejected, I acknowledged the individual's concern
  and continued to reinforce my idea as a viable
  solution.

35
Fourth Level of Evaluation Did it affect the
organization?

• 5. What if - Tried to imagine what it would be
  like to not implement this improvement, what
  options would we then have? 6. Library - The
  brainstorming that came of the meeting was great
  in that there were a number of people who could
  express their views which were very different as
  they came from people who plan, from those who
  use, and from those who maintain/monitor the
  various systems.

36
Fourth Level of Evaluation Did it affect the
organization?

• 7. Orange - When asked "what you would like to
  see in a new system", I tried to think of all the
  various uses that I would like to see the system
  perform, in many cases they were not even
  considered in the past and may be included in the
  project. "Oranging" the system and it's uses was
  very useful as it opened up new avenues for the
  entire team.

37
Fourth Level of Evaluation Did it affect the
organization?

• I was able to motivate my co-workers to apply for
  education that would qualify them for different
  careers. I was able to do this by applying some
  of the fundamentals of the flexible thinker
  course, using 'I want' to open the conversation
  and the 'yes and because' to have them follow
  through until we reached the end result.

38
Fourth Level of Evaluation Did it affect the
organization?

• For example, I came up with the idea to organize
  decorating a community centre for new year's day
  to brighten up the first meeting of the year. As
  a result, staff felt that the decorations should
  stay the entire month of January, if not beyond
  that time period.

39
Fourth Level of Evaluation Did it affect the
organization?

• In personal financial planning, setting goals to
  invest money, and its been working favourably for
  me at this point.
• In dealing with a difficult smart coding issue,
  my team and I were able to find a solution to a
  seemingly unsolveable problem.

40
Fourth Level of Evaluation Did it affect the
organization?

• I recently applied the tools in one situation
  where I received a Help Desk call from one of the
  cities we support. The problem was the
  installation of a new patch to the system where
  the database administrator sustained that one of
  the packages was wrong and missing the
  specification because it would not compile.
  Although the Help Desk call was tackled by three
  other members of my team none of them could get
  to the bottom of the problem.

41
Fourth Level of Evaluation Did it affect the
organization?

• All we new was that the package had no problems
  to compile. For me it looked more a Preconceived
  Idea problem than a technical problem. I tried to
  Orange (Redefine) the problem and I said What If
  I called this guy ( the database administrator)
  and listen to what he has to say and what he
  believes it is wrong and try a Yes and Because
  approach. This is what I found out Once he got
  an error at the installation of the patch he got
  stuck with this Preconceived Idea that the
  package was wrong and was missing the
  specification.

42
Fourth Level of Evaluation Did it affect the
organization?

• I said Yes the specification is missing Because
  is already in the system and we do not need to
  overwrite it. Once I explained to him this he
  looked for it and found it in the system. Helping
  to overcome this Preconceived Idea allowed us to
  continue the investigation and find out what the
  real problem was. It turned out that the method
  used to install the patch caused truncation of
  the code therefore the package did not compile.

43
Fourth Level of Evaluation Did it affect the
organization?

• Two teams had to agree on a data model for an
  application that they would share. We 'oranged' a
  sessions and came up with options we hadn't
  thought of before.

44
Fourth Level of Evaluation Did it affect the
organization?

• Using the I want" concept, I was able to force
  further discussions in an enterprise wide
  software project to ensure the needs of our
  department weren't lost to the larger users.
  Final outcome still to come, but at least they've
  agreed to continue talks.

45
Fourth Level of Evaluation Did it affect the
organization?

• "Hats" were a big hit. What we thought was a
  creative road block, soon became a very effective
  brainstorming session with many very positive
  results and ideas.

46
Fourth Level of Evaluation Did it affect the
organization?

• Team session to discuss the AIM to Hansen 7
  conversion. Both AIM and Migration team proposed
  various solutions (oranged it), proposed many
  "what if" that were evaluated with the "yes and
  because" techniques. When the meeting got tense,
  we reminded each other to take a deep breathe and
  relax. The meeting was the most productive the
  two teams have had in a year.

47
Fourth Level of Evaluation Did it affect the
organization?

• The best example is simply to ask my group (or
  individual) to suggest five options to solve a
  specific problem or issue so they know that
  options are always encouraged - regardless of
  whether they ultimately implemented or not. Most
  of the time, even the "far out" options can be
  accommodated in some fashion (or parts thereof).

48
Fourth Level of Evaluation Did it affect the
organization?

• "What If" concept - Upgrade to the SAP System for
  the City of Mississauga. "I want" - How will we
  benefit from the upgrade once implemented
  "Different Hats" - Not a me environment, but us
  as a whole, "The City of Mississauga" "Library"-
  Networking - working with various financial teams
  across the City Sharing thoughts and ideas "Yes
  and because - we are in it together, and change
  that will affect all of us in a positive manner
  Relax - Chill out, open the ears!!! Listen!!

49
Fourth Level of Evaluation Did it affect the
organization?

• I had to deal with a difficult customer and by
  applying 'yes and because' I was able to hear
  what the person really needed and got him onside
  to help solve the problem.
• Can't think of anything specific.

50
Breakdown of Individual Questions and Responses
51
1 I often find it difficult to create options to
many of the challenges I face
52
2 I am able to be creative in my job.
53
3 I am able to communicate my point of view
effectively.
54
4 There is resistance to new ideas in my team.
55
5 I am able to overcome the resistance of others
when working to solve a problem.
56
6 I am open to the input of others when looking
for options to address an issue.
57
7 This is an environment that promotes creative
solutions.
58
8 I am able to influence people and decisions
even when I have limited or no influence.
59
9 I can always create options to help address
difficult situations.
60
10 I know how to deal effectively with difficult
people.
61
11 My team is open to new ideas.
62
12 My ideas are valued.
63
13 I am able to be creative within my job.
64
14 I have the tools to adapt to change
effectively.
65
15 My team adapts to change effectively.
66
16 My team communicates in a way that helps
create options and builds ideas.
67
17 My team values my ideas.
68
18 My job allows me to effectively utilize my
strengths and creativity.
69
Next Steps

• Participants were asked to share any
  comments/suggestions/ideas they might have about
  Flexible Thinker and how it can be applied within
  the workplace and the organization.
• 15 people provided responses. Here are their
  responses

70
Next Steps

• The concepts of the Flexible Thinker are great
  but I'm concerned that over time, the concepts
  and buzz words will be forgotten. To keep this
  going is for the leaders to remind us and lead by
  example. Thanks Mike for a great workshop. I have
  a great small team I am leading in the first
  place, but I feel we are working even better
  together, constantly communicating and thinking
  of other ways to solve things after the workshop.

71
Next Steps

• I think there are some other work units that
  would benefit from the experience

72
Next Steps

• Excellent Workshop.....all staff should attend so
  they can adapt to change easier
• Should be implemented at the City through each
  team to make an long-term impact. Very powerful
  and applicable tools.

73
Next Steps

• I found that attending the Flexible Thinker
  workshop encouraged and promoted confidence that
  even an idea no matter how silly or irrelevant it
  may seem can sometimes lead to the solution that
  would work for all involved.

74
Next Steps

• I have been able to work with a number of
  individuals who also attended the flexible
  thinker workshop and found that I could see
  others using the tools as well. It has enabled me
  to consciously "think outside the box" and work
  collaboratively on achieving good outcomes. Great
  workshop, great facilitator. Thanks Mike for
  sharing these wonderful tools.

75
Next Steps

• I think this course would benefit everyone in
  this organization. We had a lot of fun and
  learned not to take some peoples comments
  personally.

76
Next Steps

• As with most of management that has gone through
  this course, we need everyone onboard to receive
  new ideas. Therefore, this should be a mandatory
  course that management goes through every other
  year or so. The different sections within the
  city need to know how they impact each other, and
  a forum they have to change the inter-sectional
  procedures. This would help the flow of new
  ideas.

77
Next Steps

• The Corporation is facing a situation where an
  area of its work must change and adapt to several
  real pressures. In addition to my work life being
  impacted directly, I am expected to contribute to
  the solutions to those pressures. I must bring
  all manner of support, including the Flexible
  Thinker approach, to this effort.

78
Next Steps

• I think all the concepts from Flexible Thinker
  are very good and helpful at work and in real
  life.

79
Next Steps

• The benefits as a result of taking this course
  were immediate. I would strongly recommend The
  Flexible Thinker to, not only other supervisors,
  but to any staff that has to deal with upper
  management, a team, subordinates, and the general
  public.
• The course touched on leaderless leadership. This
  would be a good topic to expand on for future
  city sessions.

80
Next Steps

• The has been a lot of growth at the City over the
  8 years I've been employed here. Many decision
  makers or influencers are young and less
  experienced. In that light changing the
  environment from one where every answer has to be
  "correct" to one where we can feel safe not to
  have all the answers makes decision making easier
  and more fulfilling.

81
Next Steps

• Should be applied as a standard leadership tool
  within the City.

82
Next Steps

• I think the 'oranging' part was most interesting,
  showing you that just when you think you don't
  have any more ideas - something else may come
  along change that

83
Conclusions

• This is the 3rd public sector study completed and
  the results of all 3 had a number of
  similarities.
• The team change in Mississauga was significantly
  higher than the Ivey Study and slightly higher
  than the Government of Alberta study.

84
Conclusions

• What has been consistently found to work and
  create a long term change is when people and
  teams go through in a fairly short period of
  time. This helps people reinforce what they have
  learned and create a common language.
• These measurements, as well as the previous
  measurement within the Government of Alberta,
  show that there are 3 major areas that this
  program has an affect on. They are

85
Conclusions

• Employee engagement with organizational goals and
  values. The questions in the survey that were
  targeted to individuals as well as the open-ended
  responses show that people felt they were more
  engaged and that they understood and bought into
  the goals and values of the organization as
  people felt that their ideas were more respected.

86
Conclusions

• Help create a broader pool of talent. The
  responses toward teams and that people were
  sharing ideas and information together helps to
  create a broader pool of talent as people are
  empowered to create ideas and learn from each
  other in the team.
• Retain and attract talent. As stated previously,
  because people feel more attached to their teams
  and more positive about the environment, they are
  much more likely to stay.