ClickerPlus Pilot Study: Applying Advertising Research Methods to the Measurement of Student Learning Goals

1
ClickerPlus Pilot StudyApplying Advertising
Research Methods to the Measurement of Student
Learning Goals

• Teaching and Technology Conference
• Baruch College
• 26 March 2010
• Prepared by Brian Moore

2
Agenda

• Background
• Advertising research methods
• Students preconceptions
• ClickerPlus Method
• Survey the audience
• Results of the pilot study
• Results of the audience survey
• Discussion

3
What is ClickerPlus?

• Quantitative method
• What college students know
• What they learn
• Pre/post measurement technique
• Borrowed and adapted from advertising research
• Student response systems (a.k.a. Clickers)

3
4
Hypothesis

• ClickerPlus made me a better teacher
• Students learned (more)
• Students appreciated the class (more)

5
Advertising Research Methods
6
How do we know that advertising is effective?

• Case History
• Dove Deodorant
• Tracking Study

7
(New)Dove Deodorant Moisturizing cream
Soothes and beautifies the underarms.
8
Dove Deodorant Communication Goals

• Be recognized as the best brand at caring for the
  look and feel of underarm skin
• Become the beauty icon in the deodorant market
  through focusing on improving the look of
  underarm skin
• Ensure that perceived efficacy remains at parity
  with the market average
• Source Euro Effie Awards Case History, 2009

8
9
To achieve the goals, people have to learn
Benchmark Survey (pre-wave) Wave 1 Survey (post-wave)
leaves underarm skin soft and smooth X X
helps me cope with the problems of shaving Y Y

10
Advertising is designed to achieve these goals.
11
Results
12
Dove deodorant owns underarm beauty.
Papers Case Studies World Advertising
Research Council
13
How do I know that I am an effective teacher?

• Establish concrete learning goals
• Measure what students know
• Measure what they learn

14
What do students know?Are they Blank Slates?
15
Intrinsic aptitudes
16
Today, the sciences of human nature have
threatened the Blank Slate by trying to delineate
what has to be present in the mind in order for
learning to occur in the first place.

• Stephen Pinker
• The Blank Slate
• 2002

17
ClickerPlus Pilot Study Results Summer 2009

• Prepared by
• Brian Moore
• Baruch College
• July 2009

18
Method

• Construct a series of true-false statements
• Have the students respond to the statements
• Using classroom response systems (clickers)
• TurningPoint
• Analyze the responses.
• Adjust ensuing lectures and tests accordingly.
• Clarify and reinforce (circa mid-semester).
• Re-administer on the last day of class.
• Compare the pre-wave to the post-wave.
• Adjust future class lectures and tests
  accordingly.

19
End of the semester July 2, 2009
Beginning of the semester June 1, 2009
Learning
True/False questions
True/False questions
Benchmarks
Changes over time
20
Sample Description and Timing

• Place Baruch College
• Course Advertising Promotion (MKT3520)
• Sample Composition Baruch undergraduates
• Sample Sizes
• Pre-Wave, n38
• Post-Wave, n 25
• Timing
• Pre-Wave June 1, 2009
• Interim Review June 18, 2009
• Post-Wave July 2, 2009

21
Limitations

• Results are merely directional.
• Sample is insufficient, i.e.
• Small
• Unequal from wave to wave
• Unrepresentative
• Need to know if the statements accurately measure
  the actual grasp of the underlying concept.

22
Instructions to the students in the classroom

• This is not a test. You wont be graded on this.
  We just want to find out what you know.

23
First Slide
Based on everything you know or may have heard
about advertising, please indicate whether you
believe that the following statements are true
or false.
24
Pick up your Clickers
25
The 10 Statements

1. The most important thing good advertising does is
   make an emotional connection with consumers.
2. The production of advertising ideas follows a
   clearly defined process.
3. Good advertising can keep a bad product alive.
4. American Idol would be a good place to advertise
   the new version of the iPhone.
5. What you say in advertising is more important
   than how you say it.
6. Emotional advertising appeals are more effective
   than rational advertising appeals.
7. Advertising increases the value of the advertised
   product.
8. If your advertising grabs peoples attention, it
   will probably work.
9. If you like a particular product, you will be
   better prepared to create great advertising for
   that product.
10. It is more effective to have your advertising
    seen or heard by more people than it is to have
    fewer people see it more often.

25
26
How did the students do?
27
Summary of before/after results

• For all 10 Statements

28
On some questions, students improved dramatically
over the course of the semester. On other
questions, they did not improve at all.
28
29
Results for each statement, before and after the
semester
30
The most important thing good advertising does is
make an emotional connection with consumers.
False If it doesnt sell, it is not creative.,
David Ogilvy (Belch Belch 285)
31
The production of advertising ideas follows a
clearly defined process.
True The production of ideas is just as
definite a process as the production of Fords.,
James Webb Young (Belch Belch 260).
32
Good advertising can keep a bad product alive.
False Nothing will put a bad product out of
business faster than a good advertising
campaign., David Ogilvy
33
American Idol would be a good place to advertise
the new version of the iPhone.
False It would be a waste of mass media on an
exclusive product, i.e. a poor return on
investment (Belch Belch 354).
34
What you say in advertising is more important
than how you say it.
True What really decides consumers to buy or
not to buy is the content of your advertising,
not its form., David Ogilvy (Belch Belch 291).
35
Emotional advertising appeals are more effective
than rational advertising appeals.
True You should always seek emotional benefits.
They are more compelling, and they are easier to
defend (Belch Belch 285).
36
Advertising increases the value of the
advertised product.
True The simple act of advertising a product
makes it appear more valuable. Good advertising
can increase value perceptions and direct
pricing strategies (Moore).
37
If your advertising grabs peoples attention, it
will probably work.
False Advertising succeeds or fails depending
on how well it communicates the desired
information and attitudes to the right people at
the right time and at the right cost, Russell
Colley (Belch Belch 220).
38
If you like a particular product, you will be
better prepared to create great advertising for
that product.
False You may not be representative of the
target consumer, and you wont be objective
(Moore).
39
It is more effective to have your advertising
seen or heard by more people than it is to have
fewer people see it more often.
False It depends on your objectives, e.g. if
your message is complicated, or If your product
is unfamiliar to people, you probably need more
frequency (Belch Belch 336).
40
Am I an effective teacher?

• Did the students learn?
• Did they appreciate the class?

41
Did the students learn? Yes and No

1. The most important thing good advertising does is
   make an emotional connection with consumers.
2. The production of advertising ideas follows a
   clearly defined process.
3. Good advertising can keep a bad product alive.
4. American Idol would be a good place to advertise
   the new version of the iPhone.
5. What you say in advertising is more important
   than how you say it.
6. Emotional advertising appeals are more effective
   than rational advertising appeals.
7. Advertising increases the value of the advertised
   product.
8. If your advertising grabs peoples attention, it
   will probably work.
9. If you like a particular product, you will be
   better prepared to create great advertising for
   that product.
10. It is more effective to have your advertising
    seen or heard by more people than it is to have
    fewer people see it more often.

41
42
Did the ClickerPlus students appreciate the
class?(Yes, but I made a lot of other changes
as well).
42
43
How did the audience do?
44
Potential ClickerPlus Benefits

• Objective
• Flexible
• Infinitely scalable
• Easy to construct and administer
• Several types and levels of learning
• Yields real-time results
• Measures the attainment of learning goals
• Inspires new goals
• Provides benchmarks for
• Overall improvement
• Developing individual goals for students
• Preserves and protects academic freedom

45
Conclusions

• ClickerPlus made me a better teacher
• Learned
• Appreciated
• Need more data
• Incorporate student test scores
• Construct better and larger samples
• Validate the survey
• Qualitative research
• Search phrase algorithm
• Automate the survey

46
Next Steps

• Awaiting response to our grant proposal from the
  U.S. Department of Education
• More instructors
• Different subjects
• Different levels
• Brian.Moore_at_baruch.cuny.edu
• 646-312-3332

47
Works Cited

• Belch, George, and Michael Belch. Advertising and
  Promotion. New York McGraw-Hill, 2009
• Georgia Tech School of Civil and Environmental
  Engineering. 2000 Instructional objective
  writing assistant
• Moore, Brian. Lectures on Advertising and
  Promotion, Baruch College, New York, June 2009.
• Ogilvy, David. Various lectures, speeches and
  books.
• Pinker, Steven. The Blank Slate. New York
  Penguin Group, 2002

48
THE END
49
Appendix

• Scatter diagram

50
LEARNING GOALS At the end of this course, you
will understand how global advertising agencies
propose integrated marketing communication
programs to global manufacturers of consumer
package goods. You will learn how to conduct
research to define the target audience and its
media habits, brainstorm to generate messaging
ideas, develop a creative strategy, create
executions derived from the strategy, produce the
executions, test whether or not the executions
communicate their intended messages, plan and
buy the media to deliver the messages to the
target audience, and incorporate all of your
findings into a persuasive, visually-intensive,
oral presentation. You will learn how to work as
part of a cross-functional team to achieve the
goals stated above.
51
The technique discriminates, and more students
answered the questions correctly at the end of
the semester than they did at the beginning of
the semester.
52
(No Transcript)
53
(No Transcript)
54
Formatting Learning Goals

• Student centered (e.g., by the end of the
  course, students will be able to),
• Outcome oriented (e.g. write a scholarly
  research paper), and
• Detailed (e.g., using MLA-style referencing and
  formatting).
• Georgia Tech 2000
• COM3150 Course Review

54
55
My Learning Goals

• Defend the primacy of strategic messaging.
• Describe and apply the advertising planning
  process.
• Recognize why good advertising cannot keep a bad
  product alive.
• Analyze the efficiency of different media
  vehicles, using syndicated research.
• Explain why the content of your advertising is
  more persuasive than its form.
• Discuss why emotional benefits are more
  persuasive than rational benefits.
• Describe how and why advertising increases the
  value of the advertised product.
• Interpret and explain the hierarchy of effects
  model.
• Identify target audiences by analyzing syndicated
  research.
• Demonstrate the trade-offs between advertising
  reach and frequency.

55
56
Less Effective Example Effective Example
This course in Engineering Graphics will introduce students to the use of computer-aided-design software At the end of this course in Engineering Graphics, you, the student, will be able to draw a multi-view representation of a solid object using a computer-aided-design software
Georgia Tech 2000
56