FUR XII, LUISS, Roma, June 24, 2006

1

• FUR XII, LUISS, Roma, June 24, 2006
• Financially Stimulated Effort Hits
• Individual Cognitive Constraints
• Evidence From a Forecasting Task with Varying
  Working Memory Load
• by Ondrej Rydval, CERGE-EI, Prague
• Acknowledgements invaluable comments
  especially from Andreas Ortmann and Nat Wilcox,
  financial support from Grant Agency of the Czech
  Republic and Hlavka Foundation
• COMMENTS WELCOME!!!

2
MOTIVATION

• I examine how performance-contingent financial
  incentives interact with intrinsic motivation and
  cognitive constraints in determining individual
  differences in cognitive performance.
• Camerer and Hogarth (1999, JRU) propose a
  capital-labor framework describing how financial
  incentives may interact in non-trivial ways with
  intrinsic motivation to induce cognitive effort
  (labor), and how cognitive effort productivity
  may vary across individuals due to their
  different cognitive constraints (capital).
• Even if salient financial incentives induce high
  effort, both financial and cognitive resources
  may be wasted for individuals whose cognitive
  constraints inhibit performance improvements.
• This prediction, if warranted, calls for
  attention to individual cognitive constraints in
  designing efficient incentive schemes in firms,
  experimental settings, and elsewhere.
• The next slide shows the main blocks of the
  capital-labor framework

3
Here is how one can think of the capital-labor
framework. I briefly outline the literature and
the main research questions
financial incentives
Camerer Hogarth (1999)
Cognitive Production
Labor theory of cognition
Capital-Labor Framework
cognitive performance
Crowding out?
intrinsic motivation
Benabou Tirole trilogy, Cacioppo et al. (1996)
cognitive effort
cognitive capital
Degree of complementarity?
Labor theory of cognition Smith Walker (1993),
Wilcox (1993)
Experimental psychology Engle Kane (2004)
4
DESIGN

• I provide an initial empirical test of the
  capital-labor framework, focusing on the
  complementarity of cognitive capital and effort.
• To impose theoretical structure on the framework,
  one can broadly think of cognitive constraints as
  a vector composed of general cognitive capital
  and task-specific capital.
• Drawing on contemporary cognitive psychology,
  I measure individual differences in general
  cognitive capital by a working memory span test
  a strong and robust predictor of general fluid
  intelligence as well as performance in cognitive
  tasks requiring controlled information
  processing.
• Since pre-existing task-specific capital (think
  of expertise) is vital for performance in many
  field cognitive tasks but is hard to measure,
  I intentionally minimize its potential relevance
  by designing a controlled laboratory experiment
  where working memory is itself the main component
  of task-specific capital, aside expertise
  acquired endogenously through on-task learning.
• The next slide shows what working memory is and
  why it is a useful measure of individual
  differences in cognitive capital

5

• Working memory is a domain-general ability to
  control and rapidly reallocate attention among
  competing cognitive uses, over and beyond
  domain-specific short-term memory capacity.
  People with high working memory are better able
  to code and store a limited amount of
  task-relevant information and keep this
  information accessible during the execution of
  complex, information-interfering cognitive and
  behavioral tasks.
• A typical working memory span test has two
  interacting components
• - processing component e.g. calculating simple
  equations (9/3)-2?
• - memory component e.g. memorizing a sequence
  of letters
• In the test, subjects observe several sequences
  with alternating processing and memory components
  (sequences have various length).
• After a given sequence, subjects must recall the
  memory components in correct order (e.g.
  letters).
• Throughout the test, subjects must also maintain
  accuracy/speed on the processing component.
• Subjects working memory score depends on the
  number of memory components recalled in correct
  order.
• Note A typical short-term memory test only has
  the memory component.

6
DESIGN cont.

• To identify the impact of working memory on
  cognitive performance, I supply external memory
  to subjects as a treatment
• In a computerized time-series forecasting task,
  two screens with forecast-relevant information
  are presented either concurrently or sequentially
  gt two between-subject treatments.
• HYPOTHESIS to be tested Since the Sequential
  (Concurrent) treatment offers less (more)
  external memory to subjects and hence features
  a higher (lower) working memory load, working
  memory should be a stronger (weaker) determinant
  of forecasting performance, after controlling for
  other between-subject cognitive, motivational and
  personality differences.
• The next several slides show how I experimentally
  implemented the time-series forecasting task and
  in particular the Sequential and Concurrent
  treatments

7
Time-series forecasting task (a la Klayman, 1988)
Subjects repeatedly forecast Omega, a sum of
Signal, Repeating pattern and Error. They observe
8-period history windows of Signal and Omega (see
the green columns). Subjects are told that to
accurately forecast Omega, they need to discover
the Repeating (seasonal) pattern from successive
values of Omega and Signal by subtracting them.
The unobserved, random Error makes this task
harder but subjects know the Error distribution
(uniform discrete). Subjects forecast next-period
value of Omega. To be able to do that (as Signal
is unpredictable), they are shown next-period
value of Signal. Concurrent treatment subjects
observe Signal and Omega on one
screen. Sequential treatment subjects observe
Signal and Omega on two successive
screens. Financial incentives to forecast
accurately are high subjects can earn up to
70-80 PPP Dollars.
8
Here is what Concurrent treatment subjects
observe on a typical screen (for period 15)
subjects combine Signal and Omega values to
forecast period-16 Omega.
                 
  Current period Current period            
    15 of 100       Time remaining 15 Time remaining 15  
                 
        Signal Omega      
      Period 8 10 64      
      Period 9 20 50      
      Period 10 10 24      
      Period 11 40 90      
      Period 12 10 36      
      Period 13 30 52      
      Period 14 20 66      
      Current period 15 10 48      
      Next period 16 30 ?      
                 
9
By contrast, Sequential treatment subjects first
observe a screen with Signal values only they
memorize Signal values and wait for the
corresponding Omega screen
                   
  Current period Current period              
    15 of 100       Time remaining 10 Time remaining 10    
                   
        Signal          
      Period 8 10          
      Period 9 20          
      Period 10 10          
      Period 11 40          
      Period 12 10          
      Period 13 30          
      Period 14 20          
      Current period 15 10          
      Next period 16 30          
                   
10
and once the corresponding Omega screen appears,
Sequential treatment subjects combine the
previously memorized Signal values with the
observed Omega values to forecast period-16 Omega.
                   
  Current period Current period              
    15 of 100       Time remaining 10 Time remaining 10    
                   
        Omega          
      Period 8 64          
      Period 9 50          
      Period 10 24          
      Period 11 90          
      Period 12 36          
      Period 13 52          
      Period 14 66          
      Current period 15 48          
      Next period 16 ?          
                   
11
Forecasting performance absolute forecast errors
The graph below shows 12-period moving averages
of absolute forecast errors, averaged across
subjects in each forecasting period, separately
for the Sequential and Concurrent treatment.
Sequential (average)
absolute forecast errors
Concurrent (average)
EARLY
LATE
12
Forecasting performance absolute forecast errors
The Concurrent treatment (with lower working
memory load) has lower absolute forecast errors
(on average) throughout the task but
statistically significant learning occurs in both
treatments between EARLY and LATE stages of the
task.
Sequential (average)
absolute forecast errors
Concurrent (average)
EARLY
LATE
13
Heterogeneity in forecasting performance
Same graph, with 10th and 90th percentiles added
to averages, reveals substantial across-subject
heterogeneity in absolute forecast errors in both
treatments. As hypothesized, working memory
should better explain the heterogeneity in the
Sequential treatment (with higher working memory
load). I focus on the LATE stage.
90th percentiles for Concurrent (o) and
Sequential ()
absolute forecast errors
Sequential (average)
Concurrent (average)
10th percentiles for Concurrent (o) and
Sequential ()
EARLY
LATE
14
Correlations between forecasting performance and
working memory
Concurrent treatment spearman -0.0006
(p0.997) pearson -0.2208 (p0.155)
As hypothesized, working memory is much stronger
negatively correlated with subjects LATE
absolute forecast errors in the Sequential
treatment (with higher working memory load). By
contrast, other measured cognitive, motivational,
personality and demographic individual
differences cannot explain between-subject
performance variation in the Sequential
treatment. I nevertheless control for these in
the formal analysis that follows
LATE absolute forecast error
working memory
LATE absolute forecast error
Sequential treatment spearman -0.3028
(p0.048) pearson -0.4540 (p0.002)
working memory
15
Testing the hypothesis that working memory is a
stronger determinant of forecasting performance
in the Sequential treatment

• I regress LATE absolute forecast error on
  working memory and other potential determinants
  of forecasting performance short term memory,
  math ability, intrinsic motivation, etc. The
  figure below shows only several selected
  specifications as explained by the labels.
• As some subjects performance is top-bounded, I
  use Censored Least Absolute Deviations (CLAD)
  estimator. So far I have 43 observations per
  treatment (students from Prague non-selective
  universities).
• The bars are coefficient estimates for working
  memory. The estimates for the Sequential
  treatment generally have economically meaningful
  magnitude. A yellow bar indicates that working
  memory has a significant impact on forecasting
  performance (at 10 level). A green bar indicates
  that, in addition, the impact of working memory
  is significantly larger in the Sequential
  treatment (at 10 level).

Estimation with working memory only
with short term memory and math ability added
with intrinsic motivation further added
with EARLY performance added as a proxy for
intrinsic forecasting ability (covariates
partialled out of the proxy)
same, but with an alternative working memory
measure that has short term memory and math
ability partialled out
Estimate for the hardest forecasting season only,
with covariates added
same, but with EARLY performance proxy added
16
Testing the hypothesis that working memory is a
stronger determinant of forecasting performance
in the Sequential treatment

• Conclusion The impact of working memory
  (cognitive capital) on forecasting performance is
  clearly stronger in the Sequential than in the
  Concurrent treatment
• but establishing this result more robustly may
  require more observations.

Estimation with working memory only
with short term memory and math ability added
with intrinsic motivation further added
with EARLY performance added as a proxy for
intrinsic forecasting ability (covariates
partialled out of the proxy)
same, but with an alternative working memory
measure that has short term memory and math
ability partialled out
Estimate for the hardest forecasting season only,
with covariates added
same, but with EARLY performance proxy added
17
CONCLUSION / ROAD AHEAD

• Working memory, or rather lack thereof, clearly
  presents a cognitive constraint on forecasting
  performance.
• In additional treatments (not completed), the
  working memory constraint is interacted with
  variation in financial incentives by offering
  subjects to purchase external memory at
  different relative prices subjects start in the
  harder Sequential treatment but can purchase
  switching to the easier Concurrent treatment.
• What individual characteristics, beside working
  memory, will determine buying behavior? Will more
  external memory be bought under higher financial
  incentives?
• In each period, subjects also bet on the quality
  of their forecasts, prior to placing a forecast.
  Financial return to betting is decreasing in
  subjects absolute forecast error.
• Especially psychologists have argued that
  performance in cognitive tasks is likely to be
  also affected by peoples confidence in their
  abilities. Bets provide a measure of confidence
  in ones forecasting abilities. Initial results
  suggest that working memory affects how quickly
  bets respond to improvements in forecasting
  accuracy.
• I will next investigate a two-equation system
  where both bets and performance are treated as a
  result of dynamic learning processes, using
  exogenous variation in the quality of forecasting
  feedback to identify the impact of bets on
  performance.