Measuring Confidence in the Police: Lessons from Research into Survey Question Design

1
Measuring Confidence in the Police Lessons from
Research into Survey Question Design

• Rethinking Confidence in the Criminal Justice
  System Conference, 13th November 2008, Newcastle
  University.
• Stephen Farrall, School of Law, Sheffield
  University

2
Outline

• Outline (v. quickly!) JUSTIS.
• Review (again v. quickly!) my past research on
  survey question design.
• Review what such research tells us about how we
  ought to approach measuring confidence.
• Critique current measure of confidence (PSA 23)
  from this standpoint.
• Tentatively probe some ways forward.

3
What is JUSTIS? Whatll it do?

• EU FP7-funded project (2008-11).
• 9 partners 7 EU countries.
• Design batteries of Qs to measure confidence
• in the police/CJS.
• Translate these into common European languages.
• Disseminate these via various forums.

4
Past Research (on measuring the fear of crime)

• Since mid-1990s I have explored in a number of
  studies how survey
• questions on the fear of crime operate. What
  have we learnt from
• studies into measuring the fear of crime?
• Standard survey Qs may not accurately capture
  respondents fears about crime.
• They appear to make it seem a very commonly
  encountered emotion.
• Problems with the questions include starting
  with how which appears to inflate answers.
• Respondents can give answers for other people not
  just themselves (crime is a problem for old
  people).

5
What have we tried to do since?

• Redesigned fear of crime surveys to use a filter
  and a time frame (Have you felt worried about
  in the past year?).
• Then asked a series of follow-ups (how much?, how
  often? could also ask about impacts etc).

6
What has been the contribution to our knowledge?

• Instead of there being 25-33 who are worried, we
  find 5-8 who have worried four or more times in
  the past year and were very/quite worried on the
  last occasion.
• Changes the picture of the fear of crime quite
  dramatically greater accuracy of which groups
  feel fearful.
• Demonstrates the importance of thoughtful
  question design.

7
What can research into survey Qs teach us about
designing good Qs?

• Let us start by thinking about what confidence
  is/isnt
• Faith or trust in someone or somebody. Certainty
  or strong expectation. (Penguin Dictionary)
• Present/Future orientated (so really a measure of
  certainty about something which at the time of
  questioning is unknowable more like a hunch
  shaped by past experiences perhaps?
• Not an experience (although again shaped by past
  experiences, both ones own and vicarious).
• It isnt satisfaction (since this is past
  orientated).

8
How then to go about measuring What might
research into survey Qs suggest to us?

• Tourangeau et al (2000) suggest there are four
• components to answering survey Qs
• Comprehension
• Retrieval
• Judgment
• Response

9
Tourangeau et als four components

• Comprehension Rs need to correctly interpret the
  Q need to react appropriately to instructions or
  specific directions (in the past year). So
  avoid double-barrel Qs or unfamiliar or vague
  terms, and terms which can be various
  interpreted.
• Retrieval recalling memories or general
  impressions to help answer the Q.

10
Tourangeau et als four components

• Judgment R assesses (judges) the accuracy of
  their recall inferences based on the process of
  retrieval integration of these products into a
  single overall assessment adjustments for
  omissions in retrieval. People may withhold an
  answer if they think they dont hold sufficient
  information they may base their answers on
  easily recalled episodes they may rely on
  stereotypes.
• Response selection choosing a response code to
  best represent their answer.

11
So what? What do we need to bear in mind?

• Some Qs are hard for Rs to understand.
• Not all events will be recalled equally well.
  More salient events and more recent events are
  over-sampled.
• Anchoring Rs select a middling answer and
  adjust up/down the scale based on what they
  a) recall and b) judge about those memories.

12
Let us look at the standard survey measure of
confidence in the police.
It is the responsibility of the police and local
council working in partnership to deal with
anti-social behaviour and crime in your local
area. How much would you agree or disagree that
the police and local council are dealing with the
anti-social and crime issues that matter in this
area? Codes Strongly agree/Tend to
agree/Neither agree nor disagree/Tend to
disagree/Strongly disagree
13
What might be wrong with this question? I
Ambiguous words or phrases

• deal with (actually tackling or simply trying
  to tackle?)
• anti-social behaviour (was this ever defined?)
  
• local area (How defined? Why just there? Why
  not elsewhere? Why assume that we spend most of
  our waking time in the area our homes are in?)
• crime issues (Which ones? Standardisation of
  interpretation assumed?)
• that matter (To whom? The R? Someone else?
  Standardisation of interpretation assumed?).

14
What might be wrong with this question? II
Presuppositions

• That there are crime issues that matter to
  the R in this area.
• That the R does agree or disagree with the
  referent of the Q (How much implies that the R
  does the uncertainty is over the extent to which
  they agree or disagree).

15
What might be wrong with this question? III
Ease of selecting an answer

• R thinks that the police are doing a great job,
  but that the local council is doing a poor job.
  Which code do they go for?
• Why ask about both the local council and the
  police?
• Why ask about both anti-social behaviour and
  crime?
• Isnt this FOUR Qs, not ONE?

16

What might be wrong with this question? IV
General Biases

• The existing Qs (e.g. PSA 23) may provoke
  unreflective answers because Rs are asked poorly
  designed Qs which appear to be unrelated to the
  sorts of immediate problems people want the
  police to response to.
• Too abstract? Not grounded enough in easily
  imaginable scenarios?
• Too easy to answer without much thought?
• Too easily contaminated by identity issues?

17
Solution I The Obvious Tricks

• The above item is crude (most single item Qs
  are).
• Using batteries of Qs is a better strategy (but
  costs more) and also (often) correlates with the
  single item (so a waste of time?).
• Using batteries takes one into the dark arts of
  scale construction and factor analysis (or
  something similar) so requires extra work and
  expertise.

18
Solution II Make Confidence in the Police
Bigger than the Police?

• Confidence in the police, isnt just about what
  the police do.
• It is also about what they might do if you needed
  them to act.
• People might feel empowered to address crime(s)
  if they believe that the police/CJS support them.
• People might be more willing to report crimes to
  the police if they feel that the police will
  properly and efficiently record such reports.

19
Big health warnings here

• Weve not piloted the following Qs.
• These may not be what we end up suggesting.
• Wed recommend that you DONT use these Qs (at
  least for the time-being).
• The JUSTIS project will keep people informed via
  its web site.

20

Using vignettes to get people thinking more
deeply about confidence.

• Vignettes require Rs to think more carefully
  about a specific situation.
• Through careful design researchers can control or
  manipulate key variables.
• Rs answers embedded in wider social contexts.

21

Example Q I

• Imagine that you came home to find that your
  home
• had been burgled. How likely do you think it is
  that
• the police will be able to
• Advise you on how to prevent similar things
  happening again?
• Capture the people who burgled your home?
• Correctly complete all the paperwork associated
  with your case?

22

Example Q II

• Imagine that whilst waiting at a bus stop one
  day, you
• saw a man steal an old womans handbag. You call
  the
• police immediately. How likely do you think it is
  that
• the police will be able to
• Get a uniformed officer to the scene before the
  man ran off?
• Eventually help the old woman recover her
  handbag?
• Help the old woman feel safe again?

23

Example Q III

• Imagine that the police arrest 16 year old child
  for a
• crime which he hadnt actually committed. How
  likely
• do you think it is that
• The Police would quickly realise their mistake
  and release the child?
• The Police would make sure that the child was
  properly represented during questioning?
• The Police would apologise to the child and his
  parents for all the trouble theyd been put to?

24
Keeping in touch with JUSTIS

• Web site http//www.eurojustis.eu/index.php.
• Project leader Prof. Mike Hough, Kings College
  London.
• For further contacts and details, please e-mail
  info_at_eurojustis.eu.