Qualitative research on weather and society: Two example projects

1
Qualitative research on weather and
societyTwo example projects

• Rebecca E. Morss (NCAR)
• Project 1 Marty Ralph (Roger Pielke, Jr.,
  Kristen Koch)
• Project 2 Fuqing Zhang, Texas AM students

2
Two small-scale, example projects

• Use of information by National Weather Service
  (NWS) forecasters and forecast users during a
  meteorological field program
• ? semi-structured, qualitative interviews
• ? participant observation
• Texas residents evacuation decisions in
  Hurricane Rita, perceptions and use of Hurricane
  Rita forecasts
• ? structured quantitative/qualitative
• interviews (oral surveys)

3
Outline for each project

• Background and motivation for project
• Research questions and project focus
• Research methods
• Data collection
• Data analysis
• Sampling, validity and reliability of results
• (Results)
• Close with suggestions

4
Field programs CALJET/PACJET

• CALJET California Land-falling Jets Experiment
  (Dec 1997 Mar 1998)
• PACJET-2001 Pacific Land-falling Jets
  Experiment (Jan Mar 2001)
• Goal Take observations in landfalling West Coast
  winter storms, to improve scientific
  understanding and weather forecasts
• Programs included researcher interactions with
  NWS forecasters and forecast users ? What is the
  value of the additional observations?

5
L
6
Research questions and focus

• How did NWS forecasters and forecast users
  (especially emergency managers) use additional
  information provided by CALJET and PACJET?
• Because understanding use of additional
  information requires understanding context
• How do NWS forecasters and emergency managers
  use information in general? (focusing on
  precipitation in California)

7
Research methods Data collection

• (Previous work Researcher discussions with,
  anecdotes from forecasters and users)
• My work
• During PACJET-2001 (FebMar 2001)
• Interviewed forecasters, researchers, and
  end-users about CALJET and PACJET, etc.
• Observed NWS forecasters use of information
• (Later work Web log and survey of forecasters
  use of PACJET observations in 2002, 2003)

8
Interviews

• Qualitative, semi-structured interviews with
• NWS forecasters
• NWS personnel (primarily managers)
• CALJET/PACJET researchers
• Emergency managers and water resource managers
• Followed an interview guide with relatively
  open-ended questions, allowing for flexibility
• Interview questions evolved as data collection
  progressed and research grew more focused

9
Observational methods

• A continuum
• Quantitative, structured observations
• Qualitative, unobtrusive observations
• Participant-observation

10
Observations of forecasters

• Qualitative participant-observation of NWS
  forecasters at Hydrometeorological Prediction
  Center 2 Weather Forecast Offices in California
• Observed forecasters conducting their regular job
  duties
• Asked forecasters to explain what they were doing
  and why
• Interspersed questions and discussion
• Sessions during ordinary operating periods and
  landfalling winter storms

11
More data collection

• Additional data collected from
• Informal interviews and discussions with NWS
  personnel, NWS forecasters, CALJET/PACJET
  participants, and users
• Written documents, including forecast products
  (no formal textual analysis)
• All data recorded by handwriting notes and typing
  up afterwards
• (No tape or video recording, to limit
  intrusiveness)

12
Why these methods?

• Study was exploratory
• (In other words, we did not know what specific
  questions to ask using more structured or
  quantitative methods)
• Wanted to understand, in detail, how and why
  people used information in different situations
• Wanted the real, inside view
• Wanted to understand use of information during
  hazardous weather situations small sample of
  events, forecasters are busy

13
Data analysis

• Read data to identify initial themes (codes)
• Cycle of
• Coding data
• Reorganizing data by code
• Rereading data
• Modifying and refining codes
• (Using Microsoft Excel)
• Additional analysis by diagramming concepts and
  relationships
• Revisited analysis cycle during writing

14
Sampling

• In this study
• Sample of forecasters was not representative (was
  biased towards certain types of forecasters)
• Sample of users was small, based on
  recommendations (snowball method)
• Samples not large enough to draw generalizable
  conclusions, but still generate interesting
  knowledge

15
Validity and reliability of results

• Many definitions of quality in qualitative
  research
• To improve validity and reliability (reduce bias)
  in results from this study
• Tried to limit assumptions when designing study,
  collecting data, and analyzing data
• Recorded data systematically
• Analyzed data systematically, checked ideas
  against data
• Collected data from different sources and
  triangulated
• Checked results with a key source

16
Project 2 Hurricane Rita

• Motivation and research questions
• Following Hurricane Rita in September 2005, a
  group of meteorology students at Texas AM
  University was interested in learning about Texas
  residents evacuation decisions, perceptions and
  use of hurricane forecasts
• Research methods
• Structured interviews, including quantitative and
  open-ended questions
• Basic quantitative and qualitative data analysis

17
(No Transcript)
18
Data collection Instrument

• Developed structured interview guide, including
  questions from students and previous studies
• Texas AM researchers trained students (briefly)
  on interview methods
• Obtained Institutional Review Board approval
• Students pre-tested interview guide
• To pre-test questions
• To obtain interview practice
• Revised interview guide based on pre-tests

19
Interview questions

• Location during Hurricane Rita? Residence
  location? Primary household decision-maker?
• Preparation, evacuation, damage for Hurricane
  Rita? For previous hurricanes?
• Reasons for evacuation decision
• Influence of Hurricane Katrina on evacuation
  decision
• Act differently next time?
• Concern about different hurricane-related threats
• Sources and perceptions of Hurricane Rita
  forecasts
• Demographic questions

20
Interview question format

• 13. Looking back on your decisions as Hurricane
  Rita approached the coast, is there anything you
  would do differently next time?
• Yes No
• If yes 13a. What would you do differently
  next time, and why?
• (Generates both quantitative and qualitative data)

21
Data collection Sampling

• Selected interview locations
• Galveston, TX
• Port Arthur, TX
• Secondary southwest Houston, TX

22
Houston
Port Arthur
Galveston
23
Data collection Sampling

• Selected interview locations
• Galveston, TX
• Port Arthur, TX
• Secondary southwest Houston, TX
• Students conducted 120 interviews (total) over
  1-2 days in each location
• Interviewees identified through opportunity
  sampling
• Recorded responses in writing (and on tape)

24
Data analysis and interpretation

• Quantitative results analyzed using simple
  methods (such as binning)
• Qualitative results analyzed using simple
  categorization
• More advanced data analysis not performed due to
  small sample and potential reliability issues
  with student-collected data
• Results not generalizable due to convenience
  sample

25
Suggestions

• Identify clear, manageable research questions
  that can be investigated using desired methods
  within intended time period and available
  resources
• but be open to new questions as study develops
  (for now and for future work)
• Develop a data collection and sampling strategy
  that fits goals and resources
• Limit assumptions as much as possible
• Keep clear, consistent records of everything
• Allow time for planning study, training,
  pre-testing questions, data analysis
• Where possible, take advantage of local resources

26
QuestionsandDiscussion
27
Results of CALJET/PACJET study

• Document what types of information NWS
  forecasters use how they combine information to
  generate forecasts
• Describe data to societal benefit links
• in potentially hazardous weather situations
• forecasters roles in translating data ?
  societal benefit
• Identify several of CALJET/PACJETs challenges
  and reasons for success

28
Results of Hurricane Rita study

• Still in progress