Nursing,%20Midwifery%20and%20Allied%20Health%20Professions%20Research%20Training%20Scheme%20Training%20Workshop%20May%202004

1
Nursing, Midwifery and Allied Health Professions
Research Training Scheme Training Workshop May
2004

• The Survey Method
• Paul Lambert
• Applied Social Science, Stirling University
• 5.5.04, 9-11am

2
Resources for this talk

• Slides
• the nature of survey research
• issues in doing survey research
• Reading guide
• Activities sheet
• introduction to research resources
• example analysis of a secondary survey dataset

3
The Survey Method
1. The Nature of Social Surveys Defining surveys survey sampling types of surveys history of surveys reactions to them
2. Doing Survey Research Research support sources Constructing data (collecting data secondary data working with variables) Data management and analysis

4
1. The Nature of Social Surveys
1a) Defining surveys
1b) Survey sampling
1c) Types of surveys
1d) History of surveys in the social sciences

5
1a) Surveys The systematic collection of
selected information from all or part of a
population(see Marsh 1982)
6
Surveys are characterised by variable-by-case
matrix
Cases ? ? Variables ? ? Variables ? ? Variables ? ? Variables ? ? Variables ? ? Variables ? ? Variables ? ? Variables ?
1 1 17 1.73 A . . . .
2 1 18 1.85 B . . . .
3 2 17 1.60 C . . . .
4 2 18 1.69 A . . . .
. . . . . . . . .
. . . . . . . . .
N
7

• Cases can be
• Any distinctive entity
• Most often, they are individuals (people)
• Variables are
• Measures of selected concepts of interest
• Indicators (our best guess at representing the
  concept)
• Variable design issues in choosing and
  formulating appropriate variables

8
Example
Cases ? ? Variables ? ? Variables ? ? Variables ? ? Variables ? ? Variables ?
Person Sex Age Height Health
1. Alan 1 17 1.73 A .
2. Bill 1 18 1.85 B .
3. Cath 2 17 1.60 C .
4. Dawn 2 18 1.69 A .
. . . . . .
N
9
The survey size

• Total number of cases ? survey size (n)
• A census covers every case in population.
• Most surveys use samples of cases.
• Larger survey size
• ? more reliable sample estimates.

10
Varieties of social surveys

• Topic choice of variables / cases
• Scale number of variables / cases
• Method data collection format
• Use type of data analysis descriptive vs
  inferential

11
Why study survey research?

• To undertake surveys
• If answers research question if attainable
• Valuable skills
• To understand / critique other peoples survey
  research based reports
• Crucial survey evidence is everywhere
• Dont just ignore / dismiss survey evidence

12
Strengths of surveys (1)

• Can be representative / large scale
• Probability theories justify generalisation from
  samples
• Surveys can handle census or other large sample
  data collections and analysis
• Parsimonious summary of the relation between
  variables on many cases

13
Strengths of surveys (2)

• Extensive methods research
• Eg, tests of reliability and validity - Bryman
  2001 pp70-74

Stability Face validity
Internal reliability Concurrent
Inter-observer consistency Predictive
Inter-observer consistency Construct
Convergent
14
Strengths of surveys (3)

• Variety of data analysis formats
• Descriptive, Inferential, multivariate
• Causal analysis defensible, eg longitudinal
• Data analysis is falsifiable
• Report writing skills, careful qualifications

15
Strengths of surveys (4)

• Accessibility of survey research
• Most research questions benefit from survey
  investigation
• Secondary datasets widely freely available
• Small scale surveys quick to conduct
• Survey results often convince others

16
Strengths of surveys (5)

• Surveys are less biased than most other social
  research methods
• Transparency of sampling methods variable
  construction data analysis
• Falsifiability
• Cynicism of receiving audiences..!

17
Examples surveys in health research

• General purpose and focussed cross-sectional
• Scottish Health Survey self-reported health and
  lifestyle of 4000 adults
• Selected population, eg sufferers of condition X
• Longitudinal follow-up studies
• Birth cohort studies parental backgrounds and
  childhood health progressions
• Ageing, status and sense of control (US) 1995
  sample of 3k in 1995, recontact 1.5k in 1998
• Experimental designs
• Smokers reactions to treatment programmes

18
1. The Nature of Social Surveys
1a) Defining surveys
1b) Survey sampling
1c) Types of surveys
1d) History of surveys in the social sciences

19
Role of sampling

• Surveys usually select only a sample of cases -
  aim to be representative of wider population
• Key idea is inference
• confidence in our ability to generalise
• Sampling inference application of statistical
  theories in order to estimate probabilities that
  a sample result is likely to have been
  unrepresentative

20
The normal (Gaussian) curve
21
Theories of sampling methods

• Sampling and probability theories tell us that
  any particular random sample is most likely to
  have the same properties as the wider population.
  We can then estimate the probability that sample
  results of a particular nature could have arisen
  by chance, rather than because they are the same
  as the (unknown) population result.

22
? If the cases in sample surveys were selected at
random, then can use sampling theories and thus
inference

23
Statistical inference

• ..causes confusion one of hardest parts of
  survey data analysis to understand..
• Phrases significance level p-value,
  confidence interval, hypothesis testing, ..
• Meaning Whether results would probably
  generalise to a larger population
• (if sample is treated as random)
• See Refs on reading list (esp Wright 2002)

24
Inferential data analysis

• Variable-by-case matrix data analysis for
  generalising findings to population
• Often distinguished from descriptive data
  analysis (results of sample only)
• Key joint influence of
• 1) size of sample
• 2) strength of data pattern
• in increasing confidence about generalisations

25

• Doing good inferential analysis is difficult
• Reliable sampling resources expensive
• Many early critiques of survey research concerned
  inappropriate inferential analysis
• Contemporary survey research tends to follow 2
  alternate strategies
• Large scale, often secondary, rigorous
  inferential methods
• or
• Small scale, primary, claims carefully qualified
  

26
Drawing samples (case selection)
Populn. Cases ? ? Variables ? ? Variables ? ? Variables ? ? Variables ?
1 - - - - -
2 - - - - -
3 - - - - -
4 1 1 17 1.73 A
5 2 1 18 1.85 B
6 - - - - -
7 3 2 17 1.60 C
8 4 2 18 1.69 A
N8 n4
27
Sampling methods

• Ways of selecting case from population

(i) Random (probabilistic) Generalisable, inferential statistics, fewer applications (ii) Non-random (opportunistic purposive) Harder to generalise, inference contested, more widely used
28
a) Simple Random Sample

• A statistical method used to choose cases
  randomly (eg random numbers)
• Every case in population has exactly the same
  chance of being in sample
• Most data analysis techniques initially designed
  for simple random samples

29
b) Systematic Random Sample

• Like the Simple RS, select cases from anywhere in
  the whole population
• An easier selection method choose every (n)th
  person for the sample
• Danger of periodicity if original population
  order has any structure, ? bias

30
Problems with sample methods selecting from whole
population

• The random part means its always possible to
  get a population coverage quite different from
  known structures
• If total population is large or dispersed, then
  coverage of random parts of it is expensive and
  time consuming few surveys use random sampling
  from whole of UK

31
c) Stratified random samples

• Modifies random sample to ensure intended
  coverage of population groups
• split sampling frame by stratification factors
• select random samples within each factor
• final sample has fixed proportions of each
• Example select 490 M and 510 F
• Properties proportionate sample correct
  representations but more expensive complex
  may need weights for analysis

32
d) Multistage cluster samples

• i) Select clusters of population at random
• ii) Sample randomly within clusters
• Eg clusters local authorities in UK
• With qualifications, may still be treated as
  random for analysis purposes
• Big reduction in costs if face-to-face contacts
• ? Most widely favoured sampling method in
  large scale survey collections

33
Example Multistage cluster sample

• Interest attitudes of Scottish school pupils
• Resources 400 interviews with pupils

34
Shetlands 2
Highlands 40
Islands 20
Moray 20
Aberdeen 40
Perth 20
Edinburgh 100
Argyll 24
Borders 10
Glasgow 124
35
Moray 40
Stirling 60
Edinburgh 150
Glasgow 150
36
Stirling 60
30 young people at Balfron School and 30 young
people at Stirling High
37
Issues in random sampling

• Only as good as underlying sampling frame (a good
  one may not be available, or not be as good as we
  think)
• Data analysis methods need adapting for
  stratified / clustered designs
• Other survey factors interact with sample
  selection issues, eg poor interviewers may
  discourage certain cases from response

38
ii) Opportunistic sampling

• Often in social research, sample design is
  opportunistic (purposive)
• Random sampling is expensive
• Random sampling is complex
• Some purported random samples are actually
  purposive anyway (understanding random)

39
a) Quota sampling

• Fill up quotas of groups of interest
• Quotas can ensure
• overall representation (cf systematic)
• broad topic coverage (eg types of voter)
• Example market researchers in street telephone
  call centres vetting contacts
• Biasses issues in how a quota fills up

40
b) Snowball sampling

• Also focussed enumeration
• Technique for contacting cases from populations
  rare / difficult to access
• Ask first obtained contact for suggested further
  contacts
• ? snowball gathers size
• Eg smaller ethnic minority groups
• Problem social networks are non-random!

41
c) Convenience sampling

• Samples whatever cases from population were
  easiest to reach, eg personal contacts
• Often no other sampling strategy involved
• Biasses likely in convenience process
• Examples most student survey projects are
  convenience..!

42
Random vs Opportunistic

• Random sampling difficult and expensive mainly
  government funded surveys
• Much data analysis / inference assumes random
  sample, but not applied to
• But random sampling is not a panacea...
• And opportunistic data is often robust

Rule Use survey documentation to report sampling
process and any errors
43
1. The Nature of Social Surveys
1a) Defining surveys
1b) Survey sampling
1c) Types of surveys
1d) History of surveys in the social sciences

44
Varieties of social survey designs

• Micro-social data
• Censuss
• Cross-sectional surveys
• Longitudinal surveys
• Cross-nationally comparative surveys
• Experiments or quasi-experiments
• Macro-social data
• Single summary statistics describing outputs
  from survey analyses

45

• Censuss
• General overview of whole population
• Disclosure risk issues
• Cross-sectional surveys
• Very widely used format
• Huge range of topic coverage
• Often used to study particular or rare
  subpopulations

46
Longitudinal datasets studies involving time

• Repeated cross-sections
• Chart changes over time, eg yearly means
• Panel and cohort samples
• recontact an initially random sample
• Learn about changers and causes of actions
• Problems of attrition
• Retrospective sample
• Rely on recall evidence of random selection
• Problems of selective recall
• Strengths understand process and causality
• Problems sampling and attrition complexity

47
Cross-nationally comparative datasets

• Focussed surveys (IPUMS censuss ISSP World
  Values Survey European Social Survey)
• Longitudinal studies (LIS ECHP CHER)
• Many analytical attractions, but issues of
  comparable analysis are complex

48
Experimental / quasi-experimental designs

• Experiment researcher intervenes in the process
  of study (quasi-experiment observe
  intervention)
• Dream of the randomised controlled trial
• Rare in sociology cost ethics more common in
  psychology certain health research fields
• Consequences
• Different methods of analysis (see eg Robson 2002
  c5)
• Less concern over inference / v large samples

49
Simple and complex survey data

• Simplest variable-by-case matrix has one sample
  of independent cases from 1 period
• More interesting social science data has more
  complex designs, eg
• Multiple records per case
• Relations between cases
• Experimental matching of designs

50
Working with complex survey data..

• Advanced tasks in data management
• File matching
• Variable transformations and treatments
• Advanced methods of data analysis
• Complex findings not easily summarised or
  communicated
• but it is simplicity of simple survey data that
  many people criticise about SDA

51
1. The Nature of Social Surveys
1a) Defining surveys
1b) Survey sampling
1c) Types of surveys
1d) History of surveys in the social sciences

52
Rise of the survey method..

• 1920s to 1960s saw vast expansion in survey
  examples
• Government, commercial, political, academic all
  conducting surveys on own topics
• Method of choice of social research
• Eg only empirical method, dominates teaching of
  research methods

53
Fall of the survey method..

• Early dominance of surveys produces backlash
  1960s onwards (eg Cicourel 1964)
• Philosophical research epistemology
• Pragmatic many badly conducted surveys
• Prejudiced fear of working with statistics
  leads researchers to avoid, ignore or oppose
  surveys
• Political white male power
• Populist lies, dammed lies and statistics

54
..Resurgence of survey method

• Critiques provoke defence
• Methodological research to avoid simplifying
  mistakes, improve sampling designs, and avoid
  unjustified claims
• Substantial support for survey research always
  there, eg Government, commerce
• Funding organisations now try to redress
  over-reaction, encourage survey research teaching
  and activities (eg ESRC)

55
Current state of play

• Survey and other research methods are largely
  separate entities
• Pragmatic Involve different people and skills
• But not methodologically appealing (eg Bryman
  1992 on mixed methods, extract in Seale 2004)
• Contemporary social science survey research shows
  strong disciplinary separatisms, eg
• UK sociology tables bivariate US sociology
  regressions Psychology Anova Economics
  regression extensions. (Accusations of
  methodolatry)

56
Importance of studying Surveys

• Whatever your views on methods, surveys are
  always will be one of most important social
  research tools - they should never be ignored!

57
The Survey Method
1. The Nature of Social Surveys Defining surveys survey sampling types of surveys history of surveys reactions to them
2. Doing Survey Research Research support sources Constructing data (collecting data secondary data working with variables) Data management and analysis

58
2. Doing Survey Research

2a) Research support
2b) Constructing data
2c) Data management and analysis
59
Research support with surveys

• References Hundreds of textbooks and papers
• See reading list, esp Buckingham and Saunders
  2004
• Internet resources, eg ESDS, RMS
• See activities sheet
• Learn from prior examples qnre schedules,
  samples,
• Learning by doing
• the apprentice model of social survey research
• workshops and short courses

60
2. Doing Survey Research

2a) Research support
2b) Constructing data
2c) Data management and analysis
61
Data construction issues

• 2.1a Collecting and entering data
• 2.1b Accessing secondary data
• 2.1c Variable operationalisations

62
2b -1 Good practice in survey collection

• Extensive history of methodological research into
  how to collect data with greater reliability and
  validity
• - see some egs on reading list (..De Vaus chpts
  7-8 Gilbert chpt 6..)

63
Data collection issues

• Appropriate sampling strategy (adhered to)
• Question wording
• Interviewer skills and contexts
• Minimise non-contacts / refusals
• Document data collection assiduously
• every badly conducted survey makes it harder for
  surveys in the future

64
Questionnaire data entry

• Codebook devise system to represent variables
  through numeric codes
• Choices in variable operationalisations and
  categorical groupings
• Small scale tap values into computer from
  schedule (laborious)
• Large scale data entry software / specialist
  staff

65
2b-2 Secondary survey data

• Vast quantity of surveys conducted
• An efficient step would be to analyse existing
  data (secondary) rather than personally collect
  your own (primary)
• Data archives collate survey datasets and supply
  them for secondary analysis

66

• Government funds many large surveys
• (also EU LAs charities commercial)
• Often made available freely or at low cost
• ? An ideal research tool (see ESRC)
• Quick to access
• Methodological rigour in sample questionnaire
  design, interview collection
• Falsifiable others can access also
• Generalisable larger scale
• Multivariate

67
Secondary analysis of surveys

• Makes particular sense when large scale datasets
  are desirable
• Also often applies to smaller surveys
• Involves particular issues of data analysis,
  management and interpretation
• Is a highly marketable skill!

68
Accessing Secondary datasets

• Internet and computing developments have
  revolutionised delivery of data resources (see
  activities sheet)
• Three steps to data access
• Find out survey details / documentation
• Apply for access from archive or collectors
• Obtain and analyse the data

69
Some drawbacks

• Distance from data collection
• Harder to assess reliability / validity
• Many variables already pre-coded
• Cant change / add anything in study
• Time delays between collection results
• Data analysis / management complex
• May be bracketed with survey originators

70
2b-3 Variables in data analysis

• Variable operationalisation key to surveys
• Choices - in initial data collection
• - in data recoding / analytic
  treatment
• Existing comment / research on many widely used
  variables (eg Burgess 1986)
• Critiques of survey research most often
  concentrate on variable operationalisations

71
Eg Education and occupation

• Education
• Changing levels of education over time
• Education as proxy for ability, intelligence?
• Occupation
• Contested meanings of labour market status
• Occupational indicators of stratification
• Occupational gender segregation

72
Variable operationalisations

• Methods guidelines on appropriate handling
• Harmonised concepts and questions textbooks
  papers / debates specific issues
• Choices / approximations always used
• Research reports and methods appendices must
  explain and justify position taken

73
2. Doing Survey Research

2a) Research support
2b) Constructing data
2c) Data management and analysis
74
Data management and analysis

• 2c-1 Manipulating variables and cases
• 2c-2 Mainstream techniques of data analysis
• 2c-3 Robust data analysis

75
2c-1 Data management

• ..is core skill in using primary and secondary
  surveys
• Occupies more time than analysis in most cases
• Main techniques cover
• Matching data files
• Coding / transforming variables
• Dealing with missing data

76
Data management

• Software packages SPSS, SAS, STATA, .. with
  wide ( revolutionary) capabilities
• Good and bad practices
• Keep logs and records of transformations
• Follow previous literature for best variable
  treatments
• Secondary dataset management tends to be
• More complex ?
• More error prone ?
• Subject to external scrutiny ?

77
2c-2 Techniques of data analysis
Ways of summarising the relationship between
variables

• Good practice
• Reflects properties of variables
• Describes output in appropriate context
• Bad practice
• Forcing data into style of analysis
• Attributing false properties to data
• Over zealous conclusions

78
Three definitions locate nearly all SocSci
techniques of QnDA

• i Level of measurement of variables
• ii Number of variables being analysed
• iii Aims of analysis (descriptive cf
• inferential)

79
i) Key organising principle Level of measurement
of a variable

• 4 levels classically (eg Blaikie 200322-7)
• Nominal (gender ethnicity)
• Ordinal (exam grades attitudes)
• Interval (age height)
• Ratio (some monetary measures)

80
Levels of Measurement

• Really only 1 distinction is most important to
  analysis opportunities
• Categorical nominal and ordinal
• Metric interval and ratio
• There is also some flexibility in assigning
  levels (eg Blaikie 2003 can usefully treat many
  vars in different forms)

81
ii) Number of variables

• Univariate Patterns in the distribution of cases
  within one variable (summary stats)
• Bivariate Relations between patterns in the
  distribution of cases over two variables
  (crosstabs and association measures)
• Multivariate Relations between patterns in the
  distribution of cases over three of more
  variables (complex crosstabs statistical models)

82
Statistics numerical summary indicators of the
properties of variables

• Univariate Statistics describe the distribution
  of one variable
• Bivariate / Multivariate Statistics describe how
  the distribution of two / more variables are
  related

83
iii) Aims of analysis

• Descriptive and explanatory
• Summarise patterns in sample / data
• Use statistical procedures to highlight real
  patterns
• Examine how some variable patterns effect
  others
• End in itself for some data (eg census)
• Inferential
• Assess the possibility of generalising our
  findings
• Prominent in sample survey analysis, but needs
  qualifications (eg a random sample)

84
Inference and statistics

• Key idea of inference is to make a qualified
  estimate of degree of confidence in our ability
  to generalise
• Sampling inference application of statistical
  theories in order to estimate probabilities that
  a sample result is likely to have been
  unrepresentative

85
Common QnDA techniques by typology (c/m
categorical/metric descriptive / inferential)

Frequency table c Mean, standard deviation m
Cross-tabulation cc Confidence interval m
Chi-square test cc Scatterplot mm
Multiple crosstab 3c Table of means mc
Boxplots mc Anova mc
Logistic regression 2 c,m Multiple regression 2 m,c
86
Example Ginn 2001, p293 in Gilbert
multivariate, categorical, descriptive
multiple cross-tabulation
Men Men Women Women
1987 1993 1987 1993
column percents, adults 20-59 column percents, adults 20-59 column percents, adults 20-59 column percents, adults 20-59
Occup pension 46 40 22 25
APP private pension n/a 13 n/a 9
Work but no pension 25 12 38 27
Self-emp 14 15 5 5
Not employed 15 20 35 34

87
2c-3 Robust Data Analysis

• Incorporates
• Use appropriate data analysis for resource
• Advanced data analysis techniques specifically
  sensitive to survey data
• Report analysis results appropriately

88
Appropriate data analysis

• Reflect appropriate levels of measurement
• Keep categories in defensible forms
• Think about social theories behind analysis
• Analysis ought to be multivariate
• Remember missing data issues

89
Appropriate data analysis

• The biggest controversy dont misuse inferential
  statistics
• Strictly should only be for random
• But can be defended for non-random, with
  appropriate language / qualifications

90
Advanced data analysis..

• Statistical research to improve accuracy of
  social survey analysis, proposes eg
• Weighting (more complex than appears)
• Missing data models
• Multilevel models
• Latent variable analysis
• Selection models
• Other difficult models

91
Report results appropriately

• Avoid over-zealous claims
• Care about term causality
• Use simpler language, tables and graphs
• Give methodological details, eg Appendix
• Many report writers have become very good at this
  hard to accuse of bad practice

92
..the End
1. The Nature of Social Surveys Defining surveys survey sampling types of surveys history of surveys reactions to them
2. Doing Survey Research Research support sources Constructing data (collecting data secondary data working with variables) Data management and analysis