Logic, validity, reliability

1
Logic, validity, reliability

• Prof Hal Swerissen
• Director
• Australian Institute for Primary Care

2
Logic
3
Program logic
Needs
4
Basic Program Evaluation
Needs
Inputs
Activities
Outputs
Impact
Outcome
Cost
Impact
Process -implementation
Process - output
Outcome
5
Basic program evaluation
Needs
Inputs
Activities
Outputs
Impact
Outcome
Cost
Impact
Outcome
Process-implementation
Process-output
Technical efficiency
Allocative efficiency
6
Attribution intervention
Problems/ needs
7
Figure 1 Primary Care Partnership program logic


Partnership

Service processes

Consumer impact

Outcomes


Better Access to Services (BATS)

Information Management (IM)



Local Service Information (LSI)

DHS Support

Consumer

knowledge


Service

Consumer

attitudes

co ordination

and community

action


health and

experience


clinical indicators

well being

Partnership

functional indicators

Integrated

Strategy

Service

Planning


Protective

factors


Integrated Service Plans (IS)

Health
Promotion Strategy (HP)

Integrated Disease Management (IDM)



Service Linkage Strategy

Partnership Str
ategy

GP Integration Strategy

Quality Improvement/Self

Assessment and Evaluation (QI)

8
Validity
9
Internal validity (cause effect)

• Did program cause observed outcomes?
• Threats to internal validity include
• history
• maturation
• sleeper effect
• backsliding
• trigger
• backlash

10
Causality and time
11
History effect
Impact/outcome
Other event
Program
Time
12
Sleeper effect
Impact/outcome
Intervention
Time
13
Backsliding effect
14
Trigger effect
15
Maturation effect
16
Backlash
Impact/outcome
Intervention
Time
17
Controlling internal validity

• Comparison groups
• Randomised
• Matched
• Statistical controls
• Time series and replication
• Logical sequence analysis

18
Comparison groups
Time 1 Intervention
Time 2 Intervention
Intervention
Time 1 Comparison
Time 1 Intervention
19
Comparison groups
Intervention
Effect
Comparison
Time
20
Comparison groups

• Randomization
• Selection (for external validity)
• Allocation (for internal validity)
• Matching (by selection)
• Statistical (post hoc statistical adjustment)

21
Statistical adjustment
Intervention
Effect
Time
22
External validity

• Can program effects be generalized to other
  settings
• Participant characteristics
• Program characteristics

23
Participant characteristics

• Selection bias
• Demographic
• Health /or social issues

24
Program characteristics

• Resource levels
• Competence expertise
• Context

25
Reliability
26
Definition

• Measuring the right thing
• Measuring the right thing accurately over time

27
Threats to reliability

• Measurement error
• Halo effects
• Hawthorne effects
• Response bias
• Observer effects
• Observer bias
• Non responses
• Theoretical (domain) sampling error
• Random variability

28
Controlling for reliability

• Defining measurement domains and indicators
• Multiple measurement methods (triangulation)
• Observational
• Unobtrusive
• Self report
• Institutional
• Repeat measurement