Conciliating scientific rigor

1
Conciliating scientific rigor pragmatics in
Outcome Evaluation Research for human service
professionalsTheoretical underpinnings and
implementation techniquesNetwork for Health
Welfare StudiesDr. Charles C. ChanDr. Amy P.
Y. HoMr. Kevin ChanFebruary 25, 2004
2
Dr. Charles C. Chan

• Convenor
• Network for Health Welfare Studies
• Associate Professor
• Department of Applied Social Sciences

3
Preamble

• Applied research can be of better service to the
  human service industry, especially in a quality
  conscious era, if it promises to pay balanced
  attention to the dialectical relationship between
  the professional / researcher and the service
  recipient.

4
Quality and Effectiveness

• Strange bed-fellows or estranged couples?

5
Quality and Effectiveness

• The overall all goal of placing priority on both
  quality and effectiveness of health programs is
  to document the synergistic contribution of its
  component parts to a holistic improvement in
  health and welfare of the population.
• WHO-EURO Working Group on Health Promotion
  Evaluation (1998)

6
Quality and Effectiveness

• The focus should not be on documenting quality or
  evaluating effectiveness of isolated
  interventions as end points, but rather on the
  relationship of a given intervention to the other
  components of the health promotion strategy. Such
  an analysis may indeed provide a fresh approach
  to the very issue of quality.

7
Definition of the Balanced Applied Research
Development Model (BARD)

• BARD is a tested approach in applied research to
  address the demands of scholarship of application
  and community benefits
• It recognizes problems central to applied
  research
• It contributes by directing decision making in
  the research process

8
Main features of BARD

• Placing value on a balanced approach to the
  scientific requirements and the pragmatics of
  outcome evaluation in human services

9
Main features of BARD

• Emphasizing integrating research into practice by
  transferring skills in data capturing to
  professionals or lay persons in service delivery

10
Main features of BARD

• Analyzing as much as possible, the triadic
  relationship between the professional, lay
  person, and service recipient and sustaining
  intervention effects through supportive
  supervision

11
Efficacy Vs Effectiveness

• Efficacy Intervention does more good than harm
  under optimum conditions
• Effectiveness - Intervention does more good than
  harm under real-world conditions
• (Flay, 1986)

12
The RE-AIM framework (Glasgow, 1999)

• Reach
• Efficacy or Effectiveness
• Adoption
• Implementation
• Maintenance and cost

13
The RE-AIM framework (Glasgow, 1999) - Contd
14
Outcome Evaluation Research in Hong Kong
Financial Landscape

• Health Services Research Fund (1995 - 2003)
• Health and Health Services Research Fund (2003 -
  present)
• Research Fund on Control for Infectious Diseases
  (2003 present)
• Community Investment Inclusion Fund (2003 -
  present)

15
Health and Services Research Fund (1995-2003)

• Capital amount of 55M
• Total number of submission 1096
• Total number of approved HSRC projects224

16
Health and Health Services Research Fund (2003 -
present )

• Capital amount of 10M
• Total number of submission 187
• Total number of approved projects4 (ESGAA
  meeting in July 2003)

17
Research Fund on Control for Infectious Diseases
(2003 present)

• Capital amount of 500M (50M given to PRCs
  Ministry of Health for SARS research)
• Open call and commissioned research activities
  tentative 30M to HKU in basic and epidemiological
  modeling research 25 M to CUHK on public health
  and emerging infectious diseases drug and
  treatment development research
• Leaving sums for the establishment of the Centre
  for Health Protection

18
Community Investment Inclusion Fund - Funding
information
19
Community Investment Inclusion Fund - Funding
information -Contd
20
Dr. Amy P. Y. Ho

• Member
• Network for Health Welfare Studies
• Senior Lecturer
• Department of Applied Social Sciences

21
OER in human service setting An example

• Promoting health and well-being of elderly
  patients with chronic illness A coordinated
  medical and social service program
• Funded by the Health Care Promotion Fund

22
Intervention flowchart
23
Intervention components
24
Issues in unblinded study

• Feasibility of blinding in human service research
• Artifacts introduced by confounders in unblinded
  study

25
Standardization of intervention

• Contrasting the notion of experimental group in
  the context of medical research and human
  services research
• Black Box phenomenon in health promotion
  research (Moore, 2003)

26
Control on extraneous variables

• Between-group imbalance in the control of
  extraneous variables
• Incorporating extraneous variables in the analysis

27
Data quality and reliability

• Staff turnover
• Artifact due to unblinded group assignment
• Reliability of self-reported outcomes

28
Issues in data analysis

• Loss to follow-up
• Causes of participants attrition in human
  services OER
• Effect of attrition on data analysis

29
Mr. Kevin Chan

• Research Associate
• Network for Health Welfare Studies
• Department of Applied Social Sciences
• Honorary Scientific Officer
• The Hong Kong Childhood Injury Prevention
  Research Association

30
Presumption of the experimentalist ideal (Smith,
1985)

• OER in human services are more than often
  violating the assumptions required in traditional
  RCT convention
• Such difference should not be viewed as a
  deviation from the conventional approach, but
  rather a calling for applied research with
  stronger ecological validity

31
In fulfilling requirements set for RCT, we often
undermine the following (Rolls, 1999)

• The difficulties associated with the random
  assignment
• The ethical and administrative objections to
  randomization (de_Raeve, 1994)
• That inputs are rarely stable
• That awareness of difference introduces bias
• That it is almost impossible to exclude
  extraneous variables
• The research is unable to say why changes
  detected have occurred (Newell, 1992)

32
Troubleshooting to barriers OER in human
service setting

• Dealing with less-than-perfect random assignment
• Adjusting for unstable input in human services
  OER
• Statistical control for unblinded studies
• The inclusion of extraneous variables as
  covariates
• Narrowing down to a closer approximation of
  causal relationship

33
Troubleshooting to barriers Group assignment in
OER for human service setting

• Balancing incentives for follow-up of control
  group
• Intention-to-treat analysis
• As-treated analysis
• Complier-Average Causal Effect (CACE) analysis

34
Troubleshooting to barriers Group assignment in
OER for human service setting Contd

• Balancing incentives for follow-up of control
  group
• Despite the lack of intervention, control group
  participants could be prescribed with
  non-specific intervention or substantial
  support to maintain their interests to comply
  with the data collection process

35
Troubleshooting to barriers Group assignment in
OER for human service setting Contd

• Balancing incentives for follow-up of control
  group
• Non-specific intervention
• -E.g. Delivery of printed health promotion
  materials
• Substantial support
• -E.g. Safety device, health supplements

36
Troubleshooting to barriers Group assignment in
OER for human service setting Contd
37
Troubleshooting to barriers Group assignment in
OER for human service setting Contd

• Complier-Average Causal Effect (CACE) analysis
• (Angrist, 1996 Little, 1998)
• Expanding the conventional regression equation
  with a new term compliance with intervention

38
Troubleshooting to barriers Group assignment in
OER for human service setting Contd

• Complier-Average Causal Effect (CACE) analysis
• (Angrist, 1996 Little, 1998)
• Difference of evaluation outcome (e.g. score on
  SF-36, probability of sustaining home injury)
  between compliers and non-compliers, with other
  missing information (e.g. covariate variables
  such as age, gender, health status, attitude
  toward the research question) adjusted and
  stratified by compliance category.

39
Troubleshooting to barriers Group assignment in
OER for human service setting Contd

• Complier-Average Causal Effect (CACE) analysis
• (Angrist, 1996 Little, 1998)
• CACE accounts for potential interaction between
  randomized group assignment compliance and
  allows more accurate estimate of program
  effectiveness by
• Increasing number of usable cases
• Subtract the artifact of compliance that
  moderates the outcome variable

40
Troubleshooting to barriers Adjusting for
unstable input in human services OER

• Explicit definition of intervention protocol
• Behavioral intervention
• Substantial input
• Personnel input
• Temporal data (Date, time, duration, frequency)
• Define boundary of intervention

41
Troubleshooting to barriers Adjusting for
unstable input in human services OER
42
Troubleshooting to barriers Adjusting for
unstable input in human services OER
43
Troubleshooting to barriers Adjusting for
unstable input in human services OER
44
Troubleshooting to barriers Adjusting for
unstable input in human services OER
45
Troubleshooting to barriers Adjusting for
unstable input in human services OER Contd

• Process analysis
• Process evaluation fills up the Black Box
  (Moore, 2003) left void in health promotion
  research and strength internal validity of the
  intervention under investigation.

46
Troubleshooting to barriers Adjusting for
unstable input in human services OER Contd

• Process analysis
• Process analysis identifies variance in protocol
  implementation participants receptiveness to
  the prescribed intervention. A process analysis
  covers
• Count of activities attended
• Perceived effectiveness of the intervention
• Compliance to intervention protocol
• Met and unmet needs related to the intervention

47
Troubleshooting to barriers Adjusting for
unstable input in human services OER Contd

• Process analysis (Contd)
• The information gathered from the process
  research should be integrated into the outcome
  research, rather than setting aside as auxiliary
  document in the discussion of study results

48
Troubleshooting to barriers Quality Assurance
in OER for human service setting

• Quality assurance exercise
• Organizational requirement
• Staff involvement

49
Maintenance and cost in OER

• Unanticipated effect of health promotion on
  social capital and cohesion (Raphael, 2000)
• Economic analysis (including Cost-effectiveness
  Analysis or Cost Benefit Analysis)

50
Conclusion
51
Decision questions generated from the BARD Model

• Q1. How can the range of problems be considered
  as part of public service (health, welfare, and
  education) priority?

52
Decision questions generated from the BARD Model

• Q2. What is the status of the efficacy studies on
  similar intervention programs in the
  international literature?

53
Decision questions generated from the BARD Model

• Q3. Who should do what and how much for whom
  before one can consider the intervention
  effective?

54
Decision questions generated from the BARD Model

• Q4. Should we vary the intensity of intervention
  (e.g. tiered multi-level program) to suit
  targeted service recipients?

55
Decision questions generated from the BARD Model

• Q5. How can staff training and documentation of
  quality manual help differentiate role and skills
  for professional and lay participation to ensure
  intervention integrity?

56
Decision questions generated from the BARD Model

• Q6. Is the technology transferred in the process
  of conducting OER sufficient to ensure human and
  financial sustainability?

57
Some final words for the Policy-makers

• WHO-EURO Working Group on Health Promotion
  Evaluation (1998) pointed out this
• Evaluation help them find out what works and how
  make decisions about future health promotion
  policy strategies and programs,.modify and
  improve current programs justify policy choices
  and increase impact and effectiveness (including
  cost-effectiveness) of health promotion
  initiatives.

58
Major NHWS OER publications

• Chan, C. C. (2000). Workbook on measurement
  use of outcomes in social services. Hong Kong
  Hong Kong Polytechnic University.
• Chan, C. C., Chow, C. B., Chan, K. (2003).
  SARS and social cohesion in Hong Kong A research
  agenda. Asian Pacific Journal of Social Work,
  manuscript in review.
• Chan, C. C., Lui, W. S., Wan, D., Yau, S.
  (2002). Evaluating Service Recipient Outcomes in
  Psychiatric Residential Services in Hong Kong.
  Research on Social Work Practice, 12(4), 570-581.
• Chan, C. C., Lui, W. S., Wan, D., Yau, S.
  (2002). A multilevel approach to supported
  employment outcomes in Schizophrenia. American
  Journal of Orthopsychiatry, manuscript submitted
  for publication.

59
Major NHWS OER publications

• Chan, C. C., Luis, B. P., Chow, C., Cheng, J.
  C. Y., Wong, T. W., Chan, K., Tsui, S. (2001).
  Validating narrative data on residential child
  injury. Journal of Safety Research, 37(4),
  377-389.
• Chan, C. C., Luis, B. P., Chow, C. B., Cheng,
  J. C. Y., Wong, T. W., Chan, K. (2003).
  Unintentional residential child injury
  surveillance in Hong Kong. Journal of Paediatrics
  and Child Health, 39(6), 420-426.
• Chan, C. C., Tsui, M. S., Chan, M. Y. C.,
  Hong, J. H. (2002). Applying the Structure of the
  Observed Learning Outcomes (SOLO) Taxonomy on
  Student's Learning Outcomes An empirical study.
  Assessment Evaluation in Higher Education,
  27(6), 511-528.

60
Major NHWS OER publications

• Chan, C. C., Wan, D., Lui, W. S. (2002).
  Determining performance standards for supported
  employment service in Hong Kong. Hong Kong
  Journal of Social Work, 36(1 2), 95-111.
• Cheung, F., Yau, A., Wong, K. (2001, February
  22, 2001). The Costs of Cost-Effectiveness An
  initial review of the competitive bidding
  experience in contracting out home care service.
  Paper presented at the Seminar on Efficiency
  Effectiveness of Social Care, Part III, The Hong
  Kong Polytechnic University, Hong Kong.
• Chow, C. B., Cheung, W. L., Chan, C. C.
  (2002, April 28-30, 2002). Transforming AEIS data
  for the prevention of unintentional childhood
  injuries in Hong Kong. Paper presented at the
  Hospital Authority Convention 2002, Hong Kong.

61
Major NHWS OER publications

• Ho, A. P. Y. (2001, May 19, 2001). Creating a
  supportive environment for elderly with chronic
  illness. Paper presented at the Healthy Ageing
  Convention, Hong Kong, Hong Kong Convention and
  Exhibition Centre.
• Chan, C. C., Chan, K. (2003).
  Competence-based evaluation of volunteer
  progress. Evaluation and Program Planning,
  manuscript in review.
• Network for Health Welfare Studies, the Hong
  Kong Childhood Injury Prevention Research
  Association. (2003). The Volunteer Home
  Visitation Programme Training Manual
• Network for Health Welfare Studies, the Hong
  Kong Childhood Injury Prevention Research
  Association. (2003). The Volunteer Home
  Visitation Programme Quality Assurance Manual

62
Key References

• Angrist, J. D., Imbens, G. W., Rubin, D. B.
  (1996). Identification of Causal Effects Using
  Instrumental Variables. Journal of the American
  Statistical Association, 91(434), 444-455.
• Flay, B. R. (1986). Efficacy and effectiveness
  trials (and other phases of research) in the
  development of health promotion programs.
  Preventive Medicine, 15(5), 451-474.
• Glasgow, R. E., Lichtenstein, E., Marcus, A.
  C. (2003). Why Don't We See More Translation of
  Health Promotion Research to Practice? Rethinking
  the Efficacy-to-Effectiveness Transition.
  American Journal of Public Health, 93(8),
  1261-1267.
• Little, R. J., Yau, L. H. Y. (1998).
  Statistical techniques for analyzing data from
  prevention trials Treatment of no-shows using
  Rubin's causal model. Psychological Methods,
  3(2), 147-159.
• McPherson, K., Britton, A. (2001).
  Preferences and understanding their effects on
  health. Quality Safety in Health Care,
  10(Supplement 1), 61i-66.
• Moore, L., Graham, A., Diamond, I. (2003). On
  the feasibility of conducting randomised trials
  in education case study of a sex education
  intervention. British Educational Research
  Journal, 29(5), 673-689.

63
Key References

• Newell, D. J. (1992). Randomised controlled
  trials in health care research. In J. Daly I.
  G. McDonald E. Willis (Eds.), Researching
  health care designs, dilemmas, disciplines (pp.
  47-61). New York Routledge.
• Raphael, D. (2000). The question of evidence in
  health promotion. Health Promotion International,
  15(4), 355-367.
• Rolls, L. (1999). The Challenge of
  Evidence-based Practice. In E. R. Perkins I.
  Simnett L. Wright (Eds.), Evidence-based health
  promotion (pp. 47-50). Chichester, New York John
  Wiley.
• Smith, G. (1985). Assessing health care a
  study in organisational evaluation / Gilbert
  Smith and Caroline Cantley. Milton Keynes Open
  University Press.
• Thompson, B., Coronado, G., Snipes, S. A.,
  Puschel, K. (2003). Methodologic advances and
  ongoing challenges in designing community-based
  health promotion programs. Annual Review of
  Public Health, 24(1), 315-340.

64
The End! Thank you for coming. For further
information Visit us at http//www.acad.polyu.edu
.hk/ssnhws/en/index.html