Topic 11 Policy Process Studies: Policy Evaluation Studies

1
Topic 11Policy Process StudiesPolicy
Evaluation Studies
PEDU 6209 Policy Studies in Education
2
Conceptualization of Policy Evaluation

• Conceptualization of evaluation
• Fitzpatrick, Sanders and Worthen define
  evaluation as the identification, clarification,
  and application of defensible criteria to
  determine an evaluation objects value (worth or
  merit) in relation to those criteria. (2004, p.5)

3
Conceptualization of Policy Evaluation

• Conceptualization of policy evaluation
• Accordingly, the constituents of evaluation
  consist of
• The evaluation object It is of course public
  policy or more specifically education policy.
• The evaluation criteria Define the criteria upon
  which the evaluations are based. They may include
  
• Policy objectives To assess to what extent the
  identified policy objectives have been fulfilled.
• Policy measures and programs To investigate to
  what extent the designed policy measures and
  programs have been implemented
• Policy outcomes To measure to what extent the
  targeted outcomes have been realized
• Cost and benefit of the policy To assess the
  cost of the implementation of policy means
  against the benefits entailed by the policy
  outcome

4
Conceptualization of Policy Evaluation

• Conceptualization of policy evaluation
• the constituents of policy evaluation consist of
  
• The judgment The evaluation can be studies and
  judge by means of
• Analytical-technical study
• Interpretive-political study
• Discursive-critical study

5
Conceptualization of Policy Evaluation

• Typology of policy evaluation
• In regards to the purpose of policy evaluation,
  policy evaluation can be categorized into
• Formative evaluations aim to provide information
  for personnel implementing the policy or carrying
  out the program. Hence, their primary purposes
  are for the development and improvement of the
  policy and/or program
• Summative evaluations aim to pass judgment on the
  performance of the policy and/or program. They
  are primarily carried out as accountability
  mechanism or auditing measures.

6
Conceptualization of Policy Evaluation

• Typology of policy evaluation
• In relation to the policy stages, policy
  evaluation can be categorized into
• Need assessments They the studies conducted at
  the formative stages of the policy process. They
  are conducted either to evaluate the urgencies of
  the policy problems or the needs that are
  required if the identified desirable states of
  affairs are to be realized.
• Process-monitoring studies They are studies
  carried out to monitor to what extent are the
  designed policy measures .and programs are
  implemented in real world situations. They also
  aim to provide information for the improvement of
  the effectiveness of the policy instruments in
  use.

7
Conceptualization of Policy Evaluation

• Typology of policy evaluation
• In relation to the policy stages, policy
  evaluation can be categorized into
• Outcome studies They are carried out to assess
  to what extent have the preconceived policy
  outcomes been attained. They can also provide
  information to the revelations of unanticipated
  consequences (desirable, undesirable or both) of
  the policy

8
Conceptualization of Policy Evaluation

• Typology of policy evaluation
• In terms of the policy context, policy evaluation
  can be classified into
• Endogenous policy evaluation They are studies
  carried out to assess the internal effectiveness
  and coherence among various parts of the policy
  constituents. The most common types of the
  endogenous evaluation studies are those to assess
  whether the policy outcomes are in congruence
  with the policy objectives.
• Exogenous policy evaluation They are studies
  assessing whether the policy substance are in
  congruence with the policy environment. These
  type of evaluation studies are especially
  informative in ever changing policy environment.

9
Frank Fischers Conception of Four Levels of
Policy Evaluation
10
Frank Fischers Conception of Four Levels of
Policy Evaluation

• The framework of practical logical of policy
  evaluation
• Post-positivist methodological approach to policy
  evaluation By post-positivism, it refers to "a
  contemporary school of social science that
  attempts to combine the discourse of social and
  political theory with the rigor of modern
  science. It calls for a marriage of scientific
  knowledge with interpretive and philosophical
  knowledge about norms and values." (Fischer,
  1995, p. 243) Accordingly, Fischer advocates that
  policy evaluation research should extend beyond
  the dominance of empirical-positivism and
  incorporate practical discourse about rightness
  of normative claims into policy evaluation
  research..

11
Frank Fischers Conception of Four Levels of
Policy Evaluation

• The framework of practical logical of policy
  evaluation
• Fischer names such a post-positivist approach in
  policy evaluation the framework of the "Practical
  Logic of Policy Evaluation".
• "It is design to illuminate the basic
  discursive components of a full or complete
  evaluation, one which incorporates the full range
  of both the empirical and normative concerns that
  can be brought to bear on an evaluation.
  Structured around four interrelated discourses,
  the approaches extends from concrete empirical
  questions pertinent to a particular situation up
  to the abstract normative issues concerning a way
  of life." (Fischer, 1995, p. 18)

12
Frank Fischers Conception of Four Levels of
Policy Evaluation

• The framework of practical logical of policy
  evaluation
• Accordingly, the framework of practical logic of
  policy evaluation is made up of
• First-order evaluation level
• Technical-analytical discourse
• Contextual discourse
• Second-order evaluation level
• Systems discourse
• Ideological discourse

13
Frank Fischers Conception of Four Levels of
Policy Evaluation

• Technical-analytical discourse Program
  verification
• By program verification, it refers to concerns
  "with measurement of the efficiency of program
  outcome. The basic questions of verification are
• Does the program empirically fulfill its stated
  objective(s)?
• Does the empirical analysis uncover secondary or
  unanticipated effects that offset the program
  objectives?
• Does the program fulfill the objectives more
  efficiently than alternative means available?"
  (p. 20)

14
Frank Fischers Conception of Four Levels of
Policy Evaluation

• Technical-analytical discourse Program
  verification
• "Program verification employs such methodologies
  as experimental research and cost-benefit
  analysis. The goal is to produce a quantitative
  assessment of the degree to which a program
  fulfills a particular objective (standard or
  rule) and a determination (in terms of a
  comparison of input and output) of how
  efficiently the objective is fulfill (typically
  measured as a ratio of costs to benefits)
  compared to other possible means." (p. 20)

15
Frank Fischers Conception of Four Levels of
Policy Evaluation

• Contextual discourse Situational validation
• Validation focuses on whether or not the
  particular program objectives are relevant to the
  situation. Instead of measuring program
  objectives per se, validation examines the
  conceptualizations and assumptions underlying the
  problem situation which the program is designed
  to influence. Validation centers around the
  following questions
• Is the program objective(s) relevant to the
  problem situation?
• Are there circumstances in the situation that
  require an exception to be made to the
  objective(s)?
• Are two or more criteria equally relevant to the
  problem situation?" (Pp. 20-21)

16
Frank Fischers Conception of Four Levels of
Policy Evaluation

• Contextual discourse Situational validation
• "Validation is an interpretive process of
  reasoning that takes place within the framework
  of the normative belief systems brought to bear
  on the problem situation. Validation draw in
  particular on qualitative methods, such as those
  developed for sociological anthropological
  research." (p.21)

17
Frank Fischers Conception of Four Levels of
Policy Evaluation

• Systems discourse Societal vindication
• "In vindication the basic task is to show that a
  policy goal (from which specific program
  objectives were drawn) addresses a valuable
  function for the existing societal arrangement.
  Vindication is organized around the following
  questions
• Does the policy goal have instrumental or
  contributive value for the society as a whole?
• Does the policy goal result in unanticipated
  problems with important societal consequences?
• Does a commitment to the policy goal lead to
  consequence (e.g. benefits and costs) that are
  judged to be equitably distributed?" (p. 21)

18
Frank Fischers Conception of Four Levels of
Policy Evaluation

• Systems discourse Societal vindication
• As second-order vindication, this type of policy
  evaluation "steps outside of the situational
  action context in which program criteria are
  applied and implemented in order to assess
  empirically the instrumental consequences of a
  policy goal in terms of the system as a whole."
  (p. 21) Hence, the methodological approach
  adopted by societal vindication is
  macroscopic-institutional analysis which usually
  takes the forms of comparative and/or
  historical-sociological methods.

19
Frank Fischers Conception of Four Levels of
Policy Evaluation

• Ideological discourse Social choice
• "Social choice seeks to establish and examine the
  selection of a critical basis for marking
  rationally informed choices about societal
  systems and their respective ways of life.
  Social choice raises the following types of
  questions
• Does the fundamental ideal (or ideology) that
  organize the accepted social order provide a
  basis for a legitimate resolution of conflicting
  judgments?
• If the social order is unable to resolve basic
  values conflicts, do other social orders
  equitably accommodate the relevant interests and
  needs that the conflicts reflect?
• Do normative reflection and empirical support the
  justification and adoption of an alternative
  ideology and the social order it prescribes? (p.
  22)

20
Frank Fischers Conception of Four Levels of
Policy Evaluation

• Ideological discourse Social choice
• "Social choice involves the interpretive tasks of
  social and political critique, particularly as
  practiced in political theory and philosophy.
  Most fundamental are the concepts of a 'rational
  way of life' and 'good society'. Based on the
  identification and organization of a specific
  configuration or values ? such as equality,
  freedom, or community ? models of the good
  society serve as a basis for the adoption of
  higher level evaluative criteria." (p. 22) The
  typical methodological approach adopted by this
  type of evaluative studies is critical approach
  to political, social and philosophical inquiry.

21
Ideological Choice Because of
Critical Discursive Perspective
Systems Vindication Because of
Interpretive Political Perspective
Situational Validation Because of
Warrant Since
Analytic Technical Perspective
Technical Verification
Data
So, (Qualifier), Conclusion
22
Evaluation Study of MOI Policy for Secondary
Schools in HKSAR
Topic 11Policy Process Studies Policy
Evaluation Studies
23
Technical-analytical Discourse Program
verification

• A policy measure in search of an objective
• Summative or even judgmental evaluation of the
  efficiency of CMI in comparison with EMI
• Informative evaluation of the efficiency of
  preparing secondary-school leavers to become
  trilingual and bilateral
• Upholding mother-tongue instruction
• Summative evaluation of the effectiveness of
  eliminating mixed-coded teaching in secondary
  schools in HKSAR

24
The Objectives of the Longitudinal Study
(1999-2002) are

• to trace the academic personal development of
  students in schools adopting either Chinese or
  English as the MOI
• to compare the degree of improvement of students
  academic and personal development in schools
  adopting either Chinese or English as the MOI
• to compare the language ability (in both Chinese
  English) of students in schools adopting either
  Chinese or English as the MOI and
• to identify facilitating hindering factors
  affecting students learning in school adopting
  Chinese as the MOI.
• (p.2 of Tender ref.
  ED/PR/EMICM/99)

25
The Objectives of the Longitudinal Study
(2002-2004)

• Study are to find out
• The effects of different MOI arrangements at
  senior levels on students learning, e.g. their
  academic performance, personal development,
  language ability and high-order thinking skills
• the major factors that enhance/hinder students
  learning in Chinese or English MOI at senior
  secondary levels and
• the major factors that enhance/hinder the
  introduction of English as MOI or in part only at
  senior secondary levels.
• based on the findings, to design/recommend
  measures that support students learning under
  different MOI arrangement for their whole
  secondary schooling.
• (Tender ref.
  ED/PR/EMICMI/02)

26
Technical-analytical Discourse Program
verification

• To compare the academic and personal development
  outcomes of students in EMI and CMI schools
  Summative, outcome and endogenous evaluation
  research
• Instructional paradox in MOI policy outcomes
• Socio-psychological paradox in MOI policy
  outcomes
• School-organizational paradox in MOI policy
  outcomes

27
Subjects

• Cohorts of Students under Study

28
00-01
99-00
01-02
02-03
03-04
04-05
05-06
98 Cohort
98 AAI
F. 2 Tests
HK CEE
F. 3 Tests
F. 4 Tests
HK ALE
99 Cohort
99 AAI
F. 1 Tests
F. 2 Tests
F. 4 Tests
HK CEE
HK ALE
F. 3 Tests
29
Teachers Perception of Learning Environment
A. Sense of efficacy B. Sense of administrative
and collegial support C. Sense of reward
recognition from work D. Sense of autonomy
empowerment E. Feeling of stress in work
Students Performance Perception of Learning
Environment A. Academic Development a.
Language Subject - Chinese Language
- English Language b. Content Subject
- Mathematics - Science - Social
Subjects B. Personal Development a.
Self-esteem b. Citizenship Development c.
Social Efficacy d. Language learning
attitudes C. Perception of Learning Environment
a. Learning process motivation b.
Teachers efficacy c. Classroom climate
d. Quality of school life
EMI
HKCEE Results
EMI
CHIG
Students Academic Socio-economic Background
Implementation of the MOI Guidance
CMI
EMI
CMID
CMI
EMI
CLOW
CMI
School Administration Learning Environment
A. Principals Leadership B.
Organizations of Academic Disciplinary Orders
C. Organizational Climate Collaborative
Culture.
Conceptual Framework of the Project
30
Technical-analytical Discourse Program
verification

• Instructional paradox in MOI policy outcomes
• (Junior Forms)

31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
Technical-analytical Discourse Program
verification

• Instructional paradox in MOI policy outcomes
• (Senior Forms HKCEE)

38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
Technical-analytical Discourse Program
verification

• Instructional paradox in MOI policy outcomes
• (Access to University Education HKALE)

49
Educational Attainment Measure JUPAS Minimum
entry requirement1. Grade E or above in either
2 AL subjects  or 1 AL subject 2 AS
subjects (other than UE and Chi Lang
Culture)2. Grade E or above in AS UE3. Grade E
or above in AS Chi Lang Culture
50
(No Transcript)
51
(No Transcript)
52
Educational Attainment Measure JUPAS Maximum
entry requirement1. Grade E or above in either
2 AL subjects  or 1 AL subject 2 AS
subjects (other than UE and Chi Lang
Culture)2. Grade D or above in AS UE3. Grade E
or above in AS Chi Lang Culture
53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
Technical-analytical Discourse Program
verification

• To compare the academic and personal development
  outcomes of students in EMI and CMI schools
  Summative, outcome and endogenous evaluation
  research
• Instructional paradox in MOI policy outcomes
• Paradox in socio-psychological outcomes
• School-organizational paradox in MOI policy
  outcomes

58
Paradox in Socio-psychological Outcomes

• Academic Self-concept
• Attitudes towards English Motives Strategies
  in English Learning
• Quality of School Life
• School Identity and Education Aspiration

59
(No Transcript)
60
(No Transcript)
61
Paradox in Socio-psychological Outcomes

• Academic Self-concept
• Attitudes towards English Motives Strategies
  in English Learning
• Quality of School Life
• School Identity and Education Aspiration

62



63



64



65



66



67



68
Paradox in Socio-psychological Outcomes

• Academic Self-concept
• Attitudes towards English Motives Strategies
  in English Learning
• Quality of School Life
• Linguistic efficacy
• Learning Opportunities
• School Identity and Education Aspiration

69



70



71
Paradox in Socio-psychological Outcomes

• Academic Self-concept
• Attitudes towards English Motives Strategies
  in English Learning
• Quality of School Life
• Linguistic efficacy
• Learning Opportunities
• School Identity and Education Aspiration

72
99-cohort 2001/2002 (F.3)



73
98-cohort 2001/2002 (F.4)



74
Educational Aspiration



75
Educational Aspiration



76
Educational Aspiration



77
Educational Aspiration



78
Technical-analytical Discourse Program
verification

• To compare the academic and personal development
  outcomes of students in EMI and CMI schools
  Summative, outcome and endogenous evaluation
  research
• Instructional paradox in MOI policy outcomes
• Paradox in socio-psychological outcomes
• School-organizational paradox in MOI policy
  outcomes
• Paradox between instructional effectiveness and
  educational advancement
• Paradox between intake-process-output in
  different phases of secondary-school education

79
Form-1 Intakes
Junior-Secondary Instructional Process
Senior-Secondary Instructional Process
HKCEE Results
Six-Form Instructional Process
HKALE Results
80
Technical-analytical Discourse Program
Verification

• To prepare of secondary-school leavers to be
  trilingual bilateral
• Typology of bilingual
• Balance bilingual
• Chinese dominant bilingual
• English dominant bilingual
• Dimensions of bilingual proficiency
• Cognitive demanding or undemanding
• Context reduced or embedded
• Levels of bilingual proficiency
• Conversational proficiency
• Context-reduced and cognitive undemanding
  proficiency
• Context-embedded and cognitive demanding
  proficiency
• Academic language proficiency

81

• Jim Cummins Conceptualization of language
  proficiency

Cognitively Undemanding
A
C
Context Embedded
Context Reduced
B
D
Cognitively Demanding
82
Technical-analytical Discourse Program
Verification

• Upholding mother-tongue instruction
• Elimination of mixed-coded teaching in secondary
  schools in HKSAR
• at the expenses of
• Opportunity to learn
• Opportunity for educational advancement

83
Contextual Discourse Situational Validation of
the Relevance of the MOI Policy

• To identify the stakeholders (e.g. students,
  parents, teachers and/or administrators in
  secondary schools, employers, higher-education
  institutions, etc.) and their definition of the
  situations and systems of relevance in regards
  to the MOI policy

84
Contextual Discourse Situational Validation of
the Relevance of the MOI Policy

• Situational paradoxes facing stakeholders
• Students and Parents Situational paradox between
  immediate learning efficacy and long-term
  prospects in educational or even socioeconomic
  advancement
• School teachers and administrators
• Situational paradox between instructional
  efficacy and graduates' prospects in educational
  advancements
• Situation paradox between current instructional
  efficacy and future efficacy of attracting
  "high-quality" primary-school leavers or even
  future prospect of survival of the schools.
• Employers The situational paradox is between the
  prospects of local and global markets.
• Higher-education institution The situation
  paradox is between short-term instructional
  efficacy and long-term development of universally
  applicable scholarship and/or professional
  competence

85
Contextual Discourse Situational Validation of
the Relevance of the MOI Policy

• To validate whether or to what extent each of the
  stakeholders systems of relevance have been
  fulfilled by the MOI policy measure.

86
System Discourse Societal Vindication of the MOI
Policy

• Systemic vindication of value orientation of the
  MOI policy
• To define the institutional features of the
  social, political and economic systems of HKSAR
  Biglossic or even triglossic structures of a
  post-colonial society, one-country-two-system
  polity, and open and small economy in
  global-informational capitalism.
• To evaluate whether or to what extent that MOI
  policy have instrumental or contributive value
  for these systemic features of HKSAR

87
Institutional features of the MOI policy
environment (socio-cultural, political and
economic systems) of HKSAR
Local political economic systems
Intra- national tertiary education institution
Global professional- intellectual community
Local community of Mother Tongue
Local tertiary education institution
Global- informational economy
Primary School System
Secondary School System
Global- political arena
International tertiary education institution
Intra- national political economic systems
88
Bilingual System with English Dominance
Trilingual system with English/Putonghua Dominance
Monolingual Local community of Cantonese
Trilingual system with Cantonese Dominance
Local political economic systems
Intra- national tertiary education institution
Global professional- intellectual community
Local community of Mother Tongue
Local tertiary education institution
Global- informational economy
Primary School System
Secondary School System
Global- political arena
International tertiary education institution
Intra- national political economic systems
89
System Discourse Societal Vindication of the MOI
Policy

• To vindicate the intensification effect of MOI
  policy on the contradictory structure of HKSAR
• Biglossic or even triglossic structures of the
  open and small economy of HK within the
  global-informational capitalism.
• Biglossic structure of the political system of
  HKSAR within the context of One Country Two
  System
• Biglossic or even troglossic structure of the
  cultural system of a post-colonial society
• To vindicate the unequalizing effect of MOI
  policy on the structural disparity among the 18
  school districts in the SSPA system in terms of
  distribution of EMI school places

90
(No Transcript)
91
(No Transcript)
92
(No Transcript)
93
System Discourse Societal Vindication of the MOI
Policy

• To vindicate the institutionalized effect of MOI
  policy on the inequality of mobility chances
  between CMI and EMI students in competing access
  to higher education.

94
(No Transcript)
95
(No Transcript)
96
Ideological Discourse Social Choice underlying
the MOI policy

• To identify the ideological stance underlying the
  MOI policy discourse
• Orientations to bilingualism
• Bilingualism as problem
• Linguistic imperialism
• Linguistic nationalism
• Bilingualism as rights
• Bilingualism as resource
• Power implications of bilingualism
• Bilingualism as coercive power
• Bilingualism as collaborative power
• Structure-agent stance in bilingualism
• Structural imposition stance
• Agent resistance stance
• Agent appropriation stance

97
Ideological Discourse Social Choice underlying
the MOI policy

• To identify the ideological stance underlying the
  MOI policy discourse
• The myth of mother tongue

98
Ideological Discourse Social Choice underlying
the MOI policy

• To reflect and choose the ideological stances
  toward Cantonese, Poutonghau and English, which
  have been embedded in the MOI policy discourse in
  post-1997 HK.

99
Evaluation Study of MOI Policy for Secondary
Schools in HKSAR
Mother Tongue or Other Tongue
100
Topic 11Policy Process StudiesPolicy
Evaluation Studies
END