ProLearnG Gendersensitive Evaluation

1
ProLearn-GGender-sensitive Evaluation

• Fellowship Project
• Elke Dall
• dall_at_zsi.at
• Draft 03.09.2004

supported by CSI KTH bmbwk
2
Draft for PROLEARN evaluation guidelines

• This document will give a general outline on
  evaluation methodologies and processes and what
  can be understood as a gender-sensitive approach
  to evaluation and evaluation of gender-issues
  in the context of e-learning projects and
  resources.
• It will first given an outline of general goals
  of evaluation and evaluation processes. Then the
  possible relations between gender and evaluation
  and goals for the gender analytical approach. The
  presentation will deal with the development of
  objectives ? questions ? indicators ?
  methodologies ? data collection ? interpretations
  ? results ? actions for gender sensitive
  evaluation of e-learning resources.

3
Preliminary words to evaluation methods

• The goals initiative ? evaluation ? learning ?
  change ? next initiative
• The process evaluating the context (effects of
  and on individuals, organisations, communities
  the broader social, political, cultural and
  economic context), the product (e-learning
  resource) and the process (development, design,
  implementation) using qualitative and
  quantitative data
• The focus complex inequalities result as gender
  interacts with power relations evaluation shall
  raise awareness towards these as simple actions
  can often avoid unequal treatment
• The values evaluation is not value-neutral ? it
  is important to involve all stakeholders and feed
  back results to ensure validity and
  accountability
• The constraints affordability, time,
  reliability, politics, context sensitivity, etc.
  have to be considered in planning the evaluation
  strategy

4
General goals of evaluation

• The goals of an evaluation process are
• To check if the project objectives have been met
  and resources wisely utilized as justification
  for the funding agency
• To identify areas for improvement in a project or
  programme
• To surface and resolve disagreements
• To set priorities and goals
• To clarify and tackle problems
• To decide new strategic directions
• To get feedback, appraisal and recognition
• To celebrate achievements
• To attract resources
• Evaluation is not just done at the end of a
  project, it should be ongoing and part of the
  project plan from the beginning. Results can be
  used and disseminated, incorporated to
  organisational learning and published (formative
  or summative).

5
General goals of evaluation

• The use of the results will depend on the
  objectives of the evaluation you have performed
• Formative Evaluation involves collecting
  information about a prototype resource that will
  help in its development and ensure it works
  effectively. Emphasis at this point is typically
  on improving usability and refining content.
• Summative Evaluation takes place at the end of
  the developmental cycle, in order to prove the
  success of the resulting resource. For teaching
  and learning resources, this may involve
  demonstrating that the resource has made a
  difference in some tangible way. This may be to
  show that users have learned something from it,
  or that it has brought about a beneficial change
  in practice.

6
Evaluation process

• In the process of evaluation it is nesseary
• to define the purpose of the evaluation (your
  values, your goal)
• to define the evaluation objectives
• to define the evaluation questions
• to break them down to categories
• to define indicators for each category to guide
  the data collection
• to define the critical stakeholders
• to identify the means to monitor and collect
  information from the stakeholders
• to collect the data (qualitative and
  quantitative)
• to categorise of the findings according to the
  evaluation questions
• to gather and document illustrating and
  supporting stories
• to reflect on the findings and to extract lessons
• to prepare the evaluation report and to reflect
  the information

7
Evaluation and gender

• Evaluation processes involve one or more
  evaluators measuring a particular resource
  against a set of criteria ranging from some
  clear simple questions to relatively detailed
  questionnaires.
• There are several possible relations between
  gender and evaluation
• Gender-sensitive evaluation the process of
  evaluation shall be gender-sensitive
• Evaluation as means for gender mainstreaming
  evaluation shall help to show problems and
  support projects in improving their gender
  sensitivity
• Evaluation from the gender point of view
  evaluation of the project performance in Gender
  Mainstreaming
• Research in the field of gender and evaluation
  basic research on the questions in this field
  (definition of gender-specific criteria and
  requirements for projects, development of methods
  to gather information)

8
Goals of the gender analytical approach

• A gender analytical approach to evaluation
  provides a framework for identifying and
  analysing specific gender issues that determine
  the different impact ICT intervention as have on
  women and men.
• The key elements are
• gender equality
• gender roles
• practical gender needs and strategic gender
  interests
• womens empowerment
• gender and technology
• The evaluation should be carried out in a team
  (gender expert, ICT expert, participation of key
  stakeholder groups, user participation members
  involved in the project from different levels).

9
Evaluation strategy layout

• Detailed strategies for the process to carry out
  the evaluation, gathering data and the
  methodologies to be used have to be defined.
• It is useful to use a form for the Testing
  Profile in order to summarize the strategy
  defined for the process (goals, objectives,
  indicators, methods and a timeline) before
  starting to gather the data. ()

a document containing an example for this
Gender Evaluation Testing Profile is also
provided in the framework of this
project. contact dall_at_zsi.at (Elke Dall)
10
Specific gender objectives

• Possible objective for gender-sensitivity are
• To ensure that womens practical and strategic
  gender needs are addressed in future programmes.
• To understand how and why an ICT project has
  negatively (positively?) affected women in a
  community for social change.
• To assess the gender equality in the project
  context and organisation.
• To assess the gender sensitivity in the processes
  carried out to produce certain resources.
• To assess gender aspects in the product
  (e-learning resources) themselves (e.g. from the
  point of view of didactics, design, language,
  etc.).

11
From objectives to questions

• Definition of the specific gender issues to
  address in the evaluation
• Check core thematic areas and the elements of a
  gender analytical framework
• Define a clear goal for the evaluation
• improvement of the product / the learning
  material
• improvement of the didactical and pedagogical
  model
• improvement of the set-up of the project (project
  organisation, project team, project
  communications)
• dealing with social inequity in the context
  (influence on the stakeholders, etc.)
• Accordingly focus on context and/or process
  and/or product when defining the categories for
  gender analytical questions.

12
From objectives to questions

• Context Analysis
• Review of factors on individual, organisational,
  community, socio-economic, cultural and political
  level
• what gender issues can be identified in relation
  to the various stakeholders in the project
  context? are gender issues (GM goals) mentioned/
  requested/ favoured by the founding body? is what
  way can the project contribute to positive social
  change in its context? is there a way for the
  project to extend its scope so it can contribute
  to positive social change? (e.g. allow women
  groups to distribute the learning resource for
  free, attract more women through more targeted PR
  and advertising of the course, etc.)
• Respect political issues when evaluating
• form the team to carry out the evaluation and to
  implement its results gender-balanced
• make sure, the evaluation is helping to build
  support and not causing irritation (consider
  group dynamics, etc.)

13
From objectives to questions

• Definition of evaluation questions
• breat down the evaluation objectives into
  categories () and articulate questions related
  to gender and e-learning
• explicitly define the queries, dont select too
  many questions, dont formulate them too vague
• consider womens participation in the project,
  development of strategies within the project to
  respond to gender issues, roles in
  decision-making in the project, project
  strategies and content,

a xls-docment containing an extensive list of
categories and possible questions will also be
provided in the framework of this project contact
dall_at_zsi.at (Elke Dall)
14
Example questions for different categories

• These categories and questions are just
  illustrative examples ()
• PROCESS OF PRODUCTION
• vision and strategy (stated goals and objectives)
• are gender-equality and equal opportunities as
  goals defined and mentioned in the project vision
  or objectives?
• is Gender Mainstreaming (GM) defined as a
  strategy in the project to achieve gender
  equality?
• were women or groups of women identified as
  specific beneficiary groups?
• team and project management
• what was the level of women and mens
  participation in project activities?
• are men and women equally paid for equal
  positions?
• are gender-biased assignments of tasks avoided
  (not following stereotypes)?
• are women in management positions?
• are both gender involved in design,
  implementation and evaluation?
• project communication
• are informal structures of communication
  transparent and opened?
• are decision-making structures transparent?
• is the team exchanging knowledge (e.g.
  peer-to-peer-teaching) and working jointly on
  didactics, technology and evaluations?
• gender competence and qualification
• are gender issues a topic in the project team?
• does anybody have the designated responsibility
  for Gender Mainstreaming?

a xls-docment containing an extensive list of
categories and possible questions will also be
provided in the framework of this project contact
dall_at_zsi.at (Elke Dall)
15
Example questions for different categories

• These categories and questions are just
  illustrative examples ()
• PROCESS OF PRODUCTION
• participatory design / user needs
• are users involved in the development and design
  (respecting gender-parity)?
• are the different needs of user groups reflected
  in design and development of the learning
  materials? (family situation, competences in the
  use of technology, hardware conditions, financial
  and economical conditions, time constraints,
  previous learning experiences, motivation to use
  multi-media learning systems, etc.)
• is the development of the material content-driven
  (by didactic and pedagogical considerations) and
  not technology-driven?
• technical support
• are the users qualified to use the technology?
  are means to acquire the qualification available?
• is support available and can it be obtained
  through different channels telephone, e-mail,
  fora, FAQ-lists, coaching, "help"-texts, etc.?
• is help context sensitive and adapted to
  different situations of use?

a xls-docment containing an extensive list of
categories and possible questions will also be
provided in the framework of this project contact
dall_at_zsi.at (Elke Dall)
16
Example questions for different categories

• These categories and questions are just
  illustrative examples ()
• PRODUCT
• success / satisfaction
• are the stakeholders (users or beneficiaries,
  team members, funding institutions, etc.)
  satisfied with the project results / products /
  materials offered? (sex-disintegrated data!)
• what were the expected and unexpected results of
  the project (in relation to gender issues)?
• topics and their presentation
• are interests of both genders respected?
• are topics presented using interdisciplinary
  examples?
• are men and women equally and balanced
  represented?
• are role models for women and men represented?
• are examples gender-neutral?
• didactics
• is a training for "how to learn with e-learning"
  provided?
• is the relevance of the acquired knowledge
  clearly demonstrated?
• are introductory notes (to content, technology
  and design) and summaries (for each module and
  separate parts of the material) available in
  order to quickly assess the content?
• are there suggestions made how to plan the
  learning?
• are different learning styles and knowledge
  levels respected?
• is groupwork, partnerwork and self-organised
  learning encouraged and in a good balance?
• is interdisciplinary application of the content
  encouraged and required?

a xls-docment containing an extensive list of
categories and possible questions will also be
provided in the framework of this project contact
dall_at_zsi.at (Elke Dall)
17
Example questions for different categories

• These categories and questions are just
  illustrative examples ()
• PRODUCT
• technology
• is the necessary equipment clear (hardware,
  software, system used, internet access etc.)?
• is the compatibility with other systems secured
  (e.g. different hardware, different displays,
  etc.)?
• is technology adaptable and personalisation
  possible?
• communication
• are first-time users welcomed (in a
  gender-sensitive way)?
• are contacts between students and teachers and
  among students possible and encouraged?
• are the learners supported to build a community?
• are technical terms explained and not assumed as
  being known?
• is feedback constructive and related to the
  content?
• language use
• is stereotypic use of language avoided?
• is discrimination (against women, but also
  against age, race, religion, minorities, )
  avoided?
• is gender-sensible language used in all texts of
  the project?
• are users addressed in a personal way using
  female and male or gender-neutral forms?
• are both genders visible in the language used?
• graphical design

a xls-docment containing an extensive list of
categories and possible questions will also be
provided in the framework of this project contact
dall_at_zsi.at (Elke Dall)
18
From questions to indicators

• Tools to capture the information about the issues
  defined have to be selected and adapted
  indicators are specific (explicit) and
  objectively verifiable measures of results or
  changes brought about by a project
• Indicators to guide the information gathering
  process have to be developed and selected
• pointers, numbers, facts, opinions or perceptions
  that look into and measure changes of specific
  conditions or situations (but not too many)
• quantitative or qualitative
• SMART specific, measurable, achievable,
  realistic, time-bound
• sex-disaggregated
• easy to use and to understand, clearly defined,
  technically sound
• should be established during the project planning
  phase, be linked to the project goals and
  objectives and relevant to the needs of the user
• developed in a participatory fashion, including
  all stakeholders and changeable during the
  process of evaluation

19
Gender indicators

• Examples for indicators
• number of mentioned gender issues in official
  statements -gt analysis of project documents,
  objectives, proposals, website, etc.)
• percentage of women in different levels and at
  different project activities
• relations between salaries of women and men
• formal decision-making structures vs. informal
  structures (methods questionnaire, interviews,
  satisfaction, etc.)
• way of participatory design, inclusion of
  user-needs
• satisfaction of the users, success rate,
  drop-out, etc.
• analysis of content examples, illustrations,
  stereotypes, role models, etc.
• analysis of language nr. of generic masculine
  expressions used, nr. of sexist expressions,
  stereotypes, etc.
• analysis of curriculum time consumption, nr. of
  possibilities of adaptation of content and
  presentation of the content (respecting learning
  styles, knowledge levels, etc.)
• communication channels nr. of contacts between
  different groups, quality of feedback, etc.

20
From indicators to a methodology

• In order to gather data on these gender
  indicators, methodologies have to be chosen and
  instruments have to be set up.
• There should be a mix of methodologies chosen to
  gather the data, they should be participatory and
  ensure collection of sex disaggregated data.
• Data and information gathering tools and
  methodologies are e.g.
• surveys and questionnaires (e.g. electronic
  questionnaires)
• interviews (with stakeholders)
• discussions or focus groups (with staff,
  beneficiaries, etc.)
• observations
• system data, log files
• records and statistics
• documents analysis (papers, reports,
  correspondence, minutes of meetings, etc.)
• online discussion areas for reflections and
  feedback
• stories (accounts that reveal the stakeholders
  perspective about the project)
• with their respective strengths and weaknesses.

21
From the methodology to the questions

• The questionnaire / interview questions should be
  a combination of open questions and closed
  questions.
• Designing the evaluation questionnaire it has to
  be considered
• how are the questions constructed?
• when are they asked?
• who will reply to them?
• how is the information received validated and
  corroborated?
• what will be done with the information collected?
• Different types of questons can include
• measurement (e.g. how many male / female students
  have successfully completed the module? exploring
  cost factors, etc.)
• comparison (e.g. made with more conventional
  means of teaching and learning)
• exploration (how was the learners experience
  during the course? subjective information on
  attitudes, motivational factors, etc.)

22
Data collection

• For instance, data could be
• quantitative data
• analysis of statistical data
• number of women and men involved
• number of women and men trained
• logfiles
• user questionnaires (e.g. on their situation
  regarding use of time, money, etc.)
• drop-outs, success rates
• Qualitative data
• interviews and questionnaires answered by
  relevant stakeholders
• project manager
• team members
• teachers, moderators, tutors
• users, students, learners
• others if applicable
• analysis of texts (project descriptions,
  proposals, presentations like website, etc.)
• lessons learned
• gathered through interviews

23
From evaluation results to actions

• The results shall be interpreted according to the
  categories and indicators.
• The data gathered and answers given shall lead to
  interpretations and then to results of the
  evaluation.
• The results shall be reflected critically and
  suggested actions shall be considered.
• Then it it is necessary to put evaluation results
  to work!
• A dissemination plan shall detail the activities
  and the timeline for sharing evaluation results
  with different stakeholders.
• The presentation of the results depends on the
  audience to which it is addressed and should be
  carefully considered. The requirements,
  recommendations and comments of the audience
  should be integrated.
• Who are the key stakeholders that will receive
  evaluation results?
• What will they want to know? How will they use
  the data?

24
Critical stakeholders

• Evaluation cannot include all groups that have
  participated, benefited or not benefited from the
  activities ? important stakeholders have to be
  selected and included in the evaluation process
  considering
• internal (project executors including project
  staff and management) and external (project
  beneficiaries) stakeholders
• direct (those who are/or were directly involved
  in project activities) and indirect stakeholders
  (those who did not participate in the project and
  may or may not have been affected by it)
• other organizations that do similar work can also
  be seen as stakeholders (offering important
  insights and sector commentary providing a
  broader, but focused, context)
• project beneficiaries /users (the primary target
  group for the e-learning initiative) are integral
  to the process of discovering and analyzing
  gender and e-learning issues.
• In some contexts, a formal report of the
  evaluation results may be required, while in
  others a more informal presentation of the
  outcome may be more appropriate.

25
Possible actions

• Evaluation Results shall result in learning and
  action
• Research and Critical Understanding Evaluation
  results can point to areas of your work that
  require additional research starting a learning
  process that will build your organisations
  critical expertise in a particular field of work.
  Evaluation results can also be used to test and
  advance analytical frameworks that inform gender
  equality and e-learning.
• Sharing Best Practices and Lessons Learned share
  information about what gender issues in
  e-learning have come up. Using evaluation results
  as material to document best practices and
  lessons learned and sharing this information
  builds up a pool of needed reference material.
• Project Planning and Design Evaluation results
  should inform future project design and
  implementation. Using evaluation results from
  completed projects as starting points for
  developing new projects maximises the
  organisations learning about its work. It also
  increases the likelihood that the next project
  will be more successful and satisfying for the
  users.
• Resource Mobilisation Evaluation results can be
  used to acquire projects and funding. Evaluation
  reflects a track record and experience of working
  in a particular area. At a broader level,
  evaluation results can demonstrate the need for
  resources to be committed to gender and
  e-learning.

26
From actions to a broader audience

• Useful ways of disseminating the evaluation
  results are
• Advocacy and Lobbying inputs in national,
  regional or global policy debates as research
  material.
• Public Education and Networking communicating
  and building relationships with constituencies
  (communities, other organisations, agencies,
  funding bodies, etc.) A participatory approach to
  evaluation requires feedback of results to all
  communities involved with a project.
• Basis for publishing articles in newsletters,
  popular and academic journals and annual reports
  and presentation in networking forums (electronic
  discussion groups or face to face meetings)
• Marketing and Public Relations advocate for the
  work to raise awareness about the project, etc.
  Using evaluation results in publicity materials
  pamphlets, web sites, press releases, etc can
  raise the profile by establishing a rigorous
  approach to work. Evaluation results can also be
  used to provide media with "evidence" and story
  ideas related to gender issues.

27
Resources

• This document is mainly based upon the following
  sources and own adaptations
• Cook, J. Evaluating Learning Technology
  Resources, Learning Technology Support Service.
  University of Bristol, UK. 1999.
  http//www.ltss.bris.ac.uk/publications/guides/eva
  luation/evaluation_1.html
• Creanor, Linda / Glasgow Caledonian University
  E-Learning Guides. 7. Evaluating E-Learning.
  http//apu.gcal.ac.uk/pages/resources.htm
• GEM Gender Evaluation Methodology for internet
  and ICTsdeveloped by APC WNSP, 2001.
  http//www.apcwomen.org/gem/go4gem/index.htm
• Wienold, Kirsten Evaluation multimedialer Lern-
  und Informationssysteme. Gender Mainstreaming in
  der Evaluation. 2002.
• Wiesner, Heike, Marion Kamphans, Heidi Schelhowe,
  Sigrid Metz-Göckel, Isabel Zorn, Anna Drag,
  Ulrike Peter, Helmut Schottmüller Gender
  Mainstreaming in "Neue Medien in der Bildung"
  Leitfaden. 2004. http//dimeb.informatik.uni-breme
  n.de/documents/projekt.gender.GMLeitfaden.pdf
• Wiesner, Heike, Heidi Schelhowe, Sigrid
  Metz-Göckel, Marion Kamphans, Ulrike Peter,
  Helmut Schottmüller, Claudia Kedenburg, Anja
  Tigges, Kirsten Wienold, Marc Jelitto, Hannah
  Cho-Heinze "GM-Styleguide" Gender Mainstreaming
  im Kontext Neuer Meiden. 2003. http//www.medien-b
  ildung.net/forum/attachments/GMStyleguideApril03.d
  oc
• Wood, Peregrine Building Gender Considerations
  Into ICT Evaluation Work. APC WNSP. 1997.
  http//www.apcwomen.org/work/research/build-gender
  .html