THE STATE OF READINESS

1
THE STATE OF READINESS national senior
certificate (NSC) examination and annual
national assessment (ANA)
Portfolio Committee on Basic Education Presented
by Nkosinathi Sishi, PhD Chief Director
Educational Measurement, Assessment and Public
Examinations RSA
31 AUGUST 2010
2
PURPOSE OF THE SOR

• The state of readiness visit in 2010 focused on
  the
• following key areas
• The readiness of the PEDs to administer the 2010
  NSC examination.
• The status of the monitoring and moderation
  systems relating to School Based Assessment (SBA)
  at the Grade 10, 11 and 12 levels.
• The State of Readiness of the PEDs to administer
  the ANA in Grade 3, 6 9.
• Readiness of learners to take the assessment

3
International comparability of QUALITY,
STANDARDS, Systems, and practices

• Principles
• Fair Assessment
• Developmental Assessment
• Assessment should be developmental rather than
  judgemental,
• Credibility of site-based assessment remains a
  big challenge.
• Fit for Purpose Assessment
• Valid Assessment, Reliable Assessment ,
  Integration, Relevance
• Credibility and Standards
• Flexibility and Recognition of Prior Learning,
• Legitimacy, Access and Redress
• Articulation and Portability.

• Strategic goal
• Credible educational measurement, assessment run
  with integrity
• Quality practices
• Strategic objectives
• National assessment and measurement of high
  international esteem
• Valid and fair educational judgments
• National historical learner records
• Integrity of Site-based assessment
• Recognition of prior learning
• Credible qualifications, certification and
  accreditation
• Educational excellence
• NQF Principles

4
Methodology

1. Hypothesis
2. Self-evaluation
3. Meetings and interviews
4. Site visits Provincial offices, printing and
   distribution centres, districts
5. Documents analysis
6. Sampling
7. Plenary
8. Shortcomings identified in the 2008 and 2009
   monitoring process

5
Specific Aim

1. Establishing system preparation for the 2010
   NSC examination
2. Combat readiness for the Annual National
   Assessment
3. Credibility and integrity of Site-based
   Assessment
4. Preparation of learners for the 2010 national
   assessment and public examinations

6
MAKING VALID EDUCATION ASESSMENT JUDGEMENTS

7
Operational Objectives

• High quality question papers are set and
  moderated
• Candidates are accurately registered resulted
  on the examination computer system
• Analytical reports are generated
• Question papers are accurately printed, securely
  packed and distributed to examination venues
• All candidates including special needs catered
  for
• Answer scripts accurately marked and moderated
• All examination processes are conducted with
  integrity
• Prioritisation of security and accountability in
  the distribution of scripts and the collection of
  answer books

8
Operational Objectives

1. SBA is completed in accordance with policy
   prescripts and moderated.
2. All examination and SBA marks are accurately
   captured.
3. All examination irregularities are appropriately
   managed.
4. All candidates are accurately and timeously
   resulted.
5. Effective feedback provided to schools.
6. All eligible candidates receive certificates.

9
National Senior Certificate Examinations 2010

• School-based assessment is ongoing throughout the
  year.
• Practical examinations in Visual Art and Design
  began on 16 August 2010.
• Practical examinations in Computer Applications
  Technology and Information Technology to be
  conducted from 13 to 15 October 2010
• The first written examinations on 25 October
  2010
• Information Technology Theory, and
• Computer Application Technology Theory paper.
• The last examinations to be written on 3 December
  2010
• Religion Studies Paper 2
• Civil Technology, and
• Hospitality Studies

10
Province Full Time Enrolments 2009 Full Time Enrolments 2010
Eastern Cape 70 622 70 852
Free State 30 462 29 094
Gauteng 101 268 94 546
KwaZulu Natal 139 965 131 379
Limpopo 93 287 93 806
Mpumalanga 55 735 55 885
North West 31 395 29 585
Northern Cape 10 765 10 399
Western Cape 46 861 46 927
Total 580 360 562 473
11
Province Part Time Enrolments 2009 Part Time Enrolments 2010
Eastern Cape 3 516 11 141
Free State 919 2 022
Gauteng 8 278 22 852
KwaZulu Natal 12 577 19 990
Limpopo 7 834 9 529
Mpumalanga 2 989 4 684
North West 1 029 2 672
Northern Cape 494 1 281
Western Cape 1 715 6 047
Total 39 351 80 218
12
2010 No of candidates 2010 No of candidates
Fulltime 562 473
Part-time 80 218
Total 642 691
2009 No of candidates 2009 No of candidates
Fulltime 580 360
Part-time 39 351
Total 619 711
13
Integrated Examination Computer System

• The examination computer system is the backbone
  of the administration of the NSC - the system is
  still being enhanced.
• All candidates have been registered on the
  Integrated Examination Computer System (IECS).
• Registration data has been audited by the DBE.
• A third schedule of Entries has been sent to all
  schools to ensure accurate registration of all
  candidates.
• At the end of August 2010, mark sheets will be
  printed.
• Letters of admission to the examination will be
  sent to schools in September 2010.
• In 2010 a full parallel run on the old system
  will be conducted to improve confidence in the
  processing of results
• Both systems are maintained and supported by the
  State Information Technology Agency (SITA)

13
14
Question Papers

• Security surrounding all processes relating to
  preparation of question papers is satisfactory.
• All question papers are set by the Department of
  Basic Education to ensure a national standard.
• 260 question papers have been set by panels of
  examiners and moderated subject specialists.
• The question papers have been moderated by
  Umalusi to ensure appropriate standard
• The first batches of question papers have been
  handed to PEDs for printing and packing
• Final handover of question papers is 10 September
  2010.
• Question papers are being brailled for blind
  candidates and adapted for deaf candidates.

15
Printing, Packing and Distribution

• All staff working in the printing and packing
  sections have security clearance and have signed
  confidentiality agreements.
• PEDs have received the first two sets of
  question papers printing.
• Question papers will be distributed to
  examination centres on the morning of the
  examination, except where extreme distances are
  involved.

15
16
Preparation for the examination

• Invigilators who supervise the writing of exams
  will be trained across all PEDs in September.
• The principal is usually the Chief Invigilator.
• Detailed regulations govern the conduct of exams
  and the managing of irregularities DBE has
  issued a Guideline document to assist this
  process
• Markers and chief markers have been appointed/
  will be appointed in September 2010.
• Marking centres have been finalized and markers
  will be trained before marking commences.

17
Answer scripts

• Each answer script is the product of a
  candidates 12 years at school and his/her
  performance is encapsulated in the script.
• In 2010, emphasis is being placed on the return
  of answer scripts, and management of scripts
  during marking process
• All scripts to be returned to the district office
  the same day
• Scripts to enjoy the same attention and security
  as the question paper.
• All scripts correctly labelled with exam number
  and subject.
• Scanning of scripts to be introduced by all PEDs
  as soon as possible.
• All scripts to be checked at each stage of the
  process.
• Implementation of sanctions where Chief
  Invigilators do not manage the return of scripts
  correctly.

17
18
Examination Centres

• All examination centres are audited to ensure
  adequate resourcing.
• All examination centres are monitored, with a
  special focus on examination centres with a
  history of irregularities.
• Centres that do not satisfy national criteria are
  de-registered and alternative centres are
  arranged.
• This includes private centres.

19
Marking

1. The quality of marking will be the focus in 2010.
2. Marking centres have been established across all
   PEDs based on the prescripts contained in the
   Policy.
3. Markers have been appointed based on the PAM
   criteria.
4. Training of markers is ongoing and specialised
   training sessions will be organised prior to the
   marking sessions.
5. National Marking Guideline discussions will be
   hosted in Pretoria to ensure standardisation of
   marking across all PEDs
6. Internal moderators to be appointed for each
   subject at each marking centre.
7. DBE will appoint external moderators to evaluate
   the marking at each centre.

20
School Based Assessment (SBA)

• School based assessment constitutes 25 of the
  final promotion mark.
• SBA is moderated at school, district and
  provincial level.
• PEDs monitor the collection of marks from schools
  quarterly.
• DBE issued a Guideline document to assist PEDs
  with the collection, control and capture of SBA
  marks to address shortcomings experienced in
  previous examinations, such as outstanding marks.
  
• SBA marks are currently being moderated by the
  PEDs and capture will be completed by 15
  November.

21
Processing and Release 2010

• Marking will be completed by 16 December across
  all PEDs.
• Mark Capture will be completed by 18 December .
• A technical team will prepare proposals in
  respect of adjustments for the pre-standardization
  meeting 21 December.
• Pre-standardization meetings will be held on 22
  23 December.
• Umalusi standardisation meeting to be held on 24
  December.
• Results will be thoroughly checked by DBE, PEDs
  and Umalusi on 28 30 December.
• Umalusi approval meeting to be held on 31
  December 2010.
• Results will be released on 6 January 2011.

22
Analysis and Feedback

• Detailed analysis of results per school, per
  district, per province, per subject will be made
  available.
• Reports from chief markers/internal moderators
  will be consolidated for distribution to schools.
• Outcomes of the Umalusi standardisation meetings
  will be made available to curriculum specialists.
  
• Workshops will be conducted with teachers and
  subject advisors.
• Under-performing schools will be brought to
  account and improvement plans will be designed.
• Curriculum specialists to develop subject
  improvement plans.

23
Overview of State of Readiness

• The monitoring exercise was severely impacted on
  by the public service strike - not possible to
  visit district offices and schools in North West
  Northern Cape.
• It was established that schools are filtering
  candidates in respect of
• progression to Grade 12 and
• the subjects that they may offer.
• School Assessment Irregularities Committees are
  not yet functional.
• The collection and reconciliation of answer books
  from examination centres the exams needs
  attention.
• At several district and circuit offices storage
  facilities were inadequate.
• In the electronic age, security of examinations
  now also means securing electronic copies of the
  question papers this needs attention.
• Varying SBA moderation practices within and
  across the provinces do not give confidence that
  all candidates treated in a fair equitable
  manner.
• DBE is engaged in the process of developing
  guidelines for moderation which should resolve
  the current plethora of practices.

24
Progress in Mpumalanga

• DBE is managing the NSC, SC and ABET examinations
  in the province and working collaboratively with
  the regional staff.
• DBE staff are deployed to the province.
• Certain functions have been decentralised to
  regions while DBE will manage centralised
  functions.
• Focus is on establishing improved systems and
  processes and building staff capacity.
• Supplementary, Senior Certificate and June ABET
  Level 4 examinations have been successfully
  administered.
• All question papers to be printed and stored in
  Pretoria, and will delivered on a weekly basis to
  the regional offices or nodal areas and daily to
  the examination centres.
• Scripts will be collected from schools daily

25
the black box

• McKinsey report quality of an education system
  cannot go beyond the effectiveness of class room
  interaction
• Low impact of teacher human capital means our
  measures are inadequate
• School organization

26
NSC CHALLENGES

• Five week world cup recess may have impacted on
  the teaching and learning momentum.
• The teacher strike has resulted in a school shut
  down.
• NSC policy is still not fully understood by
  learners, parents and educators.
• Security of question papers.
• Introduction of a new examination computer
  system.
• Moderation of the quality and standard of SBA
  needs attention.
• Conduct of examinations at private centres is not
  always satisfactory.

27
2010 ANA STATE OF READINESS

• Awareness
• On the whole provinces ,districts and schools
  visited are aware of the continued implementation
  of ANA in 2010.
• Plans are afoot to strengthen advocacy around
  awareness especially in districts and schools.
• Communication strategy
• Advocacy around ANA needs to be strengthened and
  monitored
• its purpose
• administration
• capturing, analysis, reporting, and
• post assessment improvement plans and support
• Communication and flow of information needs to be
  improved to ensure a consistent message
  communicated at all levels.

28
2010 ANA STATE OF READINESS

• Statistics
• Learner statistics must be verified before
  sending to DBE
• provinces will be requested to deal with
  shortages.
• Storage of test papers in districts
• Inadequate space to store the large quantities of
  Gr 1 6 test papers in districts.
• PEDs will need to rent warehousing space in some
  districts.

29
Recommendations

• Facilitate and maintain rigorous moderation
  systems for SBA.
• Develop a policy for the moderation of SBA
• Monitor provincial moderation systems.
• Moderate sample assessment tasks and learner
  evidence in the case of Life Orientation July
  and October 2010.
• Moderate SBA tasks and learner evidence.

30
Recommendations

• Provide diagnostic feedback to schools.
• Develop comprehensive analysis (qualitative and
  quantitative) of data.
• Detailed reports made available to provinces,
  districts and schools.
• Conduct post test analysis of ANA and public
  examinations.

31
Overview

• filtering candidates in respect of progression to
  Grade 12. It was also evident in visits to
  schools that restrictions were placed on certain
  learners in respect of the subjects that they may
  offer.
• 2. the establishment and functionality of the
  School Assessment Irregularities Committee is
  neglected

32
Overview

• PEDs need to focus on the collection and
  reconciliation of answer books from examination
  centres to marking centres and storage as
  adequate systems are not in place in all
  provinces to ensure that all scripts are
  accounted for.
• Generally storage facilities at Provincial
  offices are excellent but at several
  district/regional offices and circuit offices
  storage facilities were well below the expected
  norms and standards for a secure storage facility
  and provinces must improve storage facilities as
  a matter of urgency.

33
Overview

• Security is always a concern in examinations but
  in the electronic age, security of examinations
  is fast moving away from merely securing physical
  copies of the question papers. There is now a
  need to secure the electronic copies of the
  papers.
•  Moderation practices vary across the provinces,
  and even within provinces. The Department of
  Basic Education is engaged in the process of
  developing guidelines for moderation which should
  ensure that each candidate is being treated in a
  fair manner.

34
Overview

• One province has been requested to visit their
  printing and packing procedures to improve
  security in this area.
• The Annual National Assessments will be printed
  nationally this year. Provinces indicated that
  this would alleviate many problems. The key area
  of concern at the moment is that schools provide
  accurate statistics in respect of learner numbers
  and the Language of Learning and Teaching.

35
CONCLUSION

• Based on the information that was made available
  to the monitoring team, it can be concluded that
  the Provincial Departments of Education have made
  significant strides towards readiness for the NSC
  examinations
• However in respect to the annual national
  assessment, learner preparation has been
  disrupted by the strikes and it is recommended
  that these are shifted to February 2011. There
  are educational reasons to support that such
  tests are in future conducted earlier than later
  in the year

36
Conclusion

1. Learner preparation Registration,
   irregularities, subject changes, item analysis
2. Stratification of the learner attainment
   strategy
3. System administration
4. Validation of standards
5. Credibility of Site-Based Assessment
6. Logistics regarding the ANA
7. Analysis of ANA results
8. Item writing, banking and discrimination
9. Promotion requirements
10. Release of results
11. Top candidates