The main functions of this unit are to set standards for assessment of qualifications on the General and Further Education and Training Qualifications Framework and to ensure that assessment for certification in schools, Further Education and Training Colleges and Adult Education and Training Centres is of the required standard.
This is achieved through the following processes:
|Why does Umalusi have to quality assure matric and other examinations?|
In terms of the GENFETQA Act of 2001(as amended in 2008), Umalusi is required to approve the release of results once it is satisfied that the examinations have been conducted in a credible manner. The Act also indicates that Umalusi is permitted to adjust the raw marks when necessary.
Gaining the approval of the Umalusi Council for the release of the results is a complex process, of which standardization is one of the last steps. Approval of results is determined by how well the assessment bodies responsible for the examinations have complied with all the policies, directives and guidelines related with the qualification being resulted.
These policies and directives are issued by Umalusi and the Departments of Basic Education and Higher Education and Training, and also by the Independent Examinations Board (IEB) and SACAI: (South African Comprehensive Assessment Institute) for their own systems. Such approval is also dependent on whether any irregularities could be considered to have undermined the credibility and integrity of the examinations.
|What is standardisation?|
Standardisation is a process used the world over to mitigate the effect of factors other than the learners` knowledge and aptitude on the learners` performance. In South Africa, the standardisation of results has been used since 1918 by Umalusi`s predecessors, the Joint Matriculation Board (JMB) (1918-1992) and South African Certification Council (SAFCERT) (1992-2001). In other words, all South Africans who have obtained their qualifications through these bodies have had their subjects standardised prior to their results being announced.
A large-scale examination system such as the National Senior Certificate (NSC) inevitably experiences many sources of variability, despite the best efforts of examiners, markers and moderators. These can include changes in levels of difficulty in question papers across years: so, for example, one year`s Mathematics papers may be more difficult than the ones in the preceding years, and so, to prevent that cohort from being unduly disadvantaged, their marks may be adjusted marginally towards the historical average. Variability may also result when errors occur in papers, and inconsistency may arise in the marking, for example, across provinces.
Standardisation works on the widely accepted assumption that, for large populations, the distribution of aptitude and intelligence does not change appreciably from one year to the next or from one large population to a similarly large population; and that it is therefore reasonable to expect that, all things being equal, this year`s cohort of learners should perform at a level more or less comparable to last year`s cohort. Standardisation is the accepted process used to reduce fluctuations in learner performance that result from identified factors within the examination processes themselves rather than from the knowledge, aptitude and abilities of the learners.
|Are the standardization decisions taken by Umalusi confidential?|
Prior to 2010, the standardization decisions were treated as confidential. All international assessment bodies treat this information as confidential. In 2010, Umalusi took the decision to disclose the standardisation process and standardization decisions to take the public into confidence by demonstrating the integrity of the process.
|Who is involved in standardisation?|
Standardisation is a process where Umalusi, as the quality assuror takes into account - on a subject-by-subject basis - all the factors which may have unduly advantaged or disadvantaged the cohort of learners under consideration.
While Umalusi`s Quality Assurance of Assessment (QAA) unit is primarily responsible for the ongoing quality assurance work, it works closely with a committee of Umalusi Council, the Assessment Standards Committee. This committee is responsible for the standardisation and moderation of internal assessment results as well as the examination results for all qualifications that Umalusi certifies.
The Assessment Standards Committee comprises statisticians with relevant experience and knowledge in the handling of statistically-oriented research projects, research design, the conduct and evaluation of research projects and statistical standardisation processes.
It also includes professionals in education with specific knowledge, experience and expertise in assessment and curriculum, and in handling of system-wide assessment research projects.
|Who conducts the quality assurance processes?|
All processes are overseen by a Committee of Umalusi Council namely the Assessment Standards Committee. The Quality Assurance of Assessment (QAA) Unit ensures that all processes are carried out effectively and efficiently. Part of this is the contracting of external moderators for each subject within a qualification (sometimes moderators work in panels), monitors and other professional staff as required. Generally contracted staff is drawn from universities and senior staff levels in providers who receive annual training to ensure consistency.
|How does the standardization process work?|
It is important to recognize that many quality assurance efforts precede standardisation. These include the moderation of examination papers, and where relevant, the moderation of internal assessment components which contribute to learners` final marks; the monitoring of the conduct and administration of examinations; research into the quality and comparability of the papers across years, as well as the marking and moderation of marking. However, it is only at standardisation that the global picture of the performance of all candidates in each of the subjects can be considered.
In recent years, standardisation has been immeasurably enriched by the research processes that Umalusi has undertaken with regard to the major, high-volume subjects in both the National Senior Certificate and the National Certificate (Vocational). The inclusion of expert judgements in the standardisation process has come about as a result of the introduction of these two new qualifications at Level 4. The findings of the Umalusi evaluators supplement the information provided at standardisation by moderators and chief markers. This qualitative information is used to motivate and support the decisions on a subject-by-subject basis.
Umalusi has a set of general guidelines and principles, which have been developed over years of practice with the nation`s top statisticians and educationists, to ensure that each cohort of learners is treated as fairly as the cohorts that went before it.
|How does standardisation happen?|
In order to guide the final standardisation meetings with each of the assessment bodies - the Department of Basic Education, the Department of Higher Education and Training, ERCO and the IEB - the Assessment Standards Committee holds pre-standardisation meetings. There it receives, among other things,
It then deliberates on the preliminary results submitted by the relevant assessment body, and, in the light of all reports received, decides upon a course of action - to accept the raw marks (always the preferred option as it means that the other moderation processes have functioned well), or to consider an adjustment to the marks.
At the standardisation meetings, delegations from the assessment bodies present their understanding of the subject results for Umalusi`s consideration. Unless the assessment body presents new information regarding a factor that has not been considered in the pre-standardisation meeting - one which is felt to impact on the validity of the examination results - the decision taken at pre-standardisation stands.
At the standardisation meeting of the National Senior Certificate, for example, the delegation of the Department of Basic Education is led by the Director General. The delegation from the Department includes executive and senior managers of the Department as well as the Heads of Provincial Departments of Education and their senior managers. A similar delegation from the Department of Higher Education and Training works with Umalusi in the resulting of the National Certificate (Vocational). Each of these teams will have had their own pre-standardisation meetings to prepare for the meeting with Umalusi. The ERCO and the IEB delegations will have done the same.
Observers from the South African Qualifications Authority (SAQA), Higher Education South Africa (HESA), the South African Council of Educators (SACE) and all the teacher unions are invited to attend. In addition, colleagues from other examination boards or councils elsewhere in Africa are invited to attend as observers.
During the standardisation meetings, Umalusi and the assessment body concerned deal with the results on a subject-by-subject basis. In other words, it is not possible at that meeting to establish what the overall pass rate will be. The final pass rate will emerge only once the results for the individual candidates have been compiled - a job that can only happen after standardisation has been completed and the standardisation decisions effected.
At the standardisation meeting, the assessment body is allowed to make a case for adjusting the marks for a subject or recommends that the raw marks be accepted as final. Umalusi`s Assessment Standards Committee then may interrogate the position proposed by the assessment body, and then makes its decision in the light of all the information available to it.
Once the Assessment Standards Committee is satisfied that candidates have been fairly treated and that the results are a fair reflection of the cohort`s performance on that particular set of examinations, it recommends to Council that it approve the results, and that they may be released by the Minister concerned.
|What are the processes involved?|
|Which assessments does Umalusi quality assure?|
Umalusi is the Quality Council responsible for setting standards and quality assuring qualifications for general and further education and training. As such it has developed a sub-framework of qualifications which consists of existing qualifications and proposals for some new qualifications. The existing qualifications that are assessed through national examinations are as follows:
The Senior Certificate (SC) which was offered before 2008 and is being phased out - its final examination will be administered in 2014
The National Technical Certificate N3
Adult Education and Training qualifications;
The General education and Training Certificate for adults (GETC: ABET)