Policy linking makes use of a standard-setting methodology (the Angoff approach for those familiar with standard setting methodologies), long used in many countries, to set benchmarks (also known as “cut scores” or “thresholds”) on learning assessments. While it is an old standard-setting methodology, UIS and its partners have now extended its use to help countries set benchmarks using the Global Proficiency Framework (GPF), a framework developed by multilateral donors and partners based on current national content and assessment frameworks across more than 100 countries. The GPF provides performance expectations/standards for learners in Grades 2-6 in reading and mathematics. By linking their national assessments to the GPF, countries and donors are able to compare learning outcomes across language groups in countries as well as across countries and over time, assuming all new assessments are subsequently linked to the GPF. Policy linking allows countries to use their existing national assessments or early grade reading and math assessments to report against Sustainable Development Goal (SDG) 4.1.1 as well as some donor-required indicators, such as USAID’s Foreign Assistance (“F”) indicators.
The policy linking methodology is elaborated in the Policy Linking Toolkit, which provides guidance and templates to countries, donors, and partners who conduct policy linking workshops to set global benchmarks. The toolkit and the accompanying Quality Assurance Policy specify the steps to be taken before, during, and following the workshops to ensure consistency and, as a result of comparability of the outcomes.
The policy linking methodology has five stages:
- Collection of evidence, such as a country’s curriculum framework, the assessment instrument a country wishes to align with the GPF, data from the most recent iteration of the assessment, and details of the assessment design and sampling methodologies.
- Submission of the evidence gathered and review of the materials by the Policy Linking Quality Assurance Panel.
- Implementation of the policy linking workshop, and elaboration of a report on the workshop and evaluation outcomes.
- Submission and review of the workshop outcomes by the Policy Linking Quality Assurance Panel to independently confirm the robustness of the process and the reliability of the outcomes.
- Reporting of the results against SDG indicator 4.1.1.
Global Proficiency Framework
The Global Proficiency Framework (GPF) was developed in the Spring of 2019 by multilateral donors and partners based on research conducted by UIS into the content and assessment frameworks of more than 100 countries. It defines the global minimum proficiency levels (minimum knowledge and skills) that learners should be able to demonstrate along their learning progression in grades 2-6 in reading and math. In addition to providing expectations for students who meet minimum expectations, it also includes expectations for students to partially meet minimum expectations and to exceed minimum expectations. The purpose of the additional performance levels is to allow countries to show progress toward meeting and exceeding the target for SDG 4.1.1. UIS has plans to extend the GPF to include grades 7, 8, and 9 as well.
The GPF provides detailed minimum proficiency expectations (called Global Proficiency Descriptors—GPD) that countries, along with regional and international assessment organizations, can use as a foundation for linking existing – and future – reading and mathematics assessments to SDG 4.1.1 via policy linking or other linking methodologies. Once assessments are linked to the GPF, the results can then be compared across the different assessments and languages, both within and across countries..
Piloting the policy linking
The policy linking methodology was piloted in two countries in Asia, India and Bangladesh, in the Fall of 2019 and in Nigeria in March 2020.
In India and Bangladesh facilitators led panellists in setting benchmarks for their national assessments from 2017 based on the Global Proficiency Framework (GPF) for grades 3 and 5 in mathematics and reading (in English and Bangla, respectively). In Nigeria, facilitators led panellists in setting benchmarks for USAID’s Early Grade Reading Assessment (EGRA) in Hausa for grade 2 and 3 students. In all cases, the policy linking workshops brought together 15-20 panellists per subject and grade. Panellists were master subject teachers, curriculum/content specialists, and assessment specialists, representative of the area for which benchmarks were being set (all the main districts of Bangladesh, the Northeast and Delhi states in India, and Hausa-speaking states in Nigeria. They were nominated by their governments to be invited to the workshop.
The event consisted of three main activities:
- Reviewing the alignment of the assessments with the GPF, with each panellist rating the items. This process found:
- Matching assessments items with the established Global Proficiency Levels (GPL) in the GPF, with each panellist determining the levels of knowledge and skills required from students to correctly answer each aligned item.
- Setting global benchmarks for the assessment, with each panellist rating whether partially meets, meets, and/or exceeds minimum proficiency students would be able to correctly answer each aligned items using Angoff procedure. This rating process was completed twice, and the individual panellist benchmarks were then averaged to calculate the overall benchmarks. Then, these benchmarks were applied to the data from the most recent assessments completed in each country (2017 in India and Bangladesh and 2016 in Nigeria) to define the percentage of children who scored above each of the proficiency levels.
The panellists expressed a high degree of confidence in the policy linking methodology and the benchmarks. The outcomes and conclusions of the workshop were shared with the national governments for their feedback and comments.
A global consortium of donors and partners has developed a draft Quality Assurance Policy and mechanism for the methodology to ensure minimum standards in test validity and reliability are in place and that assessments are sufficiently aligned with the GPF ahead of policy linking workshops and to ensure rigor in the implementation of the methodology. The Quality Assurance Panel tasked with implementing this policy will review country reports and make recommendations as to whether results should be accepted by UIS for reporting against SDG 4.1.1.
Given that the policy linking methodology has not been tested for cross-national comparisons, it is necessary to evaluate if the approach works for this new purpose. As such, the Bill and Melinda Gates Foundation, in coordination with UIS, has procured an evaluation focused on reviewing the efficiency and effectiveness of the methodology and comparing it with statistical linking in an effort to identify whether it is suitable for reporting against SDG 4.1.1 and for comparing outcomes on the percentage of learners meeting minimum proficiency levels both within and between countries and over time. The results of the evaluation will be shared widely and used to determine whether UIS continues to move the methodology forward, and, if so, what modifications might be needed to improve the rigor of the methodology.
The figure below presents details on the development and piloting of the policy linking approach to date.
The following figure shows a tentative timeline for the remaining steps UIS plans to take in coordination with its partners to finalize piloting of the methodology and begin to scale it in 2020. This timeline may change as a result of the circumstances surrounding the spread of COVID-19, which may delay pilots or require contingency planning for some of the planned steps.