How comparative education data are used

 Charles Ungerleider, Professor Emeritus, The University of British Columbia

[permission to reproduce granted if authorship is acknowledged]  

To avoid making egregious mistakes and embarrassing myself, I have several respected colleagues who read my blogs before I post them. A reader of last week’s blog suggested that I devote an entire blog to the reasons why comparative analyses are helpful and to whom.  

Comparative analysis involves examining and interpreting data from different schools, districts, or educational systems to identify patterns, trends, and areas in need of improvement. Such analyses are valuable for many stakeholders, including parents, educators, policymakers, advocacy groups, researchers, and others.  

Benchmarking and performance evaluation are the primary reasons why Canadian provinces and territories cooperate under the ambit of the Council of Ministers of Education, Canada (CMEC) in the Pan-Canadian Assessment Program (PCAP) and in the Organisation for Economic Co-operation and Development (OECD) Programme for International Student Assessment (PISA). PCAP and PISA data are useful starting places for answering the question how are we doing in comparison with other jurisdictions now and over time.  

Benchmarking performance is the first step in the process of performance evaluation which, if done well, will help inform policy and decision making. Governments and policymakers use comparative data to evaluate the effectiveness of educational policies and programs. By comparing data across different jurisdictions, they can identify best practices and areas needing improvement. This information can guide the development of more effective educational strategies and policies.  

Comparative analyses are also essential in educational research. Researchers use data to study various aspects of the education system. Comparative data contribute to understanding the effectiveness of education changes and the relationship between education and socio-economic outcomes.  

From my perspective, among the most important contributions of comparative analyses is understanding educational inequities. Comparative data analysis can reveal disparities in educational outcomes across different regions, socio-economic groups, or ethnic backgrounds, indicating where policy interventions to promote equity in education are needed. Without comparing data overtime, it would be impossible to know if efforts to reduce inequalities and produce more equitable outcomes were successful.  

Comparative data can inform us about where resources are most needed, and they can help in allocating efficiently to where they are most needed. Comparative data analysis helps reveal disparities and needs across different regions or demographics and is helpful to decision-makers in evaluating their budgetary allocations, leading, one hopes, to allocations that are economical, efficient, and effective. Regular comparative analysis fosters accountability. Monitoring and reporting on performance metrics is a key responsibility of boards and ministries of education.  

Comparative analysis is a tool for strategic planning. It helps the governors of systems to set long-term goals and administrators to establish objectives based on empirical evidence. Comparative data analysis is essential for advocacy organizations that wish changes. Organizations focused on education use comparative data to highlight disparities in access to programs or outcomes.  

Comparative data analysis in elementary and secondary education is a powerful tool for improving educational outcomes. It helps us understand how distinct factors contribute to student success and helps decision-makers make informed decisions to enhance the quality of education.  

For all these reasons it is important and valuable for Canada and its provinces and territories to further their efforts to produce consistent, reliable and comparable education statistics

Data comparability in elementary and secondary education is hard to achieve in Canada

 Charles Ungerleider, Professor Emeritus, The University of British Columbia

[permission to reproduce granted if authorship is acknowledged]  

I received an email from colleagues complaining about how difficult it is to obtain comparable data about elementary and secondary schooling in Canada. “That’s the truth,” was my frustrated and, I am sure, frustrating reply.  

The decentralized nature of Canada's education system makes gathering comparable data from each of the Canadian provinces and territories in the context of elementary, secondary, and, for that matter, post-secondary education challenging. Each province and territory in Canada has jurisdiction for its education systems. That autonomy leads to subtle, but important, differences that make inter-provincial comparisons difficult. Differences among provincial systems, policies, and curricula lead to variations, albeit often minor, in what is taught, how it is taught, standards for student achievement, and almost everything else. As a result, data collected from one region may not be directly comparable to another.  

Further complicating the matter is the fact that methods, standards, and timing of data collection vary across provinces and territories. This includes differences in the types of data collected, the tools, and technologies used for data gathering, and the metrics for measuring educational processes, performance, and outcomes.  

Those seeking to make comparisons will no doubt notice that the academic year, the age at which children enter, and the number of hours of instruction differ from province to province. These variations affect the comparability of data related to enrollment, progression, and attainment.  

For those who seek to make comparisons and are cognizant of the need to control the variations affected by system differences and the demographic differences among provinces, obtaining data is a challenge. The willingness of provincial and territorial governments to participate in national data collection initiatives varies, influenced by political, fiscal, and administrative priorities.  

Canada lacks a centralized data repository.  Without a centralized national database for educational data, collecting, standardizing, and comparing data from different regions is complex and resource intensive. Canada needs a coordinated approach and standardized guidelines for data collection and sharing across Canada's educational jurisdictions to facilitate meaningful comparative analysis. The Councils of Ministers of Education, Canada (CMEC) tries to address the need for comparable data.  

The Pan-Canadian Education Indicators Program (PCEIP) is a partnership between CMEC and the Canadian Education Statistics Council, a body jointly chaired Statistics Canada and CMEC. It tries to provide a set of statistical measures on education systems in Canada using in accordance with common data standards and definitions. When I served as deputy minister of education for BC, I co-chaired the Canadian Education Statistics Council with the Chief Statistician of Canada. I saw first hand how challenging it was to produce data that would enable comparisons across jurisdictions.  

Despite considerable effort, the available data about elementary and secondary schooling are meagre. In fact, as far as achievement data are concerned, PCEIP depends greatly upon the Pan-Canadian Assessment Program (PCAP). PCAP, conducted by the Council of Ministers of Education, Canada (CMEC), assesses the abilities of 13- and 16-year-old students in reading, mathematics, and science. PCEIP also depends on the OECD’s Programme for International Student Assessment (PISA) that examines the performance of 15-year-olds in mathematics, reading, and science.  

Those of us interested in cross-jurisdictional comparisons look to PCEIP, the Canadian National Household Survey, the Census, and the Labour Force Survey and try to assemble data that would enable meaningful interpretation. It isn’t easy.  

We also seek data from the provinces and territories, but transparency and availability of educational data vary significantly among Canadian provinces and territories. Some regions are known for being more open and forthcoming with their educational data. I wish all provinces were as open about data as British Columbia. More than a decade ago, BC became the first province to ‘publish’ its data under an open license.  

BC makes a data catalogue available to the public. I often seek data about student performance, enrollment statistics, and graduation rates from the data warehouse. There are data that I want that I cannot find there. Attendance data are something I am looking forward to seeing. But getting there means the province will need to establish common definitions and data collection procedures to ensure that the data provided by one school or district is comparable with another.  

Ensuring comparability in data is crucial for accurately understanding trends and their broader implications. This requires standardized methods for collecting data, uniform definitions across studies, consistent measurement units, and synchronized timeframes for data collection. These goals are not overly complex but achieving them demands significant political commitment and administrative effort. British Columbia has exemplified such commitment and effort. However, the relative consistency in the PCEIP reports over time suggests a limited commitment from provincial politicians, regardless of their party, to understanding educational trends and their implications.