A focus on low-performing students produces better system outcomes and greater equity

Charles Ungerleider, Professor Emeritus of Education, The University of British Columbia

[permission granted to reproduce if authorship acknowledged]

In last week’s blog post, I tried to draw attention to other ways of calculating how well a school’s student population is performing. This blog makes the case that how we calculate performance reflects what we value. I argue that we should place greater value on improving the performance of low-performing students because doing so will produce better outcomes for them, for the communities in which they live, and for society.

To save you the effort of clicking to the earlier blog, let me explain that I contrasted two approaches.  One approach was to create a “performance index” by calculating the percentage of students in a school who had achieved or exceeded a “provincial standard” in a jurisdiction. This adopts what I called a levels approach. I explained that, in a levels approach, the score or grade a student earns on some assessment indicates that: the student has not begun learning or was excused from the assessment (level 0); has begun learning but hasn’t made much progress (level 1); is progressing but is not quite at grade level (level 2); is firmly performing at grade level (level 3); or is exceeding the expected performance for students at the grade level (level 4). The levels approach is the same as assigning grades (A, B, C, D, F). 

The problem with this approach is that it ignores what is happening for students in levels 0 and 1. If, for example, school performance is measured by changes in the performance index over time  it is reasonable to consider students who move from level 0 to level 1 and students who move from level 1 to level 2.  As a way of including these students in the performance index I offered the idea of a weighted mean. What I meant by a weighted mean is that I multiplied the number of students at each level by the value of the level achieved (level 0, level 1, level 2, level 3, or level 4), adding the products together, and dividing the sum of those products by the sum of all the students assessed including the students below level one and the students excused from the assessment.

I compared calculating the percentage of students in a school meeting or exceeding the provincial standard with the calculation of a weighted mean. I illustrated how using weighted means provides more information about the performance of students in a school than simply reporting the percentage at or above some level. In my illustration, I created 8 fictional schools in which the proportion of students at or above level 3 was 75% and varied the proportions of students performing below level 3.

This approach showed more clearly the mathematical differences among schools that had the same proportion of students meeting the provincial standard. I argued that the differences conveyed a more subtle and complete picture of student performance.

Let’s go one step further. In the weighted mean I calculated, the weights were the percentage of students in each level. Instead of the weights I assigned in my illustration, imagine that I have assigned greater weight to bringing students from level 0 to level 1, from level 1 to level 2, and from level 2 to level 3. (The weights still must add to 1.) I assign those weights because, I place a greater value on teaching a student who does not know how to read, for example, to read at a rudimentary level or teaching a rudimentary reader to read competently than I do on teaching a competent reader to excel. If one wants to improve the overall performance of an educational system, focusing on the low-performing students is the effective and the most equitable pathway.

I make this argument knowing that, on average, low-performing students come from less affluent backgrounds, have fewer family and community supports, and, if they remain low-performing, are more likely to drop out of school. If low performing students complete school, they are less likely to go on to further education, will have more precarious employment, earn less, pay less tax, and be more likely to have children who suffered from the same disadvantages as they did.

Breaking this cycle by focusing on improving their performance is a benefit to them, to the communities in which they reside, and to society.  

Happy holidays . . . see you in 2020, Charles

You can't change what you don't understand

Charles Ungerleider, Professor Emeritus of Education, The University of British Columbia

[permission granted to reproduce if authorship acknowledged]

Confronted by what seems to be a problem, educators often jump to a solution without knowing exactly where the problem lies. It’s not surprising, but it is a distressingly common and costly situation in education that occurs at the classroom, school district, and at the provincial level.

Imagine eight schools in which only 75% of students meet or exceed the “provincial standard” using a levels approach to educational measurement. What I mean by a levels approach is that the score a student earns on some assessment indicates that: the student has not begun learning or was excused from the assessment (level 0); has begun learning but hasn’t made much progress (level 1); s/he is progressing but is not quite at grade level (level 2); is firmly performing at grade level (level 3); or is exceeding the expected performance for students at the grade level (level 4). Although the meaning of the symbols may differ, the levels approach is the same as assigning grades (A, B, C, D, F). 

On paper, the 75% figure for level 3 and 4 students may look good. But merely adding together the level 3 and 4 students doesn’t inform decision-making at the school, district or provincial levels about students who are not at or above provincial standard.

Furthermore, in a school where 25% of the students have not met the provincial standard there may be a relatively large proportion of students who have not even achieved level 1 or who have been excused from the assessment. Acknowledging the percentage of students at each level provides a more complete understanding of student performance.

Providing Additional Information

Weighted averages provide more information than can be conveyed in a simple percentage. A weighted average – instead of treating all scores equally – considers the proportional relevance of each score.

I have computed weighted averages for the performance of students for a set of fictional schools. I did that by multiplying the number of students at each level by the value of the level achieved (level 0, level 1, level 2, level 3, or level 4) and adding the products together. I divided the sum of those products by the sum of all the students assessed including the students below level one and the students excused from the assessment to find a weighted average for each school. I think this approach provides a more nuanced look at differences among schools.

Here’s how it looks: an illustration

The table below contains the data from the fictional schools. 75% of the students in each of the schools are at or above the provincial standard (levels 3 and level 4), but there are some important differences among them that affect the weighted average of their scores. For example, the proportion of students at levels 3 and 4 at all the schools are the same, but at Bay View 50% are at level 3 and 25% at level 4, while at River View those proportions are reversed. The students at River View (weighted average 3.15) are outperforming the students at Bay View (weighted average 2.90). At Coleman the proportion of students at levels 3 and 4 are the same as Bay View, but the proportion of students at levels 1 and 2 at Coleman are exactly the opposite of those at Bay view (15 % level 1 and 10% level 2 versus 10% level 1 and 15% level 2). That small percentage change is reflected in the difference in the weighted average for Coleman (2.85).

By comparing the schools where the colours are the same (Bay View with Westbrook, River View with Sea View, Coleman with Oceanside, and Queen Anne with St. Lawrence), you can see the weighted averages decline when students who have been excused and students who are below level 1 are included in the calculation. This provides a more accurate picture of the results obtained. It also has a positive impact on morale when teachers efforts across the entire range of levels are recognized and acknowledged, as opposed to reporting only the percentage that met or exceeded the provincial standard.

Schools in which the percentage of students meeting the provincial standard were identical can be differentiated from one another. Examined more closely, the differences paint a more subtle and complete portrait of student performance. The numbers are just a starting point. Once you have them its time to dig deeper to diagnose what’s going on. You cannot change something without understanding it.

Evidence in Education: Who needs it and why?

Charles Ungerleider, Professor Emeritus of Education, The University of British Columbia

[permission granted to reproduce if authorship acknowledged]

Like many, my attention has been drawn to the impeachment hearings in the United States. Barely a few seconds after a bit of testimony, someone cherry-picks some testimonial snippet to tweet accompanied by emotive language. Both the snippet and the emotive language are offered as proof of the twitter user’s previously held position.    

The Oxford dictionary defined post-truth as “relating to or denoting circumstances in which objective facts are less influential in shaping public opinion than appeals to emotion and personal belief.” The dictionary lists as an example of usage: “in this era of post-truth politics, it's easy to cherry-pick data and come to whatever conclusion you desire.” In other words, concern for the truth is no longer a consideration.

Educators should be especially vigilant about promoting the quest for truth and the evidence upon which truth depends both for the students for whom they are responsible and for themselves.  It is though the cultivation of respect for evidence as reasonable grounds for belief and action that educators safeguard society from tyranny. Educators who do not respect evidence with regard to their own practice are poor models for students. They are unlikely to encourage students to respect evidence or to use it.

Without splitting hairs too finely, there are probably three notions of post-truth that, in practice, get conflated.  At its worst, post-truth means “I don’t need evidence to come to a conclusion” – a complete rejection of evidence even when evidence is available.  Another version is: “I distrust and reject the evidence that you use. I don’t deny that there may be evidence, I just deny the evidence you cite in support of your claims. I will assert my claims even in the face of the evidence you present.” And third version of post-truth is “I prefer my evidence to the evidence that you cite in support of your claim and will draw my conclusions on the basis of the evidence I prefer.” This is what some have called alternative truth.

What is evidence?

Evidence is the available body of facts or information indicating whether a belief or proposition is true or valid. We use evidence to show the claims we make are reasonable or that our actions are right. If someone is committed to seeking the truth, they cannot cherry-pick the data that will support their preferred conclusion(s) or justify the course of action they wish or have pursued.  

Those committed to seeking the truth will suspend judgement until a claim is supported by the evidence. In the post-truth world, conclusions are often made before evidence supporting the claim is ‘discovered’ and contrary evidence debunked.

Evidence requires objective evaluation, meaning that the evidence that something is true should be independent of my individual proclivities, emotions, and interpretations or of yours.  The evidence I cite in support of my claims should be subject to independent verification. The evidence should have been gathered using methods sufficiently well-described and well-accepted that others can apply them to arrive at an independent appraisal of the claim.  

Those committed to truth seek the facts that would challenge the claim they wish to make in order to ensure that claim will withstand close examination and challenge. Experts seek rigorous review by their peers in order to ensure that their work withstands close scrutiny.

People – especially people who claim expert (professional) status - should not do something that is contrary to the available body of evidence without strong justification. Contrary to common usage, professional judgment does not refer to the judgement of the individual practitioner. It refers to the collective judgment of professionals who are competent to examine dispassionately the entire body of evidence that has been accumulated. Such examination takes into account many things, among them: how the topic or issue was defined, how the evidence relevant to the topic was gathered, how it was analyzed, and whether the conclusions drawn are supported by the evidence collected. In assessing the entire body of evidence available on the topic, educators should seek to determine if there is rational convergence of opinion among those qualified to appraise the evidence, taking into account its quality and its strength.

Simply citing research related to one’s beliefs or actions is a common form of cherry picking evidence to support one’s inclinations or behaviour. It is not unusual to read or hear a ministry, school board or school proclaim that an initiative or policy is evidence-based because someone has found a few studies that seem broadly related. Those making strong claims about what one should do or what one should believe should be able to show the specific link between the evidence the initiative or policy.

The Fraser Institute is a good example of an organization that uses evidence selectively. It cherry-picks data and presents it in a manner that will support the conclusion it seeks. To destabilize support for public services in general, and public education in particular, the Fraser Institute is selective about the evidence that it uses and the manner in which it presents the evidence to give the impression that public education is not doing very well. The “studies” produced by the Fraser Institute lack objectivity because its views about what is the case is guided by ideology rather than evidence.

How much evidence is sufficient?

Readers will no doubt notice that I’ve referred to reviewing the entire body of evidence. There are a number of practical reasons for this recommendation. Much educational research has been carried out by individuals who have a personal stake in the outcome of the study. Most investigators do not consciously attempt to influence the outcomes of a study, but nonetheless may inadvertently influence the results through the decisions they make about the research methods they use or the inferences they draw. This threat to objectivity warrants looking beyond single studies to the larger body of work.

It is regrettable that there are few, if any, follow-up educational studies that replicate an initial study to determine, for example, whether the same result was found under the same conditions or whether the result of the initial study applies to contexts or to students who differed from those in the original study. In the absence of such replications, looking across the body of work provides valuable, though imperfect, evidence about such matters.

We are fortunate that it has become increasingly common to conduct systematic reviews of evidence or meta-analyses of evidence. Many such reviews are available to educators from the What WorksClearinghouse, the CampbellCollaboration, Eurydice web sites, from publications such as the Review of Educational Research or Hattie’s visible learning series, and from systematic reviews published in individual journals.  Systematic reviews are not flawless, but they provide a firmer foundation for claims about policy and practice than single studies or groups of studies selectively chosen to suit the policy or practice one wants to pursue.

Why evidence matters.

Tyranny becomes more likely in a post-truth environment where prominent persons and institutions manifest a blatant disregard for the truth, little respect for evidence, and antagonism toward dissent.  One of the purposes of public schooling is to cultivate the ethical commitment to truth and respect for evidence. And the responsibility of doing so falls upon educators.

"It’s never too late to turn your life around or is it?"

Charles Ungerleider, Professor Emeritus of Education, The University of British Columbia 

[permission granted to reproduce if authorship acknowledged] 

Our expectation that presidents and prime ministers are models of socially-acceptable behaviour was upended with the election of Donald Trump. As a public figure, his misanthropic, misogynistic, and prejudicial behaviour appears to make such behaviour by others seem acceptable.  

Donald Trump is by no means the first public figure whose behaviour has caused significant concern. A highly visible sandwich chain terminated its spokesperson (who lost 200 lbs eating its sandwiches) when he was arrested and convicted of possession and distribution of child pornography and other offenses. A soft drink company severed its relationship with a prominent rapper when the rapper’s lyrics made an offensive and violent reference to a civil rights icon who was the victim of violence.  

Role models for children and youth have been on my mind off and on for twenty years or more. The partnering of school jurisdictions and professional sports teams seems like a good way to capitalize on the celebrity-status of athletes to promote something like reading until an athlete on the partnered sports team physically or sexually assaults someone.   

So, it isn’t just the Bully-in-Chief that has me thinking about role models. In fact, this blog was prompted by a bon mot on a mug someone gave me: “It’s never too late to turn your life around, so why start now?”  

The witticism made me think about the things I should do or should have done to improve my health but didn’t. I quit smoking at age 38. I never should have started in the first place but had many self-defeating reasons for continuing. For many of us the cheeky caption on the mug is a reminder that we should lose weight, exercise, and do countless other things that we postpone.  

But the quip also prompted me to think about role models in the context of education. What came to mind were the efforts made in school to educate about substance abuse and programs for children and youth to persuade them not to become involved in gangs. Many such efforts make use of role models, people who abused substances and no longer do, and former gang members.  

I am not certain what children and youth think when confronted with such models, but I fear that they may say “s/he abused drugs or alcohol or was a gang member and later changed her life. I can do that (abuse substances or join a gang) and change my life later, too.”  

I have been searching the literature and contacting researchers in a variety of fields to find evidence that will either allay or confirm my fears. I’d like to report that I’ve found the answer, but, so far, I have not. Several of the experts who have responded to my inquiries have said “good question, to which I do not have an answer.” 

The frequency with which we consciously present children and youth with role models with the intention of influencing their behaviour should prompt us to seek such evidence of its impact. In the meantime, we should proceed cautiously. 

Relying on one another for success

Charles Ungerleider, Professor Emeritus of Education, The University of British Columbia

[permission granted to reproduce if authorship acknowledged]

A high school in Columbia, Maryland implemented an innovative way to encourage low income students and students of colour to enrol in advanced classes. The initiative is worth a closer look.

According to an article in the Washington Post, Hammond High School’s advanced course enrollments looked the same year after year, despite a changing demographic that saw a larger number of students of colour and students from low-income families enrolled at the school. Based on other instances where the opposite happened, I wondered what made this possible.

First, the school responded to its changing student population by eliminating low-level classes and requiring all students to enrol in a minimum of one advanced class each school year. Then, the school established a suite of supports to assist the students who struggled with the more demanding coursework, including a summer workshop to prepare them for the more advanced classes: a homework club, common planning time for teachers in the same subjects, and professional development to assist teachers to learn new ways of supporting all learners.

The Hammond experience has been that minority enrollments in advanced courses have increased and so, too, have graduation rates. The graduation rate for African-American students increased from 80% to 92% from 2010 to 2016. The increase for Hispanic students went from 81% to 95% and, for students with special needs, the graduation rate increased from 56% to 80%.

The pattern at Hammond is for students to work in groups and to collaborate with one another. Group work engenders responsibility to the group which prompts students to work harder and to help one another. The approach was questioned in one of the comments accompanying the online article: “Group work at my kids' schools is too often used to turn the high performing into the classroom teachers and to get students who couldn't do the work through the course or to lessen the grading load.” This reader wanted to know the impact of the approach on high performing students: Is this approach something that the high performers prefer or endure? Does the approach adversely affect their scholarship?

The evidence from studies of well-planned collaborative learning shows favourable outcomes for all learners.  Studies of peer-tutoring typically show that both the tutor and tutee benefit, with the greater benefits accruing to the tutor.  Notwithstanding these bodies of evidence, the concerns raised in the comment are very legitimate.

I had several points in mind in writing about the Hammond innovation. One was that performance of students regarded at risk appeared to improve when the demands and expectations increased. Another was to encourage careful examination of promising innovations to ensure that the successes are genuine and are sustained over time, addressing the issues raised by those responding to the article. A third is to suggest that the return on the investment of such innovations should be calculated to determine the cost of the incremental gains achieved and to answer the question: Can this innovation be brought to scale on a system-wide basis?

We should encourage schools and school boards to take a more experimental approach to education. But our support for a more experimental approach should be contingent upon close study, the calculation of the return on investment, and the determination that, if successful, the approach can be brought to scale.

Technology & The Transformation of Education

Charles Ungerleider, Professor Emeritus of Education, The University of British Columbia

[permission granted to reproduce if authorship acknowledged]

In last week’s, E-Learning and Student Success, I expressed concern about Ontario’s plan to require students to take four of the 30 credits required for graduation on-line. I expressed hope that those responsible for planning and implementing the change would consider the capacities of adolescent learners. Notwithstanding those concerns, what students will learn and how they learn will both be influenced by changes in information and communication technologies.

New, and increasingly sophisticated, technologies will enable teachers to conduct pre-assessments of what students know and are able to do in a domain. That information will enable teachers to link instruction closely with students’ prior knowledge, allowing them to plan instruction better matched to their students’ current level of understanding.

Technologies will provide more focused opportunities for student practice and remediation. Teachers will be able to make provision for individual students who require additional practice in order to achieve mastery of a procedure (adding, subtracting, multiplying, etc.) and become fluent in the use of the procedure. Teachers will do so by employing applications that can customize the practice problems to the individual student’s characteristics and needs, monitor the student’s success, and adjust the difficulty of new practice problems. The increased use of this and other adaptive technologies will help to reduce the gaps between students in terms of their mastery of fundamental conceptual and procedural knowledge.

Adaptive technologies have the potential to reduce early school leaving due to lack of challenge. These technologies will be able to customize problems to enable students who have mastered fundamental concepts and procedures to apply them in new and more challenging contexts. The judicious and carefully constructed use of technology will provide students with opportunities to solve problems with the conceptual and procedural knowledge they have acquired.

Problem solving need not and should not be confined to extensions or challenges for concepts and procedures already mastered. There are many elements of science and social studies in which problem solving is an appropriate vehicle for learning. There are quite a few simulation games that have been developed over the years that prompt users to design, build and manage cities and nations that must confront a variety of issues over the course of their development. The ubiquity of smartphones, inexpensive laptops and tablets will enable more deliberate use of such programs in schools. 

A related development is the use of technology to give students the experience of being in an environment that they might not otherwise experience or to immerse students in an environment for the purpose of exploration.  While virtual experiences have existed for some time, the proliferation of inexpensive devices and the exponential increase in bandwidth will enable teachers to use them more extensively and deliberately.

The more deliberate and extensive use of new technologies will have a significant impact on teachers and teaching. There will be teachers responsible for the management of the learning of a group of learners much as a general practitioner physician manages the health of a patient.

There will be educational technologists who, working under the direction of teachers, will identify the various technologies that may be mobilized to support a student’s learning. There will also be subject specialist teachers who are responsible for diagnosing the learner’s needs in their areas of specialization and planning an instructional sequence to enable the students to progress from where they are to where they should be according to the established curriculum. 

General practitioner teachers will ensure coordination among specialists and technologists, monitor the implementation of the learning plans of students for whom they are responsible, communicate with and report progress to parents and guardians. They will collaborate with teacher technologists and specialist teachers to assess, plan, implement and evaluate the student’s progress. They will also be adept in marshalling the services and resources that students with special needs require to succeed and will be the primary point of contact for their parents and guardians.

Technology will figure prominently in the management of student learning and in communication with parents and among staff members. It will facilitate student self-assessment and increase their autonomy as learners, desirable changes that will help prepare them for a life of learning. But, as I said last week, these changes must be carefully planned and implemented. That means they must consider learner characteristics and capacities.