Thoughts on the World Bank’s ‘Learning Poverty’

The definition of Learning Poverty is: the share of children who cannot read a simple story by age 10 (~grade 5).

The point of Learning Poverty is to standardize the measurement of literacy of in-school children so that it is comparable across countries. It enables us to compare learning levels in the United States with learning levels in Pakistan, even though these countries use different tests and tools to measure learning. The World Bank’s geeks have re-hashed country-level data and converted it into an international standard of measurement. Like all international indices, the purpose is for countries to see where they stand relative to each other and then try to move up. It also allows for goal-setting.

Where does Pakistan stand? In low and middle income countries, 53 percent of third graders cannot read a simple story by age 10. For South Asia, the average is 58 percent. For Pakistan, it is 75 percent!

The green above indicates that 27.3 percent of primary aged children are out-of-school. The yellow is the percent of children who are in-school but illiterate.

What is the point of “schooling” if a child can’t read after spending five years there? The World Bank’s education chief, Jaime Saavedra, was right in calling this “morally unacceptable.” If this number doesn’t rob education officials of their sleep, then they shouldn’t be in charge anymore.

Yet, I’ve been uneasy with the index since I first heard of it, and that didn’t go away as I attended events around its launch in Pakistan or asked other Pakistani education researchers what they thought of it. I don’t think the index helps us make progress where it is badly, painfully needed.

An assumption underpins Learning Poverty: that countries will be moved to act when they see how poorly they are doing relative to other countries. Presenting data for international comparability rather than national relevance comes with costs though, all of which Pakistan has seen over decades as agenda-setting in the social sectors has been outsourced to donors.

Pakistan is drowning in data, even for learning and most of it has low reliability if you really dig into it, i.e. try lining up scores year-to-year or month-to-month and they jump wildly. We have: ASER, LND, Six-Monthly Test (not public), LEAPS (not all public), PEC (and its provincial counterparts), NEAS (and PEACE), and EGRA (the one I use) to name a few. Very few people — probably only imported and domestic education economists — find meaning in all these numbers and can separate the proverbial wheat from the chaff. It might have helped if this index had a name that was more intelligible than the rest.

“Pakistan is testing rich, but learning poor.” — Jaime Saavedra

A reliable data set that can be used to compare gains year-to-year NATIONALLY would be useful. Is this what Learning Poverty promises to be? Probably not, considering that Pakistan’s Learning Poverty score is based on testing that happened in 2014. And even if a new score were announced at the end of this year, it would reflect six years of effort, not one.

The only data set collected annually is ASER. Learning Poverty conflicts with it. ASER is a Pakistani initiative (linked to Pratham which conducts ASER in India), and has been measuring the ability of fifth graders to read a story since 2008. They were collecting this data for advocacy purposes before the international community realized that widespread in-school illiteracy was a problem. In 2018, ASER reported that 56 percent of fifth graders could read a story in Urdu, Sindhi, or Pashto.

Is the Learning Poverty figure of 75 percent more reliable than ASER’s 56 percent? Maybe, maybe not. My guess is that ASER’s data collection is weaker, but their sample is larger, and at least they included Sindhi and Pashto, so that their tests capture the mother tongue language of 37 percent of the population. Learning Poverty is derived from a national exam (NEAS) and reflects whether children can read Urdu, which is the mother tongue of 8 percent of Pakistan’s population. In Sindh, children are not necessarily taught to read and write Urdu at all.

In Pakistan, reading without comprehension is a norm — it wouldn’t be surprising if the country’s highest literacy rates were in Arabic since most parents will make sure that their children read the Quran. Arabic also has a strong phonetic method. Children in Pakistan who read the Quran at home do better on EGRA literacy tests for Urdu and Sindhi.

If Pakistan’s Learning Poverty score is based on testing children to read and write a language they don’t understand, then why do we care? What learning can be gained from this kind of literacy? Perhaps we have jumped to testing too quickly — it is the wrong tool to measure the wrong problem.

What I see over and over again in development is that we ask the wrong questions. We jump to creating solutions and measuring them, when we don’t fully understand why we have a problem in the first place.

Perhaps the data is just meant for advocacy. Even there, I don’t see how another, conflicting data set helps. The assumption must be that the actual numbers don’t matter, as long as they provide opportunities to create alarm. Pakistan has already seen a big experiment in data used to create alarm: Alif Ailaan. Alif Ailaan was uniquely successful at creating alarm in a nation that is pretty tone-deaf and functionally content with its failure in education, but they squandered that influence. Alif Ailaan shaped data to meet their advocacy purposes — opting for bigger numbers rather than the most reliable ones (as stated almost bluntly in “25 Million Broken Promises,”) clubbing different problems into one to make them look sizeable and simple (out-of-school 5 to 9 year olds with 10 to 16 year olds), and avoiding serious policy questions around Pakistan’s English and Urdu textbooks or the purpose and source of bigger budgets. If Alif Ailaan had not gone so far in simplifying the problem, based on an assumption that the public needed it to be simple (which I believe is wrong), then we might be closer to understanding and therefore solving the problems they were raising. (Alif Ailaan would be a good case study in the contradictions of a donor-supported project styled as a “civil society” movement — it closed in 2018, even the website is gone.)

I don’t think people are moved by data. There is simply too much of it. It becomes noise. It removes everything that is complex, intriguing, and therefore gripping about a problem. Journalists are not allowed to fill their articles with numbers because it loses the audience; perhaps economists and academics should learn something from that.

People are moved by leadership though — government leaders or experts they can index their views against— and witnesses via direct experience or videos. Seeing real children struggling with reading would give everyone a better sense of why kids are illiterate after years in school, and create consensus on language issues which are endlessly debated on hypothetical terms, more so than a 58 or 75 Learning Poverty score. Representing children as numbers just allows us to continue to pretend they are imaginary, hypothetical beings who never age (as they wait to be enrolled), speak English, need strong moral/ideological/civics/socio-emotional training rather than storybooks that are simply fun, or don’t get cold (and therefore drop out of cement boxes called schools, even while elite parents keep their kids home because the heating at school isn’t good enough).

I recently interviewed several ministers and secretaries about education reforms. They said quality was their priority. Quality has become a buzz word now thanks to donor-funded reforms. But quality or ‘learning crisis’ can mean a lot of things, such as Pakistani schools not producing scientists and fluent English speakers. In these conversations, officials and senior administrators couldn’t grasp or process the most obvious reality in schools: kids can’t read and they don’t understand the languages textbooks are written in (English and Urdu). This in part because, I sense, that the idea of reading has changed, especially as the education system has coped with illiteracy by getting kids really good at memorizing and replicating their textbooks on exams. Independent reading with comprehension isn’t necessary.

“I think somewhere societies, NGOs, seth [the powerful] are responsible for depriving millions of children and young people from being able to realize their true potential by denying them access to good learning education….

If you go to a madrassah you see children memorizing and then regurgitating for their examination. If you go to a government school, you will see them memorizing then regurgitating for their examination. If you go to a low fee private school, it’ll be the same. If you go to any private school, which is foreign, matric board, or federal board with state curriculum and state books, it will be the same.”

— Shehzad Roy, musician and education activist

Learning Poverty should have specified literacy, since this is what the index is actually about, and it should have underscored comprehension because too many good students in the world are good at reading foreign languages (English) that they don’t understand. These students end up plagiarizing at university and become employees that employers complain can’t even write a decent email.

As a researcher I want real, meaningful data; not data that is obscured for the purpose of international comparability. The underlying data set, NEAS, actually has a lot more information that is relevant and could have shaped more meaningful conclusions around Learning Poverty in Pakistan, such as that:

58 percent of teachers in Pakistan always use local languages for classroom instruction and 28 percent sometimes use it, leaving only 14 percent who never do.  84 percent of children speak regional languages at home rather than the national language that they were tested in

Someone pointed out at the LUMS roundtable on Learning Poverty, ‘We’ve measured the problem 100 different ways, and yet we keep coming up with new ways to measure it.’ At what point will we start understanding why we have a problem, and addressing it? When will we acknowledge the need to get beyond quantifying problems to actually understanding them? Are we limited by our tools, the availability of data, or our own fascination with data? I mention that last bit as someone who has loved the data, but was later shocked by how much I had missed when I started observing classrooms and watching children try to read.

“We need to make the case to teachers as to WHY this matters. Why should they get out of bed in the morning and care if their students can read or not? What does literacy do for children?” — Khadija Bakhtiar, founder of Teach for Pakistan

Even I was confused when Khadija said this. Isn’t the value of literacy obvious? (This goes back to the problem children existing as I, well-acquainted with data, imagine them, rather than me taking the time to understand their mundane realities that shape their choices.) Her statement made more sense to me when Hina Saleem at The Citizens Foundation (formerly Global Partnership for Education) told me about her field research in Badin, Sindh which has one of the highest rates of out-of-school children with half of children not in school. When she and fellow researchers asked people about the value of literacy in rural Badin, they couldn’t come up with much. Doctors wrote prescriptions in ways they could understand and there was little that was written in the physical environment. People, places, and educations systems have coped with illiteracy.

Finally, and most importantly, Learning Poverty is weak on telling us HOW to fix the problem. We have a target to reduce Learning Poverty by half by 2030, but what’s the road to get there? Learning Poverty provides pages of recommendations in fine print that run the gamut from governance to pedagogy to mother tongue. It’s more generic than compelling or specific to the national context. Honestly, what disturbs me most, is that I don’t think that we — ‘we’ the architects of the system, ‘we’ donors and consultants, ‘we’ experts and ESPECIALLY economists, ‘we’ local leaders, ‘we’ the elite — know how to make schools places where children can reliably learn how to read. We blame teachers, even while we hold them responsible for delivering an education system that ‘we’ designed and it is designed to fail.

“With our design, we fail good and bad teachers equally. If you sent a minister into classrooms with our textbooks, he would fail.” — Sami Khan, education policy specialist

We’ve measured the problem in a dozen conflicting ways. Instead of data, international experts and influencers must do better at understanding why the problem exists. More importantly, we need national consensus on how to solve it. Before we challenge countries to do better, perhaps we need to challenge ourselves to do better.

For more, see “Why Can’t Pakistani Children Read? The Inside Story on Education Reforms Gone Wrong,” published by the Wilson Center in July 2019.

The views expressed are the author's alone, and do not represent the views of the U.S. Government or the Wilson Center. Copyright 2020, Asia Program. All rights reserved.