About Accountability : AYP Problems

W. James Popham is a leader in school measurment

September 2005

September 2005 | Volume 63 | Number 1
The Whole Child Pages 85-87

All About Accountability / AYP Wriggle Room Running Out
W. James Popham

Although I've never been a death-row inmate awaiting execution, I can imagine how such prisoners must feel as they watch their attorneys exhaust, one by one, all eligible appeals. Even though public school educators in the United States may not realize it, they are now facing a similar end-of-the-line scenario with respect to adequate yearly progress (AYP), the accountability cornerstone of the No Child Left Behind Act (NCLB).

If student scores on a state's NCLB tests aren't high enough to classify a sufficient number of those students as proficient or above, a school (or district) is designated as having failed to attain its AYP targets. Those AYP-failing schools that receive Title I NCLB funds are then placed on a sanction-laden improvement track that can soon “improve” a school all the way into nonexistence.

When President George W. Bush signed NCLB into law on January 8, 2002, critics of the law predicted calamitous consequences for U.S. public schools. They prophesied that most public schools would soon flop on AYP and, therefore, be regarded as “failing.”

Those dour predictions have not materialized. On the contrary. In many states, the number of AYP-failing schools has actually diminished during the three years that the law has been in existence. Such reductions in AYP-failing schools have prompted many NCLB proponents to gleefully contend that the law is “working the way it was supposed to work.”

But most of the early NCLB “successes” are illusory. Many public schools have been reaching their AYP targets primarily because state-level education officials have taken advantage of loopholes in the law so that their state's schools can appear to be successful. But just as is ultimately the case with death-row lawyers, these officials have now used up almost all available loopholes. Beginning this year, a noncushioned AYP sledgehammer will start pounding many schools. Let me explain why.

One of the most prominent ways in which states have shrouded low-performing schools' test performances is calculating a school's AYP using confidence intervals. A confidence interval is a plus-or-minus error band, such as those we see almost hourly before any important election. These error intervals tell us that a reported opinion poll is accurate only within a particular error range—plus or minus 3 percent, for example. Confidence intervals estimate the accuracy of sample-based data (for instance, the likely voting preferences of 2,000 voters interviewed by telephone) as a representation of the population itself (in this example, all registered voters). Of course, when a school's students take a state NCLB test, those students are not a sample. Rather, they constitute the complete population of students whose test performances will determine the school's AYP status that year. Providing an error range, then, makes no statistical sense because once you've measured an entire population, there's no need to employ sample-based estimates of that population's performance.

Despite outraged assertions from a number of qualified statisticians that applying confidence intervals in AYP calculations is flat-out wrong, federal officials have nonetheless allowed states to use confidence intervals—most likely to limit the number of schools that would otherwise take an AYP nosedive. But most states are already analyzing their schools' AYP data using confidence intervals and thus have used up whatever camouflage this statistical chicanery provides. Such states will now be obliged to identify many more AYP-failing schools.

A second AYP loophole stems from loose language in the original NCLB legislation. Even though the law calls for adequate yearly progress by schools and districts, it is technically possible to interpret the law in such a way that, for the initial few years, a school need not display any progress at all. Yet that school can still reach its AYP targets in each of those years. This seemingly contradictory situation arises because most states have adopted a cunningly staggered 12-year timeline that does serious violence to our understanding of what the “yearly” in adequate yearly progress actually means.

Most of these staggered AYP-increment timelines work in the following way: They are completely flat for three years (requiring no test score improvement at all), require higher scores for one year, then revert back to not requiring improvement for the next three years. This sort of stop-and-go timeline continues for a number of years until the law finally obliges a state to establish enormous, blatantly unattainable test score increases every year and to keep pushing until 100 percent of students supposedly reach this level of proficiency. Federal NCLB officials have also approved this tactic for monitoring schools' “yearly” progress.

When the 2004–2005 school year ended a few months ago, most states had used up their first batch of these timelines' no-progress-needed years. In many states, NCLB scores must now be appreciably higher if a school is to meet its AYP targets.

Have student scores improved on NCLB tests during the last three years? I suspect they have. But perhaps those scores have shown improvement simply because teachers and students have become increasingly familiar with the tests' content and format.

Some educators might take solace in secretary of education Margaret Spellings's much-heralded infusion of “flexibility” into NCLB compliance procedures. But don't let your hopes hop too high. Consider, for example, the increase in the number of students with disabilities whose test scores can now be counted as proficient in a school's AYP calculations. Recently, Wesley Bruce, assistant superintendent in the Indiana Department of Education, reported that applying these more flexible guidelines had, indeed, helped a number of additional Indiana schools reach their AYP targets. Unfortunately, that number was only two!

I fear that critics' pessimism regarding NCLB may not have been misplaced—but merely premature. With AYP wriggle room running out, U.S. public school educators might begin thinking seriously about what they will prefer for their last meal.


W. James Popham is Emeritus Professor in the UCLA Graduate School of Education and Information Studies; wpopham@ucla.edu.


Copyright © 2005 by Association for Supervision and Curriculum Development

© Copyright ASCD. All rights reserved.