Arne Duncan; does he understand the limits of data?

http://www.ed.gov/news/speeches/2009/07/07022009.html (at the end there are now 6 other replies that precede mine; most are quite good). Monty Neill

Duncan: But sometimes, despite our best efforts, these methods don't work. Today, America has about 5000 schools that continue to underperform year after year, despite our best efforts.

Comment: The number apparently represents those schools not making AYP for 5 years or so. Some of those schools may well be in very bad shape, others may have been making steady progress but started so far behind as to never make AYP. Some may be dysfunctional, others functional but needing more help for the school and in the communities from which the students come. How do we know the difference?
And how do we know they have received "our best efforts," especially since until a year ago the improvement fund for Title I was essentially non-existent? Whose best efforts, how are we to know they were the "best"? Were they strong efforts to do things Duncan listed immediately previous to this point (e.g., "we have tried boosting support for teaching staff and making other changes around curriculum, school day, etc.—and sometimes it has worked. I always favor more support, collaboration, mentoring and time on task")? Is it therefore necessary to take an extreme action, like privatize control of the school, or is it simply time to finally help schools get better on their own?
I am not opposed to extreme measures if a school really is dysfunctional after serious efforts at assistance. The Forum on Educational Accountability, which I chair, has stated that in the end states are responsible for their schools and students don't deserve perpetually non-functional school (see "Empowering Schools and Improving Learning," at http://www.edaccountability.org). But before generally untried nostrums - ones with only anecdotal evidence to support them, ones that seem to fail as much or more than succeed (e.g., charters, on average) - far more careful thought must go into what it takes to improve seriously troubled schools, what kinds of in-school and in-community supports are needed, etc.
Duncan goes on to cite the urgency of the situation. But doing something that has no evidence it will work in terms of improved student learning (more than test scores, however) and completion rates, and that simultaneously undermines democratic control over schools (which turning schools over to private operators very fundamentally does), is to respond via panic not thoughtful action -- if the agenda is systemic, sustained improvement.
Duncan: Now let's talk about data. I understand that word can make people nervous but I see data first and foremost as a barometer. It tells us what is happening. Used properly, it can help teachers better understand the needs of their students. Too often, teachers don't have good data to inform instruction and help raise student achievement.

Data can also help identify and support teachers who are struggling. And it can help evaluate them. The problem is that some states prohibit linking student achievement and teacher effectiveness.

I understand that tests are far from perfect and that it is unfair to reduce the complex, nuanced work of teaching to a simple multiple choice exam. Test scores alone should never drive evaluation, compensation or tenure decisions. That would never make sense. But to remove student achievement entirely from evaluation is illogical and indefensible.

It's time we all admit that just as our testing system is deeply flawed—so is our teacher evaluation system—and the losers are not just the children. When great teachers are unrecognized and unrewarded—when struggling teachers are unsupported—and when failing teachers are unaddressed—the teaching profession is damaged.

Comment: There are many problematic points here. While he describes the current testing system as "far from perfect" and "deeply flawed," he criticizes prohibitions on linking that data to teachers as a way to judge "performance." He seems to equate "data" with test scores, since he at no points suggests data is anything else. Indeed, "teachers need good data," but that must be far more than scores on the mediocre to lousy tests that now exist, from state exams to "benchmark" tests to travesties such as DIBELS.

So when he says, "But to remove student achievement entirely from evaluation is illogical and indefensible," he means that part of evaluating teachers should be their ability to raise test scores. As Alfie Kohn phrased it so sharply, this leads to "raising the scores and ruining the schools." And if some schools are already "ruined," focusing on test scores won't lead to a situation in which, to again cite Duncan, the nation can "give children the very best education possible." Schools that focus on boosting test scores do no such thing. (FairTest regularly summarizes this evidence in our quarterly Examiner newsletter and various reports; www.fairtest.org.)

Duncan then moves on to payment by results - "When great teachers are unrecognized and unrewarded." He has made clear on other occasions he very much supports payment by results, though he regularly cautions he wants to do this with teachers. But what if the teachers refuse? Meanwhile, Duncan has echoed Eli Broad in claiming that payment for performance is common in other fields. As the recent EPI book by Rothstein makes clear, it is not common among professions, and where implemented it brings about goal distortion, gaming the system, and bad consequences. George Madaus has looked at paying teachers for results in Ireland, others have looked at England, and the situation is the same: it does not work.

Again, we are in a situation in which Duncan insists that due to the dire situation, we must "do something." But the something, in this case, not only has no evidence it will not work, it has clear evidence it will not work.

Two points are here inter-twined: payment for results, and using test scores to define the results. The first has not worked in other fields, the second compounds the damage and will further intensify the score inflation now seen across the states (and that will plague a national test as well).

Teachers and their unions should flat out refuse payment for results. They will of course be attacked as protecting themselves at the expense of their students -- but the truth is, they are also protecting the children from the systemic malfeasance of allowing standardized tests to control curriculum and instruction.

Duncan is correct, as many people have pointed out, that evaluation of teachers is largely a farce - for many reasons. It should be greatly improved, first of all in order to help teachers get better. It must be tied to very different forms of professional development than the trivial time-wasters that have given PD a bad name among teachers. And assessment must be overhauled, not to have "better" ways to institute payment for results, but to have good information about student learning that students, teachers, administrators and other professionals, parents, communities and states can use to improve real learning (guide, not drive, action, as Deborah Meier puts it). FEA has much to say about most of these points in our various reports such as Redefining Accountability and the report of the Expert Panel on Assessment, at http://www.edaccountability.org (not teacher or principal evaluation, however; but see the National Staff Development Council materials).

So there is much to do, and the stimulus funds as well as an overhauled ESEA can be used to improve assessment in line with FEA and FairTest recommendations, to overhaul professional development in line with FEA recommendations, to make data mean something far more than test scores, to focus on improving school capacity to serve all children well, to educate the whole child, and to pay attention to the consequences of racism and poverty that plague so many communities and their schools. Payment for "performance" is ultimately a distraction, as are national standards and a national test, from the real work of improving schools. It is time to re-focus and move in more useful ways.

--
Monty Neill, Ed.D
Deputy Director
FairTest
15 Court Sq, Ste 820
Boston, MA 02108
monty@fairtest.org
857-350-8207; fax 850-357-8209
www.fairtest.org