History Lesson

A recurring theme in my posts will be the importance of understanding data before drawing conclusions from it. Seems obvious, but putting it into practice requires a lot of work. This is especially true in education, where spin is usually far more important than careful analysis. What Ken DeRosa disparagingly calls Stinky Research.

The most flagrant example of this was the the scandal in student testing that occurred in 1980's. There is a clear analogy between this student testing scandal and the teacher testing issues this blog is addressing, so it is worth reviewing the history. The plot was uncovered by a medical doctor, John Jacob Cannell, who began to question the spin:

My education about the corruption of American public school achievement testing was a gradual process. It started in my medical office in a tiny town in the coal fields of Southern West Virginia, led to school rooms in the county and then the state, to the offices of testing directors and school administrators around the country, to the boardrooms of commercial test publishers, to the office of the U.S. Secretary of Education, to schools of education at major American universities, to various governors’ offices, and finally, to two American presidents.

One day in 1985, West Virginia newspapers announced all fifty-five West Virginia counties had tested above the national average. Amid the mutual congratulations, I asked myself two things. How could all the counties in West Virginia, perhaps the poorest and most illiterate state in the union, be above the national average? Moreover, if West Virginia was above average, what state was below?

In my Flat Top, West Virginia, clinic, illiterate adolescent patients with unbelievably high standardized achievement test scores told me their teachers drilled them on test questions in advance of the test. How did the teachers know what questions would be on a standardized test?

Then I learned that West Virginia schools, like most other states, used what seemed to me as a physician to be very unusual standardized tests. Unlike the standardized tests that I knew - such as college entrance, medical school admission, or medical licensure examinations - public school achievement exams used the same exact questions year after year and then compared those scores to an old, and dubious, norm group - not to a national average. Furthermore, educators - the group really being tested - had physical control of the tests and the teachers administered them without any meaningful test security.

Please read the whole thing.

States still administer their own tests. But cheating has been made more difficult because students also take the national NAEP tests. For example, we have this New York Times report:

Students Ace State Tests, but Earn D's From U.S.

A comparison of state test results against the latest National Assessment of Educational Progress, a federal test mandated by the No Child Left Behind law, shows that wide discrepancies between the state and federal findings were commonplace. ...

States set the stringency of their own tests as well as the number of questions students must answer correctly to be labeled proficient. And because states that fail to raise scores over time face serious sanctions, there is little incentive to make the exams difficult, some educators say.

One of the big political compromises in NCLB was the extent to which states retained almost complete control over the quanitative aspects of testing. They define the tests and the mapping of test scores into broad categories. Even now, with the NAEP results as oversight, states still skew their student testing for political advantage. See, for example, this related story Exploiting NCLB With Social Promotion.

In teacher testing there is virtually no oversight. States choose the pass scores that define what “highly qualified” means. Something akin to the NAEP is needed for teacher testing. It could be as simple as using the Praxis II and defining additional categories as was done in Table 2 of my The Highly Qualified Math Teacher post. Note that this doesn’t prevent any state from hiring anyone they want. It just prevents them from labeling the teacher “highly qualified” unless there’s some proof. Without some kind of objective standards, putting a “highly qualified” teacher in every classroom has little meaning.