Thursday, November 30, 2006

AEI Conference on Black-White IQ Gap

Video, Audio, slides and relevant papers on the November 28th 2006 AEI conference on The Black-White IQ Gap: Is It Closing? Will It Ever Go Away? are available here.
For decades, the difference in the test scores of blacks and whites on the SAT, National Assessment of Educational Progress test, Armed Forces Qualification Test, and traditional IQ tests has been a vexed issue for American educational policy. Two of the leading scholars of this controversial topic, James R. Flynn of the University of Otago (New Zealand) and Charles Murray of AEI, will debate the causes of the difference, its implications, and recent trends. New studies of the subject by Professor Flynn and by Mr. Murray will be available for distribution at the session.

Rarely have I seen such a contentious issue discussed so civilly and scientifically. The conference left me with a lot of new information to think about.

Monday, November 27, 2006

Standards So Low a Caveman Could Meet Them

If 100 cavemen wanted to become high school mathematics teachers, how many could pass the licensure test? The answer appears below.

Teachers in core subject areas are required by the No Child Left Behind act to prove they know the subject they are supposed to teach. NCLB gives broad guidelines as to what constitutes proof, but the details are left to the states. Most states require their new teachers to take a licensure test in the content area they plan to teach. Score above the state-defined cut-score on the appropriate licensure test and you have met your burden of proof.

How high to set these cut-scores is subject to debate. What is not debatable is that examinees with zero relevant content knowledge should not be able to pass. No matter how good your teaching skills — “You can’t teach what you don’t know, anymore than you can come back from where you ain’t been.” [Will Rogers]

For secondary mathematics teachers, the Praxis II (10061) test is currently used by a majority of states for the purpose of proving sufficient mathematics content knowledge. The cut-scores vary widely.

I showed in a previous post that Colorado’s requirement was approximately equivalent to an examinee knowing 63% of the content on this high school level mathematics exam, whereas Arkansas’ standard is approximately equivalent to knowing just 20% of the content. Such extreme variation is already an indication that something is very wrong with how these state standards are set.

I say “approximately equivalent” because this equivalency assumes that the examinee takes the test only one time and has just average luck guessing on those questions he doesn’t know how to solve. However, in the real world, examinees who miss their state’s cut-off score can take the test an unlimited number of times. They are also encouraged to guess by a test format that does not penalize for incorrect answers. This situation makes it possible for examinees of much lower ability to (eventually) pass.

We can calculate the probability that an examinee with a certain true ability level will pass in one or more attempts. The examinee’s true ability level gives the percentage of questions they know how to solve. This is the score they would get on a constructed response exam, that is an exam with no answer choices. On an exam with four answer choices per problem, like the Praxis II, an examinee will correctly answer this percentage of questions plus, with just average luck at guessing, a fourth of the remaining questions. However, some examinees will have above average luck as seen in the table below.

Probability of Passing the Praxis II in Arkansas
True Ability LevelProbability of Passing
in One Attempt
Probability of Passing
in Ten Attempts
 0%  1.4%  13%
 4%  3.7%  32%
 8%  9.0%  61%
12% 19.0%  89%
16% 35.1%  99%
20% 56.0%≈100%
24% 76.9%≈100%
40%100.0% 100%
Table 1. Probability of passing the mathematics licensure test in Arkansas for various true ability levels.
An examinee with a true ability level of 20% has a better than even chance of passing on the first attempt and is all but certain to pass in a few attempts. In this sense, the Arkansas standard is approximately equivalent to knowing 20% of the material (red row). This is an extraordinarily low standard given the content of this exam. (It is sometimes misreported as 40% because this standard requires correctly answering about 20 of the 50 questions. However an examinee that knew how to solve just 10 problems would average another 10 correct by guessing on the remaining 40. He answered 40% correctly, but only knew how to solve 20%).

However, with a some luck, examinees with absolutely no relevant content knowledge can pass (blue row). If 100 cavemen were to take this exam, up to ten times each, about 13 would pass. We are not talking about the brutish-looking, but culturally sophisticated cavemen of the Geico commercial. We are talking about cavemen whose relevant content knowledge is limited to the ability to fill in exactly one circle per test question.

Now presumably such zero-content-knowledge examinees would never have graduated college. Yet the fact that the standards are set this low says that some people of very low ability must be managing to satisfy all the additional requirements and enter teaching.

Such extraordinarily low standards make a joke of NCLB’s highly qualified teacher requirements. They also make a joke out of teaching as a profession and are a slap in the face to all those teachers who could meet much higher standards.

Only teaching shows this enormous variation and objectively low standards. (Even Colorado’s 63% would still be an ‘F’ or at best a ‘D’ if the Praxis II were graded like a final exam.) In contrast, architects, certified public accountants, professional engineers, land surveyors, physical therapists, and registered nurses are held to the same relatively high passing standards regardless of what state they are in.

How is it that these other professions can set arguably objective standards, while teachers cannot? The standards in other professions are set by professional societies. Their decisions are moderated by several concerns including the possibility of having members sued for malpractice.

For teachers, the standards are first set by a group of professionals, but their recommendations can be overridden by state educrats. The educrats are concerned largely with having an adequate supply of teachers. The entire process lacks any transparency, so we cannot tell the extent to which the educrats substituted their concerns for the professionals’ judgment about standards for teaching.

In shortage areas, like mathematics, low standards guarantee adequate supply. It’s a lot less trouble for the educrats to simply lower standards than to pro-actively change incentives so that more academically able people might consider teaching math.

Something like NCLB’s requirement that teachers prove they have sufficient subject matter content knowledge is clearly needed to prevent cavemen from teaching our kids math, but the Feds trust in the states to set these standards is not justified. Under NCLB, the perfect scorer and the lucky caveman are both indistinguishably “highly qualified.” Setting higher standards would force states to begin to face the elephant in the room: Not enough is being done to attract mathematically talented people into teaching.

Monday, November 13, 2006

Reflection on the President’s Proposals

The president has proposed two new programs. One would train 70,000 high-school teachers to lead Advanced Placement courses in science and math. A second would bring 30,000 math and science professionals to teach in classrooms and give early help to struggling students.
... there was a specific concern about math and science scores. The President will build on the success of No Child Left Behind and propose -- to train 70,000 high school teachers to lead advance placement courses in math and science. We'll bring 30,000 math and science professionals to teach in classrooms and give early help to students who struggle in math so they have a better chance at good high-wage paying jobs. [Whitehouse Press Briefing]
The proposals themselves are a tacit admission that there continues to be something wrong with math and science education despite the fact that the vast majority of math and science teachers are “highly qualified”. Calculus is part of the high school curriculum. A “highly qualified” mathematics teacher should be able to teach calculus without needing additional content training, yet that is where the training in this proposal seems to be targeted
... provide incentives for current math, science and critical language teachers to complete content-focused preparation courses; [Expanding the Advanced Placement Incentive Program]
The second part of the proposal — putting math and science professionals in a classroom to help struggling students — presupposes that these professionals know how to help struggling students. Why should they? This is far more about pedagogy than it is about content knowledge. If their current teachers do not have the content knowledge to help them, why are they in the classroom?

Here’s a thought — wouldn’t it be better to have the professionals, who presumably understand the content, teach the advanced placement courses and let the teachers, who presumably know about helping struggling students, help the struggling students.

I recently had a friend resign his teaching position at a local high school. Until this September he was a scientist (physics Ph.D.) who worked in a research lab. He went alternate route, passed his Praxis II tests by wide margins, and even had the benefit of some preservice training.

As in many American high schools, seniority plays a significant role in how teaching assignments are made. So rather than being assigned to teach high level courses, where his superior content knowledge would be a big plus, he was assigned to teach fairly low level courses, where his lack of teacher training began to show. His students expected to entertained more than taught. They did not expect to have to think. They began to rebel and it was all downhill from there.

This school lost a potentially great teacher who was mis-assigned. Let’s hope the president doesn’t make the same mistake.