Trials of Testing

Since many readers complained about the previous piece we have pulled it and provided this replacement. The things we do for you people!

It takes little courage to denounce men who believe they can harness the power of their minds to fly and use a space of universal consciousness to create world peace. And, in the long run, it is of little consequence. No one can recall the obscure psuedoscientific claims of yesteryear.

But take the idea that underneath the skull lie a series of organs for human traits like acquisitiveness and amorousness which bulge and change the shape of the head with dominance. The idea seems equally preposterous but it held real sway in its era — the Massachusetts Medical Association and the president of Harvard threw their weight behind it (Paul, 7) and phrenology continues to be remembered today.

Such absurd ideas are not remnants of a bygone past — just replace “organs” with “genes” and you’ll have the new “science” of evolutionary psychology (formerly sociobiology), an absurdity which Harvard University’s own president has thrown the institution’s weight behind. And yet one rarely sees “pro-science skeptics” challenging its claims. Indeed, scientific magazines write them up with only minor questioning, saving their ire for those who dare criticize the ideas.

But at least such fields have critics (and I count myself among them). There are related claims, however, that exercise much more power over our lives and (perhaps as a result) are far less challenged. One of their creators explained that they would “promote personal development”, “manage conflict”, and “increase human understanding worldwide.” (Paul, 121) But instead of Vedic science, she was talking about here creation: the Myers-Briggs personality test.

I have written before about the failures of experiments to provide evidence in favor of our concepts of personality or intelligence and how despite this many continue to believe in them. One can discuss how even studies by proponents find that IQ lacks validity and that 47% of people have a different Myers-Briggs personality type on a second administration of a test. But this somehow seems not to convince. So let us try another tack: let us look at how these tests are made.

The history of the IQ test — along with a number of other supposed ways of measuring “intelligence” — is detailed in Stephen Jay Gould’s classic The Mismeasure of Man. It was originally created by Alfred Binet to find children in French schools who might need special tutoring. Binet thought that by locating and helping these students, one could make sure that everyone learned all the material. Binet composed the test by throwing together whatever questions came to mind: things about shapes and numbers and words. He just wanted to see if some kids were having trouble, he made no attempt to make sure the result was a balanced measure of “intelligence”.

Lewis Terman, a professor at Stanford University, imported the Binet test to America, added some more random things and mixed it all up a little, and called the result the Stanford-Binet intelligence test (a name which is still used today).¹ One of the test’s first applications was American Psychological Association president Robert Yerkes’s attempt to classify the people recruited for the Army. Among the questions:

Crisco is a: patent medicine, disinfectant, toothpaste, food product
The number of a Kaffir’s legs is: 2, 4, 6, 8
Christy Mathewson is famous as a: writer, artist, baseball player, comedian

Recent immigrants, whose command of English might be understandably weak, were allowed to take a pictorial version: drawing “a rivet in a pocket knife, a filament in a light bulb, a horn on a phonograph, a net on a tennis court, and a ball in a bowler’s hand (marked wrong, Yerkes explained, if an examinee drew the ball in the alley, for you can tell from the bowler’s posture that he has not yet released the ball).” (Gould, 230)

Terman, meanwhile, conducted a longitudinal study of the people his IQ test marked as “gifted”. Joel Shurkin, based on exclusive access to the records, documented the full story in his book Terman’s Kids. Among the study’s participants was a man named Jess Oppenheimer. “Gave the impression of being very pushy and forward although he did not show these characteristics during the interview,” wrote one of Terman’s assistants. “I could detect no signs of a sense of a humor.” (Shurkin, 54) Oppenheimer went on to create and write the shows I Love Lucy and Get Smart.

The story of personality tests is little better. In her book The Cult of Personality (recently republished as The Cult of Personality Testing), Annie Murphy Paul (a former senior editor for mass bi-monthly Psychology Today) describes the history of all the major personality tests. Take the Minnesota Multiphasic Personality Inventory (MMPI), which was created in a similar way to the IQ test.

The test was created by psychologist Starke Hathaway and neuropsychiatrist J. Charnley McKinley by simply coming up with a bunch of true-or-false statements that they thought might indicate whether the respondent had a mental illness. Among them:

I have never had any black, tarry-looking bowel movements.
I have had no difficulty starting or holding my urine.
I have never indulged in any unusual sexual practices.
There is something wrong with my sex organs.
I believe there is a Devil and a Hell in the afterlife.
Everything is turning out as the Bible said it would.
I think I would like to belong to a motorcycle club.
Often I feel as if there were a tight band around my head.
I loved my father.
I like to flirt.
I believe my sins are unpardonable.
I have a good appetite.
I think Lincoln was greater than Washington.
Women should not be allowed to drink in cocktail bars.
A large number of people are guilty of bad sexual conduct.
If the money were right, I would like to work for a circus or carnival. (Paul, 53)

The resulting test was administered to the patients at the University of Minnesota mental hospital as well as the (presumably sane) staff there (all white, Protestant, Minnesotans who came to be known as the “Minnesota Normals”). Statistical analysis was then done to determine which questions more accurately predicted whether the user had a mental illness and more specifically, what kind.²

This was quickly generalized: people who scored above-average on the scales for Hysteria or Depression (but not high enough to actually have a mental illness) could be said to have hysterical or depressive personalities, even though there was absolutely no evidence to support this leap (not that it was on particularly sturdy ground to begin with).

The resulting test was used to analyze people in business, the army, court, high school, and at the doctor’s. It was “used to screen job applicants, offer vocational advice, settle custody disputes, and determine legal status.” (Paul, 58f) And while the test engendered some backlash, it continues to be used frequently today, often as the a requirement for getting or keeping a job. Paul notes “the MMPI (in an updated version) is employed by 86% of clinical psychologists and administered, by one estimate, to 15 million Americans each year.” (63) For example, it is used by 60% of police departments to evaluate prospective officers. Meanwhile, studies show that such tests can reject as high as 60% of healthy applicants.

This is but one example — and one chapter in Paul’s book — but all the others all have similar stories. An absurd test, concocted through absurd means, completely untested, ends up becoming a powerful societal force. All the more reason for us to speak out about them.

You should follow me on twitter here.

October 28, 2005