Tuskegee and the Obama Effect

By now you've all heard the fascinating news of a study (New York Times: "Study Sees An Obama Effect As Lifting Black Test-Takers") that seems to demonstrate an Obama effect on the black/white gap in test scores. In short, a team led by a Vanderbilt University researcher administered a series of 20 questions (drawn from verbal section of the GRE) shortly before Obama's nomination and again after his acceptance and then again after the election. Black performance on the test improved after Obama's acceptance, and rendered the black/white gap in test performance nonsignificant.

I'm the first to admit the potential for an Obama effect. Every time I hear him speak I think of the power of a role model, and dream of possible studies that could uncover such an effect.

But in this case, I'm not so sure what's being captured is an effect of Obama on the confidence of black students in their academic performance. Here's why:

(1) The students taking the test at each administration were different students. If the same kids took the test repeatedly, obviously we'd expect their scores to increase.

(2) According to the lead researcher, in a personal communique with me, while the pool of potential participants was constructed at time 1, the actual sample at each time was based on volunteers offered a monetary incentive to participate (what size incentive? I don't know).

There are more critical pieces of information missing as well:

(a) Whether the reasons for participation vs. non-participation differed by race, and are correlated with test-taking ability.

and

(b) Whether the rates of participation were similar for both racial groups.

What we do know is that ever since the Tuskegee Study of Untreated Syphilis (TSUS), African Americans are less likely than Whites to volunteer for participation in research. Given the known gaps in achievement, if they knew anything about what the study required they may've also simply lacked the confidence to participate. This is completely understandable. The question is, could it influence the findings in this study? Are there other plausible explanations for the change in test scores observed in the study?

Yes. Let me suggest just a few.

(1) A disproportionate effect of the economy on black's financial status. The study took place during a year of steady decline in the economic standing of many Americans. Is it possible that the money offered for participation wasn't enough to offset the concerns of higher-achieving black students about research (or to offset the opportunity costs associated with participation)? But that by time 2, the money was simply worth more (e.g. more effective as an incentive) and induced greater participation of black students? I'm positing that during the period whites were both less affected by changes in the economy and overall less averse to volunteering to take a test.

(2) An effect of Obama on black's trust in society, including researchers. So at time 1 the black students in the pool are generally more suspicious and only the lower-achievers are affected by the monetary incentive enough to overcome that suspicion and take the test. At time 2, they're feeling more goodwill towards the world, and higher-achieving black students are willing to participate.

(3) Maybe higher-achieving black students, when asked twice to do a study, tend to do it? I don't know if nonrespondents at time 1 were asked again.

These are just three ideas about how sample selection could bias these results. I have many more. What about the gender composition of the samples? ( Black men have lower test scores on average and are generally less likely to participate in studies. )

I want to quantify the good feelings we're all having in the post-Bushie world too. I get the motivation. But I don't think we should get too carried with feel-good stories on studies that have not yet undergone peer review.