3 August 2008

Guest Blog: On Biases

Posted by rebecca

Hello, my name is Rebecca Harbison, and I’m a grad student in astronomy at Cornell University, and guest-blogger.

Some information about me. I work on Saturn’s rings using VIMS, the Visible-Infrared Mapping Spectrometer. Right now, I’m trying to measure how small the smallest particles are in Saturn’s Main Rings, by looking at how they diffract sunlight. (Think of me as sitting in a theater, staring at the light from the projector and trying to guess at how much dust is in the air by how much I can see in the beam.) In the future, I hope to expand my rings work to model composition and surface properties.

I’ve done other stuff over the years as well. My first-year project (which I’m still chugging away at) was a study of the rotation of Hyperion, which is one of Saturn’s moons. It moves in an interesting way — instead of spinning neatly, its odd shape and tides from Saturn make it tumble. Having a model of this not only lets us know what Hyperion is like inside (probably: a mix of rock, ice and empty space, with better packing towards the center), but also helps the people interested in its surface know what they are looking at if they can’t get the high-res pictures. Also, before I came to Cornell, I looked at active galaxies and helped measure how they varied (in hopes of learning about how the black holes inside them worked).

I’ll probably be trying to cover some non-Mars things from my corner of the (outer) Solar System.

* * *

So, for my first post, I’d thought I’d talk about something a little more down-to-earth. Mostly because I had this already written when Ryan asked me. It was based on this article, which ran on CNN last week — for those of you with subscriptions to Science magazine and willing to read through the statistics, the study is here. Expect a Cassini-themed post by Thursday. I’m still getting the hang of this blogging thing.

The article notes that, historically, women have done worse on the SAT and ACT than men

So, let’s try to figure out why:
1. Women are less capable than men at math. You can subdivide this into nature versus nurture (versus both) — that women don’t do well in math because they aren’t taught right, versus some inborn inability to do math.
2. The SAT and ACT has some bias that makes women score lower than men. Remember, standardized tests weren’t engraved onto stone tablets and passed onto teachers by some higher power — they were designed by people. I had my first encounter with this when my fourth grade teacher explained she couldn’t skip reading aloud the instructions, despite the fact we all knew them by heart after two days of testing — that’s the ‘standardized’ in ‘standardized tests’, and the tests-writers’ goal is to have as little bias as possible.

So, let’s investigate #2. The SAT and ACT is disproportionately taken by high-school students attending a four-year college. Now, you say, that must mean that the SAT and ACT is preferentially sampling the academically-inclined students, so that shouldn’t affect the results — including the students who are going into a trade or a two-year school would lower both scores.

However, currently women make up a majority of students entering four-year colleges. In other words, more women take the SAT then men since more women attend a college that requires SAT scores for admission (or ACT). This suggests an alternate hypothesis: that women are scoring lower on average because they have a larger range of students. Like, if you take an average (either a mean or a median here) of 1 to 5 for the men, and 1 to 7 for the women, the women’s average will be smaller because the extra students are added on to the low end of the distribution.

So, let’s look at that bias. So, our hypothesis is:

Only including college-bound students in the test biases it to make women appear to be less capable at math, because more less-capable-at-math women want to go to a four-year college than less-capable-at-math men.

Let’s test this. Thanks to the No Child Left Behind Act (which is finally doing something useful, hooray!), we have math tests taken by all high-school students in public schools. (The fact the NCLB Act doesn’t sample private schools or homeschooled students might be a problem.) Two states also mandated that all students take the ACT, regardless of their intentions after high school. And, what do you know — the results show that there is no gender bias in tests given to a sample of high-schoolers who aren’t selected by future plans. Women do just as well in math tests as men, if you include the men and women not heading for a four-year college. That certainly supports our hypothesis. It also doesn’t support our hypothesis that there is some inherent difference in women (or their education) — if so, we’d expect there to still be a gap.

(For that matter, the article also notes that higher-level (problem solving versus recall and simple calculations) isn’t tested that much in normal tests, and that should be measured as well. So knowing what you aren’t testing is also important.)

In addition to being wonderful news for science and math, it also shows the importance of making sure you understand the biases of your studies. Here, I’m going to do a tie in back to planetary science. Because we live on Earth, we tend to expect things to behave a certain way. Sometimes this is good — we see something that looks familiar, and can tell what is interesting and what isn’t. Sometimes this gets us into trouble — our intuition isn’t always right, and changing conditions (or just different time scales, or temperatures, or gravities) can make something appear very odd.

The important thing is to understand what our biases are and to investigate them. Because, sometimes you discover something really awesome, and it’s nice knowing someone won’t come around and reveal the man behind the curtain.