3 August 2008
Guest Blog: On Biases
Posted by rebecca
Hello, my name is Rebecca Harbison, and I’m a grad student in astronomy at Cornell University, and guest-blogger.
Some information about me. I work on Saturn’s rings using VIMS, the Visible-Infrared Mapping Spectrometer. Right now, I’m trying to measure how small the smallest particles are in Saturn’s Main Rings, by looking at how they diffract sunlight. (Think of me as sitting in a theater, staring at the light from the projector and trying to guess at how much dust is in the air by how much I can see in the beam.) In the future, I hope to expand my rings work to model composition and surface properties.
I’ve done other stuff over the years as well. My first-year project (which I’m still chugging away at) was a study of the rotation of Hyperion, which is one of Saturn’s moons. It moves in an interesting way — instead of spinning neatly, its odd shape and tides from Saturn make it tumble. Having a model of this not only lets us know what Hyperion is like inside (probably: a mix of rock, ice and empty space, with better packing towards the center), but also helps the people interested in its surface know what they are looking at if they can’t get the high-res pictures. Also, before I came to Cornell, I looked at active galaxies and helped measure how they varied (in hopes of learning about how the black holes inside them worked).
I’ll probably be trying to cover some non-Mars things from my corner of the (outer) Solar System.
* * *
So, for my first post, I’d thought I’d talk about something a little more down-to-earth. Mostly because I had this already written when Ryan asked me. It was based on this article, which ran on CNN last week — for those of you with subscriptions to Science magazine and willing to read through the statistics, the study is here. Expect a Cassini-themed post by Thursday. I’m still getting the hang of this blogging thing.
The article notes that, historically, women have done worse on the SAT and ACT than men
So, let’s try to figure out why:
1. Women are less capable than men at math. You can subdivide this into nature versus nurture (versus both) — that women don’t do well in math because they aren’t taught right, versus some inborn inability to do math.
2. The SAT and ACT has some bias that makes women score lower than men. Remember, standardized tests weren’t engraved onto stone tablets and passed onto teachers by some higher power — they were designed by people. I had my first encounter with this when my fourth grade teacher explained she couldn’t skip reading aloud the instructions, despite the fact we all knew them by heart after two days of testing — that’s the ‘standardized’ in ‘standardized tests’, and the tests-writers’ goal is to have as little bias as possible.
So, let’s investigate #2. The SAT and ACT is disproportionately taken by high-school students attending a four-year college. Now, you say, that must mean that the SAT and ACT is preferentially sampling the academically-inclined students, so that shouldn’t affect the results — including the students who are going into a trade or a two-year school would lower both scores.
However, currently women make up a majority of students entering four-year colleges. In other words, more women take the SAT then men since more women attend a college that requires SAT scores for admission (or ACT). This suggests an alternate hypothesis: that women are scoring lower on average because they have a larger range of students. Like, if you take an average (either a mean or a median here) of 1 to 5 for the men, and 1 to 7 for the women, the women’s average will be smaller because the extra students are added on to the low end of the distribution.
So, let’s look at that bias. So, our hypothesis is:
Only including college-bound students in the test biases it to make women appear to be less capable at math, because more less-capable-at-math women want to go to a four-year college than less-capable-at-math men.
Let’s test this. Thanks to the No Child Left Behind Act (which is finally doing something useful, hooray!), we have math tests taken by all high-school students in public schools. (The fact the NCLB Act doesn’t sample private schools or homeschooled students might be a problem.) Two states also mandated that all students take the ACT, regardless of their intentions after high school. And, what do you know — the results show that there is no gender bias in tests given to a sample of high-schoolers who aren’t selected by future plans. Women do just as well in math tests as men, if you include the men and women not heading for a four-year college. That certainly supports our hypothesis. It also doesn’t support our hypothesis that there is some inherent difference in women (or their education) — if so, we’d expect there to still be a gap.
(For that matter, the article also notes that higher-level (problem solving versus recall and simple calculations) isn’t tested that much in normal tests, and that should be measured as well. So knowing what you aren’t testing is also important.)
In addition to being wonderful news for science and math, it also shows the importance of making sure you understand the biases of your studies. Here, I’m going to do a tie in back to planetary science. Because we live on Earth, we tend to expect things to behave a certain way. Sometimes this is good — we see something that looks familiar, and can tell what is interesting and what isn’t. Sometimes this gets us into trouble — our intuition isn’t always right, and changing conditions (or just different time scales, or temperatures, or gravities) can make something appear very odd.
The important thing is to understand what our biases are and to investigate them. Because, sometimes you discover something really awesome, and it’s nice knowing someone won’t come around and reveal the man behind the curtain.
Your analysis seems to be in accordance with the data you present, but I think the data is incomplete.
You mention that higher reasoning and higher mathematics are not generally tested. Because NCLT tests to an established standard (that is one implemented in all schools, to their detriment quite often), the data for high school students show that when taught, the students learn or test equally well.
I would suggest that it may be that male students more commonly seek (or be encouraged to seek) advanced studies in math and the physical sciences in high school, and or in college. The higher scores on average for men taking the SAT may indicate two related trends: that women more commonly pursue liberal arts degrees or degrees in the soft sciences, and that men more commonly seek out degrees in hard sciences and engineering. The disparity in testing should be accounted for by the prevalence of male students interested in math and science on the SAT, and not necessarily by the “lower-end” female students who are taking the test.
Also, do the data show any disparity in the SAT language scores, the AP language scores, or the SAT II writing scores? What about the AP Math scores- a pool of data which includes more self-selected participants. If your conclusion is correct, then AP math scores aught to be even between genders, as students receive the same instruction for these classes.
As to the grading process for the AP Language and Literature exams, my knowledge of the exams leads me to predict that grading is strongly gender biased. I briefly held a TA-ship as a transcriber for an AP prep course. (I transcribed photocopies of sample tests) The tests are written by hand and read in the original by several graders. The graders also “calibrate” their scoring by comparing their scores (which correspond to a rubric) on a large number of sample tests, which are introduced throughout the grading process. Despite this attempt at standardization, the teacher I worked with informed me that female students “just write better,” and further that any exam reader would become highly aware of the gender of the test-takes through their penmanship and style. It seemed to me at the time that typed papers should be a necessity in standardized writing tests- although they are logistically problematic. Still, elements of style cannot be accounted for, if they are highly indicative of gender.
Do you tend to believe that, given all else being equal, interest in the sciences and math aught to be equal between genders? I am not terribly convinced that this would be the case. While inequity in teaching should be eliminated where it is harmful, I am not convinced that students of different genders share the same academic needs, or that they should be taught, as they are under NCLB, to the same standard. The focus on the non-gender-biased standard would seem to generate a certain level of gender bias in its own right- the idea that our expectations of both genders should be demonstrably equal across the board. As has been pointed out recently in our own school newspaper, universities full of only women are possibly as undesirable for society as universities only full of men. This will never not be a thorny subject, but I’d be interested to know how you would go about changing the landscape of universities in this country, or if you would be interested in doing so.