The price of a GRFP, part 2

(This post is a continuation of the price of a GRFP, part 1. If you’re not caught up, you can go back and start from the beginning.)

In this next installment about the GRFP I’ll be aiming to answer the second question in my first piece, about the change in NSF demography from before the Dear Colleague Letter announcing the ineligibility of graduate students to apply twice.

How has this policy change impacted diversity?

One simple way to look at this is to start with the applicants being awarded who come from prestigious undergraduate institutions. As we know from the last post, the # of awards to these institutions is already fairly lopsided. Check out this cumulative distribution function of sorts, with the % of total awards on the Y axis, given to N schools on the X axis:

cdf.png

The top 3 schools recieve 10% of awards. The top 10 get about 25% of them. The top 30 get about 50% of all 2000 GRFPs. Once you’re out in the top 100 you’ve given away nearly 3/4 of your awards, with about 600 schools yearly splitting the last 25% amongst themselves.

But how has the share of awards from year to year to those top schools changed?

1. Top 10 schools recieve over 50 more awards than last year.

shorter_top_10_comparison_before_after.png

Here schools are ranked not by selectivity but by the top NSF-awarded schools. (Of course you can tell these two things go hand in hand.) There’s been a big uptick in awards in the first year this policy came into play, and it’s not precedented by change in the previous year.

2. In fact, Top 10 schools recieve quite a few more awards than they have for a while.

Since we have data going back quite a bit, you can extend the plot further back in time:

num_to_top_10_winning_schools.png

Beginning in 2011, there’s actually a persistent decrease in the number of awards going to top 10 schools. This levels off in 2013 around where it remains – until the spike in 2018. Also note that the change in this year is about 2 times larger than any change in previous years (and those were all decreases).

But the claim was that the number of undergraduate applicants would increase, and that they’re more diverse.

These two figures only address undergraduate schools — not the applicants who are still undergraduates. We don’t directly have undergraduate status, but can infer it imperfectly by looking for applicants who (a) list their undergraduate institution as their current school, or (b) list no current school at all. Anyone in the third category, (c) list a different current school than their undergraduate institution, is most likely a graduate student; we assume others are most likely to be undergrads.

This method is imperfect, but we can use it to understand at least some of the changes in undergraduate awardees over this same time.

3. The same # of undergraduates win the GRFP, but more of them come from top 30 schools.

awards_to_undergrads.png

Here, we see that there’s been a decrease over time in the overall number of likely undergraduate awardees — consistent with a trend that might concern the NSF, and prompt them to put in place a policy like this! But interestingly, the number of undergraduate awardees increases in 2016, not 2017 — and the only increase from before the policy (2017) to after (2018) is in the number of those awards going to undergrads from those top schools.

At least this policy stops people from reapplying a million times, right?

I was interested in understanding the landscape of NSF reapplicants for obvious reasons — because this is the applicant pool most significantly decreased by the policy. Now that graduate students can’t apply twice, who does that really curtail from winning the award?

I identified probable reapplicants in the dataset by comparing the NSF awardees 2012-2018 with the honorable mentions from 2011-2017. I then split these up by undergraduate school of origin.

4. Reapplicants from the least selective schools are most harmed.

reapplicant_change

There’s a precipitous drop of about 60% in reapplicants winning from 2017 to 2018 — indicating that indeed they’re dropping out of the pool! But when you compare reapplicants from top and middling school backgrounds on the left, their 60% award drop pales in comparison to the 80% drop for those from the least selective schools.

There are a few more analyses I want to do, but for now, these are the main takeaways: the new policy is definitely impactful. But it seems to have result in a precipitous increase in awards to top schools. Does that fulfill its mission of expanding diversity? Well, maybe, if the most diverse students at those top schools are getting those awards.

But even for fully funded students, the NSF isn’t the difference between the dream of science and the reality of it. How many times have people called it a medal, or a feather in a cap? Should its mission be to decorate people who have funding or to change the lives of people who don’t?

Last but not least, if you’re curious about any of these things, I’ve put together a shiny app that lets you play with your stats– and see how many GRFP slots there are for you.

What are your odds?

If you’re interested in adding any other sliders, shoot me a comment or tweet at me – I’ve got a lot of data ūüôā

The price of a GRFP, part 1

I had some downtime a while back (literally; the cluster I work on was down) and so I cracked open some analysis I’ve been doing on the side for a while. I like to switch off analyses and work on some side projects to keep me working, but not burnt out, and so I picked up a dataset I’ve been working on for a little while: the NSF GRFP awardees.¬†

A dear colleague letter

About 2 years ago, the NSF made a policy change announcement, summarized here (capitalization & other emphasis mine):

NSF will limit graduate students to only one application to the GRFP, submitted either in the first year OR in the second year of graduate school. …¬†GRFP continues to identify and to inspire the diverse scientists and engineers of the future, and especially encourages women, members of underrepresented minority groups, persons with disabilities, and veterans to apply…. This is a more diverse population than admitted graduate students.

Lots of great commentaries have touched on the diversity challenges of the GRFP and the potential effects of the policy change proposed. Others have even looked into some of the representation details themselves in their commentaries.

But I want to ask two questions in a data-driven way this year, the first year that the rule is fully in place**:

Is the NSF fulfilling its mission to inspire “diverse scientists of the future”?

and

How has this policy change impacted diversity?

**While it was announced a little while back, this is the first year that the rule will be fully in effect as last year students who’d applied as 1st years were allowed to take the 2nd shot they’d expected to get.

Note that the NSF’s mission isn’t to award the absolute best scientists regardless of diversity. It isn’t an award sheerly on academic merit, on the number of Science and Nature and Cell papers you’ve got (or even for stellar bioRxiv preprints). It’s aiming to¬†identify and inspire a diverse future.

The NSF GRFP is not an R01. It doesn’t fund high-risk experiments and expect grand returns. For 2000 people a year, give or take a few, it provides a chance to do research without financial pressure, as it is — with a scant living wage and freedom from burden.

But is it providing those resources to people who genuinely don’t have them? Is it funding fully-funded scientists with star-studded credentials or people who had to take student loans through college?

I downloaded data about 28,106 NSF recipients from 2011-2017, and matched the 700+ institutions they hail from with the College Score Card, a resource from the US Department of Education which has extremely comprehensive** information on accredited colleges in the United states.

**When I say this I’m not kidding. They’ve got the percentage of Pell Grant recipients who died within 2 years of starting at the university. All kinds of intensely random stuff. It’s really cool.

And I used this to study¬†only the undergraduate institutions¬†of NSF GRFP recipients. I’d theorize that properties of these reflect their opportunities prior to the beginning of their scientific career as graduate students. While I don’t know the story of any individual NSF recipient, I can say a lot about how diverse their undergraduate school’s populations are.

1. The most expensive undergraduate schools have an extreme excess of recipients.school_cost_award_excess.png

The median school with any undergraduates who win the NSF has 2 winners in any given year. However, if you stratify schools by cost, the top most expensive schools have over a fourfold excess of winners. The cheapest schools have extremely few. These correspond to yearly tuition differences of nearly $10k at private schools and $5k at public schools Рfairly large differences, compounded over 4 years. It could cost an undergraduate $40k extra to attend a school that has any shot at getting them an award.

How are they paying for that?

2. The schools GRFP undergrads go to have smaller Pell grant populations.

pell_schools_v_all.png

The Pell grant is a subsidy that the US Government provides for students who need it in order to afford tuition. They’re limited to first-time bachelors’ degree recipients in genuine financial need. And notably, although quite a few students get them across the entire College Score Card dataset, the proportion needing them at schools that GRFP recipients come from is strikingly lower, for public and private schools.

To get a sense of the starkest differences, I look at schools which had even 1 undergrad recieve an NSF from 2011 through 2017. 

3. There is a difference of nearly $30k between family incomes at schools with and without even a single NSF recipient.

Screenshot 2018-04-02 23.22.38.png

Each College Score Card school has a reported family income for dependent students (e.g., students being claimed as dependents by their parents). In both public and private schools, the difference in mean between schools with just one NSF recipient (to say nothing of those with outlandishly many) is $30k – ironically, that’s about the size of the extra graduate stipend they’re about to win.

4. There are 16% more first-generation undergraduate students at schools with no NSF GRFP undergraduate recipients.

Screenshot 2018-04-02 23.22.21.png

At both public and private schools, according to the College Score Card, there’s a difference of 12-14% in the number of first generation college students in the school population between the schools whose undergrads earned even 1 award during 2011-2017 and those who earned none at all.

“But Natalie, what if the students at these high-income schools who are winning awards are there on scholarships, and your plots don’t represent them?”

Sure, that’s obviously possible and a big caveat. I think there are an incredible amount of deserving, hardworking people of diverse backgrounds at top institutions. ¬†Certainly everyone who gets the GRFP, and many people who don’t, deserve it.

So really we want to ask: Should we consider the NSF GRFP a success if it by and large gives resources to schools that already have the resources to recruit and inspire diversity? What about the incredible deserving people at other institutions who could truly be inspired by the opportunity to attend graduate school?

“These are honestly just a few measures of economic opportunity and equality. I’d rather see…”

I like your style! To placate you, check out this Shiny I built app where you can look at a lot more about the schools and compare award winners and non-winners.

GRFP Undergrad Institutions

“So then what are you saying?”

This part of the mission statement has stuck with me throughout this analysis:

GRFP continues to identify and to inspire the diverse scientists and engineers of the future, and especially encourages women, members of underrepresented minority groups, persons with disabilities, and veterans to apply.

This is a great and noble goal. But do I buy that the entire pool of outstanding diverse future scientists is hiding inside the same few halls of learning? No. There are graduate institutions who win an extreme excess amount of GRFPs where, as I know myself (since I trained at one!), students are already fully funded, mostly on RAships.

There the NSF GRFP becomes a cap feather, not a guarantor of stability the way it could be in another circumstance.

So does that fulfill its mission?

——–

But let’s get back to the bigger question. We may not be surprised to find that the NSF GRFP is not awarded to the most diverse, most needy group of future inspired scientists (regardless of its mission).

But how has the new policy affected that?

top20.png

For more that, stay tuned for part 2, as I crunch the numbers for the 2018 NSF GRFP, relative to the 2011-2017 classes.

How has this policy change impacted diversity?