Big Data in Astronomy
Alongside the recent explosion of “Big Data” into the public consciousness, there has been a similar transition into the age of “Big Astronomy“. Astronomers have always been adept at drawing conclusions using advanced statistics and data analysis. Now, with the advent of extremely large simulations like Illustris and surveys like the upcoming LSST, astronomers are increasingly gaining experience in dealing with datasets vastly larger than could ever hope to fit on a single computer.
With these skills in hand, many scientists with Astronomy degrees are seeking (and finding!) very successful jobs in data science and analytics. In April 2016, the Wolbach Library at the Harvard-Smithsonian Center for Astrophysics hosted a panel discussion featuring several former astronomers who have since leveraged their degrees to find data science jobs. This “Alternative Careers” panel was extremely well attended, and I was one of a few dozen students, postdocs, and even faculty who stopped by. The following is a summary of the panel, and the thoughts these panelists had for astronomers who are interested in considering jobs outside of academia.
The Panelists
Nathan Sanders received a PhD in Astronomy and Astrophysics from Harvard in 2014, and is now the Senior Director of Quantitative Analytics at Legendary Entertainment, the Hollywood studio which produced the Dark Knight trilogy, Inception, and the upcoming film Warcraft. Nathan is also a co-founder of Astrobites.
Cameron McBride received his PhD from University of Pittsburgh, followed by postdoc positions at Vanderbilt and the Harvard-Smithsonian CfA. He is now a Senior Data Scientist at True Fit, a startup providing data analytics to the fashion industry.
Nadieh Bremer received her Masters in Astronomy from Leiden University before working as a data analyst for Deloitte Consulting. After discovering her passion was in data visualization, she began working at Adyen as a Data Visualization Designer, and hosts her own data-viz blog, VisualCinnamon.
Three data scientists with degrees in astronomy joined the Alternative Careers panel. (Left) Nathan Sanders of Legendary Entertainment and Astrobites. (Center) Cameron McBride of True Fit. (Right) Nadieh Bremer of Adyen.
Why do astronomers want jobs in data science?
A frequently discussed topic was simply why astronomers are looking for positions in Data Science. Many astronomers are interested in jobs outside of academia because of how competitive the faculty job market is, and because of the increasing difficulty of obtaining funding through research grants.
However, when asked why they decided to go into a non-academic career, none of the panelists said their primary reason was monetary or because of the tough job market in astronomy.
McBride emphasized how much he enjoyed the low-stress environment of working in a startup. Bremer said she enjoyed the opportunity to rapidly present the results of new research, without having to go through the process of writing papers. All three mentioned that the transition to data science has allowed them to tackle new challenges using tools no one has ever tried before. Sanders added that this provides an opportunity to make a really big impact within the company or the industry.
Why do data science companies want astronomers?
The panelists also shed light on the other side of the equation: why astronomers are successful at getting these data science jobs. As mentioned in the introduction to this article, astronomers today are routinely trained to handle “big data”. But astronomers have additional skills, which are difficult to foster in other fields that also deal with very large datasets.
According to Sanders, a major upside to training as an astronomer is how independent PhD thesis research is. While some astronomers work in very large collaborations, it is also very common for them to work one-on-one on a small project with their advisor. This gives them experience in seeing a project from start to finish, as well as in how to design a good experiment.
McBride added that astronomers tend to be less specialized than scientists in other fields, allowing them to learn a variety of useful skills. For example, it is extremely common for astronomers to be very experienced in programming, even developing their own analysis pipelines. As he put it, “We make our own tools when we need a better one”.
What do you miss from astronomy/academia?
When asked what elements of academia they miss, Bremer and McBride both mentioned that they occasionally miss doing “fundamental research” on topics with more impact on science. McBride also said he misses having the opportunity to interact with students and advocate for science and teaching. Sanders discussed the decreased timeframes of project, often having only a few days or weeks to work on a chunk of a project before having to move on to another “crisis”.
Bremer found that the consulting workflow also restricted her to firm deadlines and frequently-shifting projects, but these were advantages for her, as she got the opportunity to constantly adapt to new datasets and new programming languages. McBride, meanwhile, mentioned that working for a startup allowed him more freedom to explore his own new ideas.
Sanders, who was brought on early in the lifespan of Legendary’s QA department, was given the opportunity to import some aspects of the academic process into industry. He introduced journal clubs and group meetings, which helped foster the exchange of ideas between people working on different projects.
How can I pursue a job in data science?
It seems clear that astronomers will continue to have a major impact on the growing industry of data science. For those academics who are considering jobs outside the field, McBride recommends pursuing a few data science topics a bit more deeply. Experiment with small projects (even side projects outside astronomy) to help increase your confidence with particular languages, tools, or statistical techniques. Yet you don’t need to be a world-expert in machine learning to get a job. As Sanders puts it, “Companies are looking for generalists”, people who show they can take existing tools and find new uses for them. He added that “there is no such thing as an original idea”, and that exposing yourself to ideas from other disciplines is essential.
The panelists emphasized how useful networking can be in finding industry jobs. Bremer went to in-house days and networking fairs, and shadowed people in different industries before choosing consulting, while Sanders struck up collaborations with students in statistics and computer science departments. Bremer further mentioned that none of your decisions will be binding. “Even if you make a bad choice” with a company or industry, “you can try again!” she said, speaking about her transition from consulting into data visualization with Adyen.
Even within astronomy, there are great opportunities for developing strong skills as a data scientist. The LSST Collaboration is accepting applications for its first Data Science Fellowship Program, with the goal of teaching graduate students essential skills for dealing with the kinds of big data that will come from future surveys.
The most valuable resources, however, are our fellow astronomers who are making these transitions themselves. Many have begun discussing their choices and methods in blog posts, and other interviews. Other online resources can be found here and here. At your own institution, you can speak to your department or library about holding panels like the one summarized here. Your department will be the best resource for finding former students who are now engaged in careers outside academia.
Ongoing Discussion
If you’re interested in jobs in data science, keep the conversation going! Comment below with your questions, thoughts, or personal experiences about jobs inside/outside academia. If you have good resources that have been helpful for you, you can also link to them in the comments, and we’ll add them to this post.
