• CGI Exoplanet Imaging Data Challenge (now closed)
• Microlensing Data Challenge #1 (now closed)
  • Results
• CGI Exoplanet Spectral Imaging Data Challenge (now closed)
  • Results

Coronagraph Instrument Exoplanet Imaging Data Challenge (now closed)

What was the Challenge?

The Roman Exoplanet Imaging Data Challenge is for exoplanet scientists who are interested in learning the art and science of high contrast imaging of exoplanetary systems. Roman's Coronagraph Instrument (CGI), with a possible Starshade, is a high contrast exoplanet imaging instrument planned for flight in the next decade. The Data Challenge is an excellent way to become familiar with the intricacies of the first spaceborne mission with a coronagraph using active wavefront control, as a pathfinder to future flagship missions. The challenge will focus on point-source extraction (angular and reference differential imaging) and astrometry (orbital fitting).

Some of the problems that participants will face:

• Extract hidden objects around a nearby star with PSF subtraction and eventually post-processing
• Discriminate injected planet(s) from eventual background, noise, or contamination source(s) using parallax and proper motion information
• Find orbital solution(s) combining:
  • Several astrometric epochs throughout several years of the mission including both Hybrid Lyot and Starshade coronagraphs
  • Known radial velocity measurements
• Determine physical parameters of the found exoplanet(s): phase(s), albedo(s)

For more information about the data challenge events, data and results, please visit the data challenge website.

What were the Results?

The results of the Coronagraph Instrument Data Challenge were announced at a public event on April 30th, 2021. During this Final Jamboree, the team announced the winners of the data challenge, the prizes, and discuss the lessons learned. The team also described access to the legacy tutorials and data. The finalist teams gave feedback on their techniques used to complete the challenge.
 

Microlensing Data Challenge #1 (now closed)

What was the Challenge?

WFIRST is expected to detect thousands of microlensing events, including hundreds of planetary ones. Traditionally, interesting lightcurves were modeled one-by-one given that the most exciting events were relatively rare. Such interactive modeling will not be possible as the data set grows, so to fully exploit this dataset, analysis techniques need to be upgraded.

To stimulate research in this area, there will be a series of microlensing data challenges, the first of which is based around a large set of simulated WFIRST lightcurves. This dataset has now been released, with a submission deadline of Oct 31, 2018.

For more information, please visit http://www.microlensing-source.org/data-challenge/ .

There is also a python notebook tutorial to provide a starting point for newcomers wishing to take part.

What were the Results?

A total of 293 light curves in filters Z087 and W149 were simulated by Matthew Penny, and this included 74 single lenses (including free floating planet candidates), 83 binary star lenses, 43 planetary binary lenses, and 93 cataclysmic variables. The simulated light curves mimicked the cadence, length, and noise properties of the multi-year WFIRST Bulge survey.

Four teams entered the challenge, including 16 participants in total, seven of whom were newcomers to the microlensing field. Teams were tasked with classifying each light curve and providing fits to the model parameters of each. Teams were evaluated on the accuracy of their fitted parameters, the efficiency and scalability of their software and modeling approaches, innovations they brought to the table, and the extent to which newcomers participated.

All teams used publicly available software, and progress was made on the issue of scalability, although there is still room for improvement. Some new approaches to classification and detection were developed but are still in their early stages. Results from each of the teams showed that, when microlensing events were properly classified, parameters were also accurately derived, modulo some known weaknesses such as a tendency to overestimate the impact parameter. However, the problem of classification is non-trivial, particularly for subtle anomalies in the light curves. Formal benchmarking, which was not attempted in this challenge due to logistical reasons, would have provided a more meaningful comparison between teams. More details can be found in this slide package.

Written feedback was sent to each team, and a paper documenting the challenge and its results is now being written. A second data challenge, building on the lessons learned here, will soon follow.

Coronagraph Instrument Exoplanet Spectral Imaging Data Challenge (now closed)

What was the Challenge?

Welcome to the inaugural CGI Exoplanet Data Challenge! The WFIRST mission was in Phase A, during which time the science and instrument performance requirements were defined for exoplanet imaging and spectroscopy. In order to provide the project with the best possible inputs before the end of Phase A in 2017, we were seeking participation from teams with spectral retrieval expertise through the data challenge.

The Challenge ran from August 2016 to March 2017. The 2016 Challenge consisted of a blind spectral retrieval exercise using simulated, extracted spectra for several known RV and/or hypothetical discovery exoplanets. The spectra will NOT need to be extracted from simulated IFS data. Instead, we explored the impact of signal-to-noise ratio and spectral resolution on the detection/measurement of atmospheric abundances and other planet properties. Even with that relatively simple goal, we expected the Challenge to be non-trivial!

What were the Results?

The CGI Exoplanet Spectral Imaging Data Challenge is a working group within the SIT "Harnessing the Power of the WFIRST-Coronagraph" (PI, Margaret Turnbull). The goal is to provide a quantitative and qualitative feedback to the teams defining the specifications and requirements of the WFIRST coronagraph by blindly retrieving realistic estimates of the expected planetary and atmospheric properties that can be recovered in the WFIRST mission.

Renyu Hu, Tyler Robinson, and Jake Lustig-Yaeger produced simulated data representative of the Integral Field Spectrograph (IFS) instrument of WFIRST. The simulations covered different types of planets (Hot giants, Sub-Neptunes, and rocky ones). They included different instrumental modes of the WFIRST IFS and covered a representative range of different signal-to-noise scenarios. They had different concentrations of CH₄, NH₃, an H₂O in their atmospheres, as well as different temperatures and masses. In this Data Challenge, we considered two giant planets, one sub-Neptune, and two rocky planets.

There were numerous participants from institutions in Asia, Europe, and the US, with several members outside the WFIRST community, and we held regular telecon meetings to share our results.

One important outcome from the Data Challenge was the cross-comparison of the atmospheric modeling among the different teams, significantly increasing the agreement among them, while still leaving room for a variety of atmospheric models. Some teams were able to recover atmospheric parameters.

Two teams, NEMESIS (P. Irwin, J. Eberhardt, and R. Garland) and the team led by M. Marley, R. Lupu, and M. Nayak, were able to recover the atmospheric parameters for the first two giant planets including a variety of models with clouds and hazes. Our major conclusion is that CH₄ concentration was well recovered for the WFIRST data. NH₃ and H₂O recovery was acceptable. The latter was highly dependent on the signal-to-noise ratio and absorption tables used to derive the results. Finally, fundamental parameters, such as the mass of the planet, had a high statistical precision but were somewhat biased depending on the atmospheric model.

The figure above shows the results from two teams. Left: Data and retrieved spectra (credit M. Nayak). Right: Recovered atmospheric parameters for one gas giant (credit: R. Garland).