As Earth’s climate continues to change in ever more dramatic ways, NASA is a core contributor to our understanding of Earth’s complex climate system. Developing the models to explain and predict the future evolution of this system requires massive amounts of data from numerous Earth observing instruments, many of which are developed and operated by NASA. The CLARREO Pathfinder (CPF) mission plays a unique role in not only gathering new measurements but also improving the accuracy and consistency of existing measurements (made by other instruments) via its inter-calibration and SI traceability features.
The core of the CPF mission is the HyperSpectral Instrument for Climate Science (HySICS) a next gen hyperspectral imaging spectrometer developed by the Laboratory for Atmospheric and Space Physics (LASP) at the University of Colorado Boulder. A defining feature of the HySICS instrument is the ability to measure Earth reflected solar radiation, Moon reflected solar radiation, and incoming solar irradiance directly with the same sensor via a 2-axis gimbal and variable aperture. This enables on-orbit calibration as well as reference to a well-characterized source. The pointing capability also enables the unique inter-calibration feature of CPF, where the HySICS instrument will be tasked to take measurements which correspond physically and temporally with the VIIRS and CERES instruments aboard the NOAA20 satellite. These coincident measurements can then be used to increase the accuracy of these existing sensors. The CPF mission calls for the HySICS instrument to be hosted on the International Space Station for a full year of measurements once installed.
The improved accuracy of CPF measurements translates directly into more accurate climate models. Better models show not only how the climate is changing, but the factors which create the largest impact. Better models provide policy makers with the means to create more effective policies which result in significant economic, defense, and societal benefits.
In preparation for launch, a critical task involves building the C-SDS and IC-SDS data systems which will produce the L1 (measurement) and L4 (inter-calibration) data products respectively. The CPF Science Data System (C-SDS), developed by LASP, takes the raw data from the HySICS instrument and produces the L1 products which contain calibrated and georeferenced radiance and reflectance data. The Inter-calibration CPF Science Data System (IC-SDS) is being developed by the inter-calibration science and data management teams at NASA Langley Research Center (NASA LaRC) supported by Mechdyne.
A mission-critical deployment
Central to this effort are Mechdyne Senior Software Engineers Adam Thurston and Aron Bartle from the company’s Embedded Software Services division. They form the CPF Inter-calibration Data Management team which is tasked with designing and developing the automated system needed to support the CPF inter-calibration mission objective. This requires implementing the algorithms developed by the science team, composing and executing the resulting workflows in a reproducible and scalable way on cloud resources, and finally designing and developing an AWS serverless system which will integrate with existing NASA archive and metadata services to acquire the dynamic dependencies required by the science workflows. The duo leverages expertise in cloud-based serverless architecture, Python/Fortran/C++ based numerical computing, distributed systems and networks, the Nextflow execution engine, and Linux scripting and automation in support of this task.
The high degree of training and deep technical knowledge required by such work is not easy to come by, and NASA looks to Mechdyne both for qualified software engineers and for the stability it needs from its team members. Adam and Aron exemplify the kind of people NASA seeks. Both have been working with Mechdyne for more than twenty years, and today each has amassed several years of experience on the CLARREO project specifically. “We’re pretty well enmeshed on the client side,” says Aron. “We are treated as full members of the team – because that’s really what we are.” Adds Adam: “We work with the data management lead and science lead to determine priorities, but system design and implementation decisions are left to us. We ensure that the solutions meet the requirements – written and unwritten.”
The data management team has also been instrumental in some of the major decisions which have shaped the project, including the adoption of the Python programming language for all production algorithm implementations and the decision to exclusively use cloud resources for both the processing and analysis systems. “CLARREO was one of the first science data systems to be designed from the ground up to take advantage of new serverless technologies” says Aron.
Prior to joining the CPF mission, Adam worked on several of Mechdyne’s high-performance distributed visualization products including Conduit, TGX, and Meeting Canvas. He also specialized in the integration of large-scale immersive VR systems. Aron has been supporting NASA contracts for more than 12 years and before that worked on high-performance immersive visualization products including vGeo and Conduit.
Understanding enabled, resources conserved
One measure of the value the Mechdyne team brings to the CPF project can be found in the amount of time and resources that have been conserved by the move to a serverless architecture. “The inter-calibration workflow has a spiky nature, jobs may sit idle for days or even weeks until dependencies become available. Once a job is ready to run, however, we want it to complete as quickly as possible.” says Aron. “This pattern is difficult to support efficiently with a dedicated compute cluster located in an on-prem datacenter. By transitioning to cloud-based on-demand computing resources, we can avoid the acquisition and support costs of a cluster (that may only be used a fraction of the time) while also shortening processing time by scaling the compute resource to match the job dynamically as it is needed.”
Adam is quick to add, however, that moving to the cloud does not automatically save money or time. “When resources can be scaled easily, there is a temptation to just throw more hardware at a task to meet performance goals. This can quickly backfire both in increased cloud spend and increased system complexity which makes it more difficult to predict future performance. As a small but experienced team we rely on our background in distributed systems and performance analysis to optimize the system where it will be most effective while still being responsive to changing requirements.”
And what does the future hold for the Mechdyne team and the CLARREO Pathfinder mission? “We look forward to the launch of the CPF mission and its installation on ISS. The instrument, and algorithms that have been developed to support it, will make valuable contributions to the important field of climate science,” says Adam. “The knowledge that Mechdyne has played an import role in bringing this about is gratifying both on a professional and personal level.” Going forward, the capabilities honed by the team will be transferrable to a range of other current and future NASA projects. “The knowledge of how you discover, acquire, and efficiently process data is broadly applicable,” says Aron. “As NASA’s needs evolve, I’m confident in our ability to evolve along with them.”
