Researchers successfully train a machine learning model in outer space for the first time

Machine learning has a huge potential for improving remote sensing – the ability to push as much intelligence as possible into satellites will make space-based sensing increasingly autonomous. This would help to overcome the issues with the inherent delays between acquisition and action by allowing the satellite to learn from data onboard. Vít’s work serves as an interesting proof-of-principle.

Professor Andrew Markham (Department of Computer Science), supervisor for the project

Data collected by remote-sensing satellites is fundamental for many key activities, including aerial mapping, weather prediction, and monitoring deforestation. Currently, most satellites can only passively collect data, since they are not equipped to make decisions or detect changes. Instead, data has to be relayed to Earth to be processed, which typically takes several hours or even days. This limits the ability to identify and respond to rapidly emerging events, such as a natural disaster.

To overcome these restrictions, a group of researchers led by DPhil student Vít Růžička (Department of Computer Science, University of Oxford), took on the challenge of training the first machine learning program in outer space. During 2022, the team successfully pitched their idea to the Dashing through the Stars mission, which had issued an open call for project proposals to be carried out on board the ION SCV004 satellite, launched in January 2022. During the autumn of 2022, the team uplinked the code for the program to the satellite already in orbit.

The researchers trained a simple model to detect changes in cloud cover from aerial images directly onboard the satellite, in contrast to training on the ground. The model was based on an approach called few-shot learning, which enables a model to learn the most important features to look for when it has only a few samples to train from. A key advantage is that the data can be compressed into smaller representations, making the model faster and more efficient.

Vít Růžička explained: ‘The model we developed, called RaVAEn, first compresses the large image files into vectors of 128 numbers. During the training phase, the model learns to keep only the informative values in this vector; the ones that relate to the change it is trying to detect – in this case, whether there is a cloud present or not. This results in extremely fast training due to having only a very small classification model to train.’ Whilst the first part of the model, to compress the newly-seen images, was trained on the ground, the second part (which decided whether the image contained clouds or not) was trained directly on the satellite.

Normally, developing a machine learning model would require several rounds of training, using the power of a cluster of linked computers. In contrast, the team’s tiny model completed the training phase (using over 1300 images) in around one and a half seconds.

When the team tested the model’s performance on novel data, it automatically detected whether a cloud was present or not in around a tenth of a second. This involved encoding and analysing a scene equivalent to an area of about 4.8 x 4.8 km2 area (equivalent to almost 450 football pitches).

According to the researchers, the model could easily be adapted to carry out different tasks, and to use other forms of data. Vít Růžička added: ‘Having achieved this demonstration, we now intend to develop more advanced models that can automatically differentiate between changes of interest (for instance flooding, fires, and deforestation) and natural changes (such as natural changes in leaf colour across the seasons). Another aim is to develop models for more complex data, including images from hyperspectral satellites. This could allow, for instance, the detection of methane leaks, and would have key implications for combatting climate change.’

Performing machine learning in outer space could also help overcome the problem of on-board satellite sensors being affected by the harsh environmental conditions, so that they require regular calibration. Vít Růžička said: ‘Our proposed system could be used in constellations of non-homogeneous satellites, where reliable information from one satellite can be applied to train the rest of the constellation. This could be used, for instance, to recalibrate sensors that have degraded over time or experienced rapid changes in the environment.’

This project has been summarised in a pre-print publication ‘Fast model inference and training on-board of Satellites’ available at arXiv. The work was also presented at the International Geoscience and Remote Sensing Symposium (IGARSS) conference on 21 July 2023.

This project was conducted in collaboration with the European Space Agency (ESA) Φ-lab via the Cognitive Cloud Computing in Space (3CS) campaign and the Trillium Technologies initiative Networked Intelligence in Space (NIO.space) and partners at D-Orbit and Unibap.

Illustration of the data used for training the tiny cloud classification model (left), and the predictions on new scenes (right). The entire training process took about 1.5 seconds, including the time for encoding the complete training dataset, and 10 epochs of training a classification model. Image credit: Sentinel-2 data (ESA) processed by Vít Růžička.