AI and machine learning

In depth

What it is used for

A review of more than 200 studies found deep learning applied across remote sensing image analysis, including image fusion, image registration, scene classification, object detection, land use and land cover classification and segmentation [1]. Land cover segmentation labels every pixel. Dynamic World, for example, applies deep learning to 10 m Sentinel-2 imagery to publish land use and land cover predictions continuously, alongside new acquisitions, rather than once a year [2]. Object detection finds and outlines individual things such as ships, buildings or solar farms. A survey of 119 papers on finding vessels in optical imagery noted that radar is still the leading technology for maritime monitoring, and that weather, sea state, cloud, haze, sun angle and the sensor all affect detection accuracy [3]. A global inventory built with machine learning located 68,661 commercial, industrial and utility-scale solar facilities [4].

Clouds must be masked before almost any optical analysis. Methods that classify each pixel by its spectrum alone tend to mistake bright surfaces such as snow, ice and buildings for cloud, and a convolutional network that also uses the spatial pattern improved detection, especially over snow and ice [5]. Super-resolution predicts a finer image than the sensor recorded. One network sharpens the 20 m and 60 m Sentinel-2 bands to 10 m, and it is trained by shrinking real images, so its accuracy can only be measured at that reduced scale, where the true finer image exists [6]. At the full scale there is nothing to check it against. Detail a super-resolution model adds is invented from what it learnt in training, not measured, and should not be used to count, measure or identify objects smaller than the sensor's own pixel. Change detection compares the same place on different dates, and the same kinds of network are used for it alongside the methods on the analysis methods page. Mapping a flood or a burn scar after an event is a closely related task, and both were among the tasks a foundation model described below was fine-tuned for [7].

The main kinds of model

A random forest is a collection of decision trees, each grown on a random sample of the training data and a random choice of the inputs, which vote on the answer [8]. In remote sensing it handles many correlated inputs well and is fast, but it is sensitive to how the training sample was designed [9]. A convolutional neural network learns its own features from the pixels, in layers that build from edges and textures up to whole objects [10]. The U-Net design labels every pixel [11], and the cloud detector above is built on it [5]. A transformer replaces convolution with self-attention, which relates every part of an image to every other part [12], and transformers have spread to very high resolution, hyperspectral and radar imagery [13]. A foundation model is pretrained by self-supervision on a large unlabelled archive and then fine-tuned for a particular task with a small labelled set. Prithvi, a transformer pretrained on more than 1 TB of Harmonized Landsat and Sentinel-2 imagery, was fine-tuned for cloud gap filling, flood mapping, burn scar segmentation and crop segmentation [7].

The limits a buyer should know

Figure 1 sets out the steps behind a machine learning map and what to ask about each. The first is the labelled training data. Models typically need large training sets, made by drawing on high-resolution imagery, collecting in the field or reusing existing data. Those labels are often assumed to be true but almost always contain errors, from how the sample was designed and how it was collected, and those errors can cause substantial errors in the map. They are often not accounted for or reported [14].

The second is validation against independent ground truth. The standard way to report the accuracy of a classified map is an error matrix, which counts a sample of locations by their mapped class and their reference class [15]. Good practice draws that reference sample by a probability sampling design and reports overall accuracy together with the accuracy of each class and estimated areas with confidence intervals [16]. The sample must be independent of the training data, or the check measures only how well the model remembers its examples. With the map's classes in the rows and the reference classes in the columns, overall accuracy is

where $\mathrm{OA}$ is the overall accuracy, a fraction without unit; $n_{ii}$ is the number of sample locations that the map and the reference both put in class $i$; $q$ is the number of classes; and $n$ is the total number of sample locations.

Overall accuracy weighs each class by how often it occurs, so a map can score well overall while missing most of a rare class. That is why accuracy should be reported per class, from both sides of the matrix. User's accuracy is the share of locations mapped as a class that really are that class, and producer's accuracy is the share of locations truly in a class that the map finds [15, 16].

where $\mathrm{UA}_i$ is the user's accuracy of class $i$, a fraction without unit, whose complement is the commission error; $n_{ii}$ is the number of sample locations correctly mapped as class $i$; and $n_{i+}$ is the number of sample locations the map puts in class $i$, the total of row $i$.

where $\mathrm{PA}_j$ is the producer's accuracy of class $j$, a fraction without unit, whose complement is the omission error; $n_{jj}$ is the number of sample locations correctly mapped as class $j$; and $n_{+j}$ is the number of sample locations the reference puts in class $j$, the total of column $j$.

The third is domain shift. A model succeeds only as far as its training samples represent the images it is applied to. When those samples come from another image or another region, differences in acquisition and atmospheric conditions, or changes in what is observed such as crops at another stage of the season, shift the data and are likely to make the model fail. Domain adaptation methods aim to make a classifier robust to such shifts [17]. Accuracy measured in one place, season or sensor is therefore evidence about that place, season and sensor, and a buyer applying a model elsewhere should ask for a check there.