The classification of land cover using eo-learn. Part 3

When results are needed better than “satisfactory”

Part 1
Part 2

The transition zone from the winter season to the summer, is made up of images of Sentinel-2. You can see some differences in the types of cover on the snow, which was described in the previous article.

Foreword

The past couple of weeks have been quite difficult. We published the first and second parts of our articles on the classification of cover in the scale of the whole country with the help of the eo-learn framework. eo-learn is an open source library for creating an interlayer between receiving and processing satellite images and machine learning. In previous articles in the examples, we indicated only a small subset of data and showed results only on a small percentage of the entire area of ​​interest (AOI). I know it looks at least not very impressive, and perhaps very rude on our part. All this time you have been tormented by questions about how you can use this knowledge and move it to the next level.

Do not worry, it is for this that you need the third article in this series! Grab a cup of coffee and sit down ...

All our Data are Belong to You!

Are you already sitting? Maybe leave the coffee on the table for a second, because now you will hear the best news for today ...
We at Sinergise decided to publish the full data set for Slovenia for 2017. Is free. You can easily access 200GB of data in the form of ~ 300 EOPatch fragments, each approximately in the size of 1000x1000, at a resolution of 10m! You can read more about the EOPatch format in the last post about eo-learn , but in fact it is a container for geo-temporal EO (Earth Observation) and non-EO data: for example, satellite images, masks, maps, etc.

EOPatch structure )

We did not bungle when we downloaded this data. Each EOPatch contains Sentinel-2 L1C images, their corresponding s2cloudless mask, and the official land cover map in raster format!

Data is stored on AWS S3 at: http://eo-learn.sentinel-hub.com/

Deserializing an EOPatch object is fairly simple:

 EOPatch.load('path_to_eopatches/eopatch-0x6/') 

As a result, you will receive an object of the following structure:

 EOPatch( data: { BANDS: numpy.ndarray(shape=(80, 1010, 999, 6), dtype=float32) } mask: { CLM: numpy.ndarray(shape=(80, 1010, 999, 1), dtype=uint8) IS_DATA: numpy.ndarray(shape=(80, 1010, 999, 1), dtype=uint8) IS_VALID: numpy.ndarray(shape=(80, 1010, 999, 1), dtype=bool) } mask_timeless: { LULC: numpy.ndarray(shape=(1010, 999, 1), dtype=uint8) VALID_COUNT: numpy.ndarray(shape=(1010, 999, 1), dtype=int64) } meta_info: { maxcc: 0.8 service_type: 'wcs' size_x: '10m' size_y: '10m' time_difference: datetime.timedelta(-1, 86399) time_interval: (datetime.datetime(2017, 1, 1, 0, 0), datetime.datetime(2017, 12, 31, 0, 0)) } bbox: BBox(((370230.5261411405, 5085303.344972428), (380225.31836121203, 5095400.767924464)), crs=EPSG:32633) timestamp: [datetime.datetime(2017, 1, 1, 10, 4, 7), ..., datetime.datetime(2017, 12, 25, 10, 15, 32)], length=80 ) 

Access to different EOPatch attributes is as follows:

 eopatch.timestamp eopatch.mask['LULC'] eopatch.data['CLM'][0] eopatch.data['BANDS'][5][..., [3, 2, 1]] 

EOExecute Order 66

Great, the data is loading. While we are waiting for the completion of this process, let's take a look at the capabilities of a class that has not yet been discussed in these articles - EOExecutor . This module is engaged in the implementation and monitoring of the pipeline and allows you to easily use multithreading. No more searches on Stack Overflow on how to parallelize the pipeline or how to make the progress bar work in this mode - we have already done everything for you!

Additionally, it handles any errors that occur and can generate a brief summary of the progress. The last is the most important moment in order to be sure of the repeatability of their results in the future, so that the user would not have to spend precious working time searching for the parameters that he used last Thursday at 9 am after a whole night of revelry (alcohol and programming do not mix worth it!). This class also generates a nice dependency graph for a pipeline that you can show your boss!

Pipeline dependency graph generated by eo-learn

Machine learning experiments

As promised, this article is mainly intended for studying different models with eo-learn using the data provided by us. Below we have prepared two experiments where we study the effect of clouds and different resampling algorithms during temporal interpolation on the final result. After all this, we will start working with convolutional networks (CNN), and compare the results of the two approaches - pixel-by-pixel analysis by the decision tree and in-depth training using convolutional neural networks.

Unfortunately, it is impossible to give a clear answer about what decisions should be made during the experiments. You can study the subject area more deeply and make assumptions to decide whether the game is worth the candle, but in the end, all the same, the work will be reduced to a trial and error method.

Playing with clouds

Clouds are a huge pain in the EO world, especially when it comes to machine learning algorithms, where you want to define them and remove them from the data set for interpolation using missing values. But how much benefit from this procedure? Is it worth it? Rußwurm and Körner, in their article Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders, even showed that for deep learning the filtering of clouds may be absolutely unimportant, since the classifier itself is able to detect clouds and ignore them.

Activation of the input layer (top) and the modulation layer (bottom) in the sequence of images of a specific fragment for a neural network. You may notice that this fragment of the network has learned how to create masks of cloudiness and filter the results obtained. (Page 9 at https://www.researchgate.net/publication/322975904_Multi-Temporal_Land_Cover_Classification_with_Sequential_Recurrent_Encoders )

Briefly recall the structure of the data filtering step (details in [previous article] ()). After receiving the Sentinel-2 snapshots, we start filtering cloud snapshots. All images in which the number of non-cloud pixels does not exceed 80% are subject to filtering out (the threshold values ​​may differ for different zones of interest). After that, to obtain pixel values ​​on arbitrary days, cloud masks are used to ignore such data.

So there are four possible behaviors:

1. with filter images, given the cloud masks
2. no filter snapshots, given the cloud mask
3. with filter images, not including cloud masks
4. no image filter, disregarding cloud masks

Visual display of the stack of images from the satellite Sentinel-2. Transparent pixels on the left imply missing pixels due to cloudiness. The stack in the center shows the pixel values ​​after filtering the images and their interpolation with the cloud mask (Case 4), and the stack on the right shows the result of the interpolation in the case without filtering the images and without cloud masks (1). (Approx. Trans. - apparently, in the article a typo, and implied the opposite - in the center of the case 1, and the right - 4).

In the last article, we already performed a variation of case 1 and showed the results, so we will use them for comparison. Preparing other conveyors and training the model sounds like an easy task — you just have to make sure that we compare the correct values. For this, it is enough to take the same set of pixels for training and validating the model.

The results are shown in the table below. You may notice that, in general, the effect of clouds on the model's performance is quite low! This may be due to the fact that the reference card is of very good quality and the model is able to ignore most of the images. In any case, this behavior cannot be guaranteed for any AOI, so do not rush to throw this step out of your models!

The impact of different approaches to resampling

The choice of temporal resampling parameters is not obvious. On the one hand, we need a detailed array of images that demonstrate well the details of the original images - I want to include the closest number of images to the original data. On the other hand, we are limited by computational resources. Reducing the resampling step doubles the number of frames after interpolation, and thus increases the number of features that are used during training. Is such an improvement worth the resources spent on? This is what we have to learn.

For this experiment, we will use variation 1 from the previous step. After interpolation, we do resampling with the following variations:

1. Uniform resampling with an interval of 16 days
2. Uniform resampling with an interval of 8 days
3. The choice of "best" dates, the number coincides with case 2.

Sample 3 is based on the largest total number of total dates for all EOPatch in the selected AOI

The graph shows the number of EOPatch fragments that contain data for each day of 2017 (blue). The red lines show the optimal resampling dates, which are based on the Sentinel-2 snapshot dates for a given 2017 AOI.

Looking at the table below, you can see that the results are not very impressive, as in the past experience. For cases 2 and 3, the amount of time spent grows about twice, but the difference with the initial approach is less than 1%. Such improvements are too subtle for practical use, so it can be considered a 16-day interval suitable for the task.

The results of the overall accuracy and weighted F1 for different conveyors with a change in approach to resampling.

Deep Learning: Using Convolutional Neural Network (CNN)

Deep learning has become a standard approach to many tasks, such as computer vision, natural language processing, and signal processing. This is due to their ability to extract patterns from complex multidimensional input data. Classical machine learning approaches (such as decision trees) have been used in many tasks related to temporal geodata. On the other hand, convolutional networks were used to analyze the spatial correlation between adjacent images. Basically, their use was limited to working with single shots.

We wanted to study the architecture of deep learning models, and try to choose one that is able to analyze both spatial and temporal aspects of satellite data simultaneously.

To do this, we used temporal full-convolutional networks (Temporal Fully-Convolutional Netvork, TFCN), or rather, the temporal extension to U-Net, implemented in TensorFlow. More specifically, the architecture uses spatio-temporal correlations to improve the outcome. An additional advantage is also the fact that the network structure allows a better representation of spatial relationships at different scales due to the process of encoding / decoding in U-net. In the same way as in classical models, at the output we get a two-dimensional matrix of labels, which we will compare with the truth.

We used the trained model to predict the labels on the test set, and the values ​​obtained were verified with the truth. Overall, the accuracy was 84.4% and F1 was equal to 85.4%.

Comparison of different predictions for our problem. Visual image (left top), true reference map (top right), LightGBM model prediction (bottom left) and U-net prediction (bottom right)

These results show only the initial work on this prototype, which is not highly optimized for the current task. Despite this, the results converge with some statisticians obtained in the area. To unlock the potential of a neural network, an optimization of the architecture is required (feature set, network depth, number of bundles), as well as setting hyper parameters (learning rate, number of epochs, class weighting). We expect to delve into this topic (ha-ha) even more, and plan to distribute our code when it is in an acceptable form.

Other experiments

You can think of many ways to improve current results, but we can’t bust or try them all. It is at this moment that you appear on the scene! Show me what you can do with this dataset and help us improve results!

For example, one of our colleagues in the near future will be engaged in the classification of cover, based on the temporal stack of individual images using convolutional networks. The idea is that some surfaces, for example, artificial, can be distinguished without temporal features - quite spatial. We will be happy to write a separate article when this work leads to results!

From translator

Unfortunately, the next part of this series of articles did not come out, which means that the authors did not show examples of the source code with the construction of U-Net. As an alternative, I can offer the following sources:

1. U-Net: Convolutional Networks for Biomedical Image Segmentation - Olaf Ronneberger, Philipp Fischer, Thomas Brox is one of the basic articles on U-Net architecture that does not involve temporal data.
2. https://eo-learn.readthedocs.io/en/latest/examples/land-cover-map/SI_LULC_pipeline.html - The eo-learn documentation page, where the (possibly) more recent version of 1.2-part pipelines is located.
3. https://github.com/divamgupta/image-segmentation-keras - A repository with several networks implemented with keras. I have some questions about the implementations (they are somewhat different from those described in the original articles), but in general the solutions are easily adapted for personal purposes and are quite working.