Cloud processing

Processing large-volume geospatial data in the cloud.

Challenge

Several partners have shared their experience in implementing, deploying, and operating scalable EO services, as well as their experience in data mining. This consortium formed an ideal team combining complementary experience, capacities and motivation:

Here, mundialis specialized in the analysis of remote sensing data and the processing of extensive geodata in high-performance data centers with open source software. Given that mundialis is also heavily involved in the activities of OSGeo and the open source geospatial and EO community, the goal was to provide access to the rich remote sensing data to different groups of people. The experience in scalable cloud processing and analysis routines for Sentinel and other satellite data could be ideally brought into the project. The motivation was to integrate processing chains into operational cloud platforms to improve the geoprocessing cloud engine Actinia, an OSGeo community project, in terms of stakeholder scenarios.

Services
  • Automated deployment of the infrastructure
  • Creating Kubernetes clusters and automatically updating them using Helm charts.
  • Optional use of Charliecloud containers to minimize potential security risks
  • Parallelized work processes
  • Data processing: compatibility with the openEO API
  • STAC catalogs
Result

STAC catalogs

Kubernetes cluster

parallelized work processes

Customer
German Aerospace Center (DLR)

Story

In recent years, the tremendous growth of Earth Observation (EO) data, reaching 12 terabytes per day through the Sentinel program alone, has created new challenges for archiving, distributing, and using these ever-growing time series. In response, mundialis has developed forward-looking prototypes for cloud-based infrastructures that enable efficient processing and analysis of large EO time series.

Projects implemented major components of pilot applications focused on sentinel time series. This includes deploying actinia and the openEO GRASS GIS backend on Kubernetes clusters and automatically updating them using Helm charts for Kubernetes. In addition, the methods for time series analysis have been extended both in GRASS GIS and in the openEO GRASS GIS backend.

Most recently, an innovative redesign of actinia took place, in which actinia was divided into a manager and a worker part in order to optimize the work processes. This division was made possible by outsourcing the job queue to a Redis database and supported by newly developed actinia plugins for parallelization.

Another advance was the integration of actinia into a high-performance cluster (HPC) using Charliecloud containers and the implementation of the actinia-stac plugin to create STAC catalogs directly from the data. All these innovations lead to automated installation of actinia via pipelines, including the creation of Charliecloud containers, minimizing potential security risks.