Scalable distributed workflow for high-resolution land cover and land use classification across large geographic areas (Mediterranean Basin). Orchestrates Sentinel-2 satellite granule ingestion with Dask, atomic MongoDB coordination locks, and MinIO/S3 lifecycle management, reducing data processing overhead by 80%.
Classifying multi-temporal land cover over extensive geographic areas such as the Mediterranean Basin requires processing hundreds of Sentinel-2 granule tiles across four seasons. The sheer volume of raw multi-band raster data (terabytes) easily causes network bottlenecks, disk exhaustion, race conditions between workers, and days of computing overhead.
Designed a divide-and-conquer distributed architecture decoupling compute workers from storage. Workers coordinate dynamically via an atomic tile_claims collection in MongoDB with TTL-based crash recovery, download granules on demand, compute 11 multi-spectral and biophysical composite bands (NDVI, NDWI, SAVI, BSI), and stream intermediate products into MinIO/S3 object storage with automated chunked purge lifecycles.
Engineered the Dask distributed pipeline and parallel tile/season execution flow, achieving an 80% speedup in end-to-end dataset generation. Implemented distributed synchronization locks preventing duplicate tile downloads, and containerized the end-to-end pipeline with Docker Compose for reproducible execution across heterogeneous compute nodes.
Published in Springer Nature Journal of Big Data (DOI: 10.1186/s40537-023-00770-z). Serves as an institutional foundation for automated land use classification and agricultural monitoring models in European research programs.
EnlargeHigh-resolution classification map generated from multi-temporal Sentinel-2 composites and DEM data over the Iberian Peninsula.