Multimodal foundation models for EO
Multimodal Foundation Models for EO
Foundation Models (FMs) represent the latest leap forward in Artificial Intelligence (AI), following the era of Deep Learning. Trained on vast amounts of unlabeled data through self-supervised learning (SSL), these models capture rich patterns that can be applied to a wide array of downstream tasks—even with limited or no additional training data. This paradigm holds particular promise for Earth Observation (EO) and Earth Sciences by enabling breakthroughs in analytical, predictive, and even prescriptive capabilities.
In EO and Earth Sciences, FMs can significantly enhance applications such as weather prediction and geospatial semantic data mining. By analyzing large-scale climate and atmospheric datasets, they deliver more accurate forecasts across different time horizons and reveal complex patterns in environmental systems. Their latent space representations and embeddings also enable powerful insights while reducing the need for extensive labeled data—a critical advantage in remote sensing, where labeling is often expensive and time-consuming.
Despite these benefits, integrating FMs into EO workflows poses distinct challenges. EO data often spans multiple modalities, resolutions, and spectral bands, requiring specialized adaptation and careful model updating—especially for “digital twin” scenarios where AI must remain synchronized with real-world changes. Moreover, FMs demand significant computational resources and optimized training strategies, particularly when handling enormous, continuously growing geospatial datasets. Evaluating and benchmarking FMs for these specialized applications further complicates their deployment, as existing benchmarks may be limited in scope.
This session will focus on Multimodal foundation models for EO
Instructor
Manil Maskey
Biography
Manil Maskey is a Senior Research Scientist with the National Aeronautics and Space Administration (NASA). He also leads the Advanced Concepts team, within the Inter Agency Implementation and Advanced Concepts at the Marshall Space Flight Center and Science Mission Directorate’s Artificial Intelligence initiative at NASA HQ. His research interests include computer vision, visualization, knowledge discovery, cloud computing, and data analytics. Dr. Maskey's career spans over 21 years in academia, industry, and government. Dr. Maskey is an adjunct faculty at the UAH Atmospheric Science department, a senior member of Institute of Electrical and Electronics Engineers (IEEE), chair of the IEEE Geoscience and Remote Sensing Society (GRSS) Earth Science Informatics Technical Committee, member of American Geophysical Union (AGU) and AGU Fall Meeting Planning Committee, member of European Geosciences Union (EGU), and member of Association for Advancement of Artificial Intelligence (AAAI).
Gabriele Cavallaro
Biography
Gabriele Cavallaro (Senior Member, IEEE) received his B.Sc. and M.Sc. degrees in Telecommunications Engineering from the University of Trento, Italy, in 2011 and 2013, respectively, and a Ph.D. degree in Electrical and Computer Engineering from the University of Iceland, Iceland, in 2016. From 2016 to 2021, he served as the deputy head of the "High Productivity Data Processing" (HPDP) research group at the Jülich Supercomputing Centre (JSC), Forschungszentrum Jülich, Germany. Since 2022, he has been the Head of the "AI and ML for Remote Sensing" Simulation and Data Lab at JSC and an Adjunct Associate Professor at the School of Natural Sciences and Engineering, University of Iceland, Iceland. From 2020 to 2023, he held the position of Chair for the High-Performance and Disruptive Computing in Remote Sensing (HDCRS) Working Group under the IEEE GRSS Earth Science Informatics Technical Committee (ESI TC). In 2023, he took on the role of Co-chair for the ESI TC. Concurrently, he serves as Visiting Professor at the Φ-Lab within the European Space Agency (ESA), where he contributes to the Quantum Computing for Earth Observation (QC4EO) initiative. Additionally, he has been serving as an Associate Editor for the IEEE Transactions on Image Processing (TIP) since October 2022.
Iksha Gurung
Biography
Iksha Gurung is a Computer Scientist working with University of Alabama in Huntsville, supporting National Aeronautics and Space Administration Inter-Agency Implementation of Advanced Concepts Team (NASA-IMPACT). He leads the development and machine learning team in NASA-IMPACT. His projects include applying machine learning to Earth science phenomena studies and scaling the solutions to production.
Johannes Jakubik
Biography
Johannes is a researcher at IBM Research. He has a background in applied machine learning, having worked on a variety of topics at KIT, ETH Zurich, and during internships in industry. At IBM Research, Johannes is responsible for finetuning geospatial foundation models to downstream applications in collaboration with NASA Impact.
Alexandre Strube
Biography
Alexandre has a PhD in High-Performance computing by the University Autònoma de Barcelona. He worked at the Performance Analysis team at the Jülich Supercomputing Centre from 2010 to 2015, on the Application Support team from 2015 to 2019, and since then he is a Consultant at Helmholtz AI. He is also one of the maintainers of the whole Scientific software stack on Juelich's supercomputers, and he is the official maintainer of LMOD, the module system, for Debian and Ubuntu operating systems. Alexandre develops and maintains Blablador, the LLM inference infrastructure of the Helmholtz Foundation.
Rocco Sedona
Biography
Rocco Sedona (Member, IEEE) received the B.Sc. and M.Sc. degrees in information engineering from the University of Trento, Trento, Italy, in 2016 and 2019, respectively, and the Ph.D. degree in computational engineering from the University of Iceland, Reykjavik, Iceland, in 2023. He is a member of the “AI and ML for Remote Sensing” Simulation and Data Lab, JSC, Germany. His research interests primarily lie in the field of deep learning and its application to remote sensing data. He has extensively utilized optical satellite data acquired by Landsat (NASA) and Sentinel (ESA) missions toward near real-time land-cover classification. In addition, he specializes in distributed deep learning on high-performance computing systems, an area of study that he has been actively engaged in since 2019.
Þorsteinn Elí Gíslason
Biography
Þorsteinn Elí Gíslason received a B.Sc. degree in Physics and an M.Sc. degree in Computational Engineering from the University of Iceland. His master's research focused on foundation models for Earth observation as part of the Prithvi-EO-2.0 project, where he contributed to both pretraining on high-performance computing (HPC) systems and downstream validation. He is currently a Researcher at the Jülich Supercomputing Centre, Forschungszentrum Jülich, where he works on foundation model research for remote sensing. His contributions span both pretraining large-scale models on HPC systems and fine-tuning them for Earth observation applications. Additionally, he is involved in improving hybrid workflows that integrate HPC and cloud resources.