ChatGeo-Magi

ChatGeo-Magi is a conversational AI system I developed while conducting research with NOAA and the University of Colorado Boulder. It combines retrieval-augmented generation with tool calling to make geomagnetic models and datasets accessible through natural-language queries.

I built the Python system using LangGraph and AWS, integrating NOAA REST APIs to calculate magnetic field components, retrieve scientific data, and generate visualizations. I also developed automated evaluation workflows using DeepEval and Opik to assess response accuracy and hallucination risk across model configurations.

The project led to conference presentations and a research manuscript. Explore the source code on GitHub or read the manuscript preprint on ESS Open Archive.

Original submitted abstract

The following abstract is preserved from the submitted manuscript. It describes the research at submission and does not indicate its current publication status.

We introduce ChatGeo-Magi, a conversational AI agent that leverages Retrieval-Augmented Generation (RAG) and structured tool calling to provide intuitive, grounded access to geomagnetic knowledge and resources. The geomagnetism group in the Cooperative Institute for Research In Environmental Sciences (CIRES) with the National Oceanic and Atmospheric Administration (NOAA) develops and distributes magnetic field models essential for navigation, exploration, and research. However, accessing these specialized datasets often requires domain expertise to navigate complex interfaces.

We implement two architectures: a dual Large Language Model (LLM) system for reliable single-stage API interactions, assessed through qualitative case study, and an agentic framework enabling multi-step reasoning, evaluated quantitatively across three local models. Critically, tool outputs in the agentic system are reintroduced into model context to ground responses in computed results rather than prior knowledge or approximation. The system can query NOAA geomagnetic APIs, perform calculations for magnetic field components, and generate time-series plots and contour maps, in addition to answering domain-specific questions.

Evaluation reveals that expanding the RAG corpus with curated support emails produced small and inconsistent effects across models, suggesting that data quality and retrieval precision may be more important than simply expanding the retrieval corpus. The agentic system demonstrates greater flexibility for complex queries and visualization, demonstrating the potential of LLMs in geoscience applications.