A retrieval-augmented generation assistant that lets you ask natural-language questions about air quality across Mexican cities, built as the capstone for the DataTalks LLM Zoomcamp 2026. This repo is the RAG/LLM serving layer only — it reads pre-built BigQuery mart tables produced by a separate upstream pipeline, air-quality-dlt-dbt-dagster, and turns them into a queryable chat interface.
Architecture & Stack
- Summarization → Dagster asset (
generate_summaries, monthly schedule) readsair_quality_marts.fct_city_daily_aqianddim_stationsfrom BigQuery and generates 6,596 natural-language text chunks — 6,097 daily summaries (city × parameter × date) and 499 monthly summaries (city × parameter × month) - Embedding → Dagster asset (
embed_chunks) embeds each chunk with OpenAItext-embedding-3-small(1536 dims) into pgvector on PostgreSQL - Retrieval → semantic (vector-only) and hybrid (vector + BM25 via Reciprocal Rank Fusion) modes, selectable per query
- Generation → OpenAI
gpt-4o-mini, with query-language detection so the response comes back in the same language (English or Spanish) as the question - Serving → FastAPI backend (
chat+feedbackrouters) behind a Next.js + Tailwind CSS chat UI - Monitoring → query logs and feedback land in PostgreSQL, visualized in a 10-panel Grafana dashboard (including LLM cost panels)
- Infrastructure → Docker Compose (pgvector, Grafana, API)
Key Technical Achievements
Hybrid retrieval that meaningfully outperforms semantic-only search — retrieval_eval.py measures Hit@1/3/5 at 50% for semantic-only retrieval versus 75% for hybrid (vector + BM25 RRF), at the cost of extra latency (565ms → 776ms average).
Keyword-matching bug fix that unlocked hybrid search — extract_keywords() was leaving punctuation attached to tokens (e.g. “PM2.5” stayed pm2.5), which never matched the undotted pm25 stored in chunk content, starving BM25 of real matches and letting near-universal tokens like the bare year dominate ranking. Stripping common English/Spanish punctuation (including ¿/¡) before token matching took hybrid Hit@1/3/5 from 50% to 75%.
LLM-as-judge evaluation — llm_eval.py scores generated answers with gpt-4o-mini as judge, averaging 4.0/5 overall, 4.0/5 relevance, and 4.8/5 accuracy across the evaluation set.
Bilingual query handling — incoming queries are language-detected and the system prompt forces the LLM to respond in the same language, so English and Spanish questions both get natively-phrased answers instead of a mismatched translation.
Clean upstream/downstream separation — this repo never runs dlt or dbt itself; it strictly reads BigQuery marts populated by the ETL pipeline, keeping ingestion/transformation and RAG serving as independently deployable systems that only share a BigQuery dataset as contract.