Applied AI / RAG Evaluation
CivicLens RAG — NYC 311 Operations Copilot
Hosted hybrid RAG system for grounded NYC 311 documentation Q&A with semantic and full-text retrieval, deterministic RRF, validated citations, explicit abstention handling, and bounded approved analytics.
Next.js is the recruiter-facing product UI. Streamlit remains a separate engineering, validation, and debugging client. CivicLens is not a live NYC municipal service.
Quick Scan
Project at a Glance
The product boundary, retrieval design, and measured scope in five facts.
Project type
Applied AI / Hybrid RAG
Status
Hosted Portfolio Demo
Retrieval
Semantic + Full-Text + RRF
Application
FastAPI + Next.js
Evaluation
24 questions / 14 retrieval-eligible
Problem / Solution / Outcome
Traceable answers over operational knowledge
CivicLens makes retrieval quality and answer grounding inspectable instead of treating RAG as a black box.
Problem
Operational documentation, runbooks, field definitions, and system guidance are difficult to search reliably while preserving source traceability.
Solution
Manifest-controlled ingestion and section-aware chunks feed PostgreSQL/pgvector, lexical retrieval, deterministic RRF, grounded generation, and application-owned citation validation.
Outcome
A hosted non-production RAG application with measurable retrieval quality, validated citations, bounded analytics, safe-abstention behavior, and deployment proof.
System Architecture
One product boundary, two deliberately separate answer paths
The browser UI delegates all AI behavior to FastAPI, where CivicLens routes to either hybrid document RAG or bounded approved analytics.
Hosted product path
Browser
Vercel Next.js
Render FastAPI
CivicLens Orchestration
Render FastAPI remains the AI application boundary. Hybrid RAG uses externally managed Neon PostgreSQL + pgvector; Next.js contains presentation and one typed API client, not RAG logic.
Branch A
Hybrid RAG
Manifest-Controlled Sources
Section-Aware Chunks
PostgreSQL Authority
pgvector Dense + PostgreSQL FTS
PostgreSQL Hydration + Corpus Check
Deterministic RRF
Optional Bounded Reranking
Grounded Generation + Citation Validation
Branch B
Bounded Approved Analytics
Typed Allowlisted Tool
Checked-In Sample CSV
Bounded Read-Only Result
This branch is not unrestricted text-to-SQL, an autonomous agent, or arbitrary tool execution. Streamlit consumes the same FastAPI contract as an engineering/debug UI.
Measured Retrieval Performance
Hybrid search improved coverage on a small curated benchmark
The approved real-local comparison evaluates three retrieval strategies without mixing those results with the hosted runtime profile.
83.9%
Recall@5
Hybrid retrieval
92.9%
Expected-source retrieval
Hybrid retrieval
14
Retrieval-eligible questions
24-question fixture
Real-local evaluation profile — not the hosted runtime
24-question curated fixture; 14 retrieval-eligible questions; sentence-transformers/all-MiniLM-L6-v2 at 384 dimensions; PostgreSQL/pgvector; PostgreSQL full-text retrieval; RRF_K=60; top_k=5; optional cached cross-encoder reranking.
Semantic
- Recall@5
- 0.6607
- MRR
- 0.5857
- Source
- 0.7857
Hybrid
- Recall@5
- 0.8393
- MRR
- 0.7071
- Source
- 0.9286
Hybrid + Reranking
- Recall@5
- 0.8214
- MRR
- 0.7619
- Source
- 0.9286
| Strategy | Recall@5 | MRR | Expected Source |
|---|---|---|---|
| Semantic | 0.6607 | 0.5857 | 0.7857 |
| Hybrid | 0.8393 | 0.7071 | 0.9286 |
| Hybrid + Reranking | 0.8214 | 0.7619 | 0.9286 |
Recall@5 and MRR use section-level relevance and macro averages over n=14. This is a small portfolio benchmark, not a production-scale or statistically significant result.
What the experiment showed
- Hybrid retrieval raised Recall@5 from 0.6607 to 0.8393 versus semantic-only retrieval.
- Expected-source retrieval improved from 0.7857 to 0.9286 with hybrid search.
- Reranking increased MRR from 0.7071 to 0.7619, but slightly reduced Recall@5 from 0.8393 to 0.8214.
- The more sophisticated strategy did not dominate every metric, so no configuration is presented as universally superior.
Evaluation boundary
The evaluator records retrieval configuration and denominators separately from routing, citations, and safe no-answer behavior. It does not use an LLM judge or tune retrieval to hide failed cases.
Failure Analysis
Known failure modes
The evaluation exposes the current decision boundary instead of presenting only successful examples.
95.83%
Routing accuracy
n = 24
92.86%
Citation presence
n = 14
92.86%
Citation validity
n = 14
33.33%
Safe no-answer accuracy
n = 6
Observed failures
Safe no-answer accuracy was 33.33% across six no-answer cases. Four questions expected to abstain — q015, q021, q022, and q024 — were answered. Adversarial q023 routed to approved analytics instead of document RAG and was counted as unsupported.
Improvement boundary
The next experiments should test stronger abstention thresholds, better unsupported-intent detection, and tighter analytics routing guards. These are documented directions, not completed improvements.
Controlled Outputs
Grounding safeguards and bounded decisions
Application-owned validation protects the document path, while a fixed tool registry constrains the analytics path.
Grounding & Citations
- Generation receives only the question and allowlisted retrieved evidence fields.
- Stable chunk IDs, not model-generated display numbers, define citation identity.
- CivicLens rejects citation IDs outside the retrieved result set and rebuilds trusted provenance itself.
- An answered response with zero valid citations is converted to safe no-answer behavior.
- Public API responses omit raw retrieval chunks, provider payloads, credentials, and backend configuration.
Four approved tools
Strict typed inputs resolve through an immutable allowlist and read only four hard-coded checked-in sample CSV files. There is no SQL selection, arbitrary file path, dynamic import, or unrestricted tool execution.
Technical Implementation
Applied AI components with explicit ownership boundaries
The implementation makes corpus identity, ranking behavior, grounding, and product delivery independently inspectable.
Traceable Ingestion
Manifest authorization, normalized SHA-256 hashes, provenance, and stable document identities make the curated corpus auditable.
Section-Aware Chunking
Markdown heading paths and section provenance stay attached to deterministic chunk IDs through storage and retrieval.
Hybrid Retrieval
Dense semantic candidates and PostgreSQL full-text candidates are fused with deterministic Reciprocal Rank Fusion.
PostgreSQL + pgvector
PostgreSQL owns canonical text, provenance, hashes, corpus identity, and lexical search; pgvector is the default dense store.
FastAPI Application Boundary
A provider-neutral contract validates requests, delegates orchestration, returns allowlisted fields, and sanitizes backend failures.
Next.js Product UI
The Vercel interface is a typed presentation client for FastAPI, with no retrieval, generation, database, or provider logic.
Reproducible delivery
Ordered migrations, rerun-safe bootstrap, Docker packaging, readiness checks, isolated CI paths, and recorded deployment evidence support reproducible review. Streamlit remains available as the engineering and debugging client.
Hosted Proof
A real, deliberately non-production product path
Dated validation demonstrates the browser-to-backend workflow without claiming production availability or municipal operations.
Vercel
Next.js Product UI
Recruiter-facing browser experience and typed FastAPI client.
Render
FastAPI Boundary
Validation, orchestration, retrieval, generation, and sanitized responses.
Neon
PostgreSQL + pgvector
Canonical corpus state, lexical retrieval, and hosted dense-vector storage.
Hosted validation completed
- Grounded documentation answer rendered with CivicLens-validated citations and provenance.
- Approved analytics result rendered from the deterministic allowlisted CSV-backed path.
- Unsupported question returned safe abstention with zero fabricated sources.
- Dated deployment checks recorded healthy FastAPI liveness and readiness responses.
Hosted runtime profile
Deterministic local-deterministic-1536 embeddings, hybrid retrieval, Neon PostgreSQL + pgvector, and ANSWER_PROVIDER=openai for grounded generation. CivicLens validates citations before the public response.
Review hosted screenshotsNon-Production Portfolio Demo
Render Free can cold-start. The demo has no SLA, production authentication, HA, or live NYC 311 operational claim.
Honest Scope
Limitations remain explicit
The case study separates measured portfolio evidence from production and generalization claims.
- The RAG corpus is small, curated, and version-controlled rather than connected to live NYC 311 ingestion.
- The 24-question portfolio fixture is too small to establish statistical significance or production answer quality.
- Analytics uses four fixed read-only tools over checked-in sample CSV outputs, not unrestricted production text-to-SQL.
- The hosted demo is non-production, may experience Render Free cold starts, and makes no continuous-availability or SLA claim.
- Production authentication, authorization, HA, autoscaling, disaster recovery, rate limiting, and monitoring are out of scope.
- Evaluation and hosted runtime configurations intentionally differ: the approved benchmark uses the real-local MiniLM profile, repository defaults keep OpenAI optional, and the hosted demo uses a separate deterministic embedding/OpenAI grounded-generation configuration; documented abstention and routing failures remain part of the evaluation record.
What This Demonstrates
Applied AI & Data Science signals
The project combines retrieval experimentation and failure analysis with grounded product and deployment boundaries.
Retrieval experiment design
Hybrid search and ranking
Recall@K and MRR evaluation
Failure analysis
Grounded generation
Citation validation
Safe abstention design
Reproducible evaluation
Production-minded API and deployment boundaries
Explore CivicLens
Try the hosted demo or inspect the evidence
Start with the product experience, then review the source, measured retrieval results, architecture boundaries, and dated deployment proof.