Using the LLM as a Query Planner

For my thesis work I built a natural-language interface over large, fragmented, structured geospatial datasets. The LLM in that system never writes the final query and never reads the data. It acts as a query planner. It decides where a question should go and what constraints apply, then hands off to engines that actually execute.

I went this way because I think the two usual approaches give the model the wrong job. Naive text-to-SQL hands the model a schema, asks it to write the full query and then hopes the result is syntactically valid and also what the user meant. Every join and every filter is generated text, so each one is a place to be quietly wrong.

Classic RAG retrieves a pile of rows or chunks, pastes them into the context window and asks the model to reason its way to an answer. That works on small tidy data. On datasets like mine it means paying tokens for a language model to do work a database engine does better and faster. The engine is also deterministic, which the model isn't.

How the planning works

Given a question, the model makes a structured routing decision. It picks the right data partition and works out which predicates can be pushed down to deterministic execution engines: PostGIS for the spatial side, Neo4j for the graph side. Those engines do the filtering. Unnecessary scans and filters never run. The model also never has to wade through raw data it didn't need.

To check that this held up outside my own head, I built four retrieval architectures and benchmarked them end to end: vector RAG, schema-routed, graph-indexed and text-to-SQL. Across those paradigms the design cut inference token cost by 3x. I'm not surprised by the size of that number. A routing decision doesn't need a context window stuffed with rows.

Accuracy came down to the routing index

Routing-index quality was the primary driver of answer accuracy. Once the model's job shrinks to sending a question to the right place with the right constraints, the open question stops being whether the model reasons well. It becomes whether the routing decision was correct. That is a smaller thing to get right and a far easier one to check. A routing decision can be inspected and tested. When one fails you can usually see why.

I think this also explains why the approach held up across very different retrieval paradigms. The model's role is the same whether the target is a vector index, a schema, a graph or a SQL engine.

The hard engineering problem in this system was the index and routing layer underneath the LLM, the part that tells the model what exists and where it lives. On the next system like this, that layer is where I'd spend the engineering time first.