Guardrails Bound the Damage, They Do Not Close the Capability Gap

I built this system believing that careful orchestration could carry a weaker model. Then I benchmarked it. With the pipeline held fixed and only the model swapped, the three local models took over 180 seconds per query, failed on more than 80 percent of them, and answered correctly somewhere between 6 and 13 percent of the time. A stronger hosted model, given the same schema grounding, the same validation and the same retry logic, came back in about 45 seconds with no errors and 87.5 percent accuracy.

The database holds tool condition monitoring data at TUM's Institute for Machine Tools and Industrial Management: sensor traces, wear measurements and process parameters from milling experiments. Engineers who knew exactly what they wanted still had to know the schema and write SQL to get it. What separates this from a demo is that the data sits next to production, where a confident wrong answer is worse than no answer and an unsafe write is not fixed by apologizing.

So the model never simply writes a query and runs it. The pipeline breaks into stages: decide whether the question concerns the database at all, fold in conversation context, pick candidate tables by relevance to the schema, generate SQL against only those tables, validate it, execute it read-only, retry within a fixed budget on recoverable errors, then summarize. Not for elegance. A single opaque call fails as one undifferentiated event, while stages let you point at the transition that broke and log it there. Nearly everything I learned about this system came from being able to ask which stage.

Grounding did most of the work against hallucination by shrinking the space in which the model could be wrong. We extracted the schema, tables and columns and foreign keys, in both machine-readable and human-readable form, then used it to narrow the prompt, hint at joins and allowlist what a query was permitted to touch. Invented columns and impossible joins dropped sharply. The cost is a dependency: answer quality now tracks metadata quality, and when the schema changes the grounding has to be regenerated or the system quietly starts reasoning about a database that no longer exists.

The safety layer taught me a distinction I use constantly now. An instruction is not a guarantee. Asking a model for read-only queries is a request it can fail under unusual phrasing. Validating the statement, rejecting writes and suspicious patterns, allowlisting schema elements and running inside a read-only transaction with a timeout are enforcement, and they hold whatever the model emits.

Which is why the benchmark result is clarifying rather than embarrassing. The safeguards decided whether failures were safe and legible. The model decided whether there were failures. I had been treating those as one property. A system can be well built, in the sense that nothing dangerous executes and every error is attributable, and still be unusable because the model underneath cannot write correct SQL fast enough.

The numbers deserve their caveats. Accuracy came from a modest set of representative queries with some manual review, so 87.5 percent is a model-selection signal inside this project rather than a figure worth quoting elsewhere, and retrieval was structured schema selection rather than vector search, which bounds what the system can answer at all. What stayed with me is the habit of building the harness that can tell me whether I am looking at the model or at my own code. I needed that same separation later, when the question was whether a benchmark score moved because of the model or because of the judge.