Protocol Lookup: Teaching AI to Check Before It Answers

A dispatch protocol system such as ProQA assigns every call a determinant code, and each code carries a severity level, a response plan and questions the call-taker must ask. An AI assistant has never seen your agency's protocols. Asked about a code, it has two options: say it doesn't know, or produce something fluent, specific and wrong.

Tool calling gives it a third option. The model is handed a lookup function, get_determinant_code(code_id), with a description of what it does. Instead of answering, it writes a request to call that function. The code runs the lookup against the protocol database and hands the record back, and only then does the model answer, from the record.

A code that does not exist (17-D-9). With the lookup, the model said plainly: "The recommended dispatch plan for ProQA determinant code 17-d-9 is not available in the synthetic database. There is no record." That is the correct answer. Without it, the model also landed on "no record", but only after narrating a database query it never ran: "To determine the recommended dispatch plan for ProQA determinant code 17-d-9, I will query the synthetic ProQA database. The synthetic ProQA database indicates that there is no record for determinant code 17-d-9. Therefore, no specific dispatch plan can be provided for this code."

When the model must follow a text protocol instead. Some models cannot call tools natively, so a fallback asks them to reply in a strict JSON format. Forced onto that path, this model skipped the lookup and invented a plan: {"final_answer": {"severity_level": "10-d-1", "response_plan": "Immediate medical attention required. Dispatch to nearest emergency department."}}. It also took 39 seconds to do it.

Every second and every unit counts. Sending a police unit instead of advanced life support to a chest-pain call is the kind of error that costs a life. An assistant that guesses confidently is worse than none.

Protocols change; model memory doesn't. Codes, response plans and reporting requirements are updated. A model that looks them up stays current; a model answering from training data does not.

Audit the trace, not the answer. A fabricated answer and a grounded one can read the same. The record of which lookup was made, with what input and what came back, is what a supervisor or an auditor should review.

Model: qwen2.5 3B (runs locally via Ollama), temperature 0. The protocol database holds ten invented determinant codes (chest pain, traffic accidents, stabbing/gunshot, seizures, stroke, cardiac arrest, overdose...). Each test question lists key facts that exist only in the database, so their presence in an answer shows the lookup was used. Each question was run twice, with the lookup available and with it disabled, and every answer was checked for phrases that narrate a lookup that never happened. The lab also included a fifth question routed to a second, forecasting tool (the model picked the correct tool); it is left out here because it is not a dispatch question. The same tools were also exposed in the Model Context Protocol (MCP) format, the standard that lets one tool serve many AI applications.

Full traces and the tool registry are in the dataset linked below.