Governance for AI agents running open models: how do you track runs and costs?

When agents run on open models via local runtimes, governance often gets ignored: who ran what, on which model, and at what token cost.

I’m working on a setup (Sharkly.ai, disclosure: I work on it) where every agent run logs its model, trigger source, responsible person and token usage.

Questions:

  • Do you log the model and version per agent run for reproducibility?
  • Are there open tools or standards for agent audit logs that you’d recommend?

Yes. For open models, “model and version” has to mean more than a name like llama3:8b, because local tags can be repointed and one name can cover several quantizations. What I’d record per run:

  • weights identity: Hub repo id plus commit revision, and the sha256 of the file actually loaded, including the quantization
  • runtime and version (llama.cpp build, vLLM, Ollama) and the sampling parameters
  • a version or hash of the system prompt, agent config and tool definitions
  • one entry per tool call: tool name, arguments, result status, and a call/request ID that the downstream system logs as well, so the two sides can be joined
  • the trigger, any human approval (approver plus record ID), and token counts

On standards: the OpenTelemetry GenAI semantic conventions define agent and tool-execution spans and attributes such as gen_ai.request.model, gen_ai.response.model, gen_ai.tool.call.id and token usage. They’re still marked Development, so pin the version you emit. For audit purposes, write the log to append-only storage that the agent’s own credentials can’t modify or delete.

One useful addition for local models is to separate the audit log from the cost ledger. Token counts alone can hide the real cost, because a run that retries, loops, or calls tools repeatedly can burn a lot of GPU time even when each step looks cheap. I would record, per run, the start and end time, the number of model calls, the number of tool calls, and whether the task finished, failed, or was stopped by a human. Then review cost per successful task, not just cost per run, so a task that needed three retries stands out. A simple daily cap on model calls per agent, plus an alert when a single run crosses a threshold, usually catches runaway loops earlier than a monthly invoice review does.

i’d log the trigger, the model and version (the actual file hash, not just the tag), every tool call with its arguments and result, and who approved anything that writes to a real system. the approval part ends up being the most useful, more than tokens, because when something odd happens you want to know which person said yes and which record it touched. also worth putting the agent’s call id into the downstream system’s own log so you can join the two later.

full disclosure i built truescope, a business app with a documented rest api, and that’s the pattern i’d use for an always on agent doing scheduled work like watching stock and drafting pos, with a person approving the important ones. might be worth a look Agentic ERP: AI Agents for Small Business Workflows | TrueScope

I agree that model identity and token counts are necessary, but they are not enough to explain a governed run.

For the work I’m doing around deterministic worker construction, I would keep a run receipt that separates at least four things: the model/runtime used; the authority and task package assembled before inference; the candidate output; and the independent decision to release, HOLD, deny, or escalate it.

The useful cost unit is not only tokens per call. It is also cost per completed, accepted task—plus the model calls, tool calls, retries, and failed or stopped attempts that led there. A cheap-looking run that loops three times or produces an output that cannot be released is not equivalent to one bounded call that completes under its contract.

I also think failed cases must remain visible. If the audit trail records only successful outputs, it cannot show whether the system actually controlled the failure path.

The practical question I’d add is: can a reviewer reconstruct not only what model ran, but why a proposed result was authorized to matter?