AI Usage & Run Log

The AI Usage Console

AI Usage answers the questions an administrator actually asks about AI: what it is costing, which feature spends it, how often it fails, and what a particular call came back with. It reads the AI run log, where every call the platform makes to a provider is one row, and it is where you send a test call through any role.

Where to find it

Architect Panel → Activity:

• AI Usage — usage by period, Test a role, recent calls and what the log keeps
• AI Run Log — the raw rows behind it, one per call

Architect Panel → Configuration:

• AI Settings — where the logging and retention settings it reports are changed

What is on the screen

From the top: a Reporting to ActiveManage card (only on an installation that reports AI spend to ActiveManage), then four cards. On a multi-tenant installation the figures are for the tenant you are signed in to. The screen is for architects only and never shows a key.

Usage for a period

Choose From, To (inclusive) and Group by (Feature, Model, Service or Day), then Show. The default is the last 30 days by feature.

• Four tiles: AI calls; Not usable (an error, an answer that failed its checks, or a refusal, each of them money spent on nothing); tokens of every kind; and the List-price estimate.
• A bar per day, with the unusable part in red. Hover for the day's figures.
• A table: Calls, OK, Not usable, Input, Output, Cache write, Cache read, Cost, Rated and Avg rating for each feature, model, service or day. Every row opens the list of exactly its own calls.

Cost is an estimate from list prices, not an invoice. A figure shown with ≥ includes calls on models that could not be priced, so the real total is higher. estimated marks calls whose token counts were estimated from characters because a stream stopped before the provider reported them. A mixed currency badge means the currency label was changed during the period.

Test a role

Every role the installation configures, what it resolves to, whether its provider is configured, and a Test button. See AI Usage & Run Log for the test call.

Recent AI calls

Filter by Feature, Model, Outcome (ok, error, invalid answer, refused) and Service, then Filter. Fifty calls a page, with Newer and Older. Each row shows when, the feature, the role, the service and model, the outcome, tokens in and out, cost, how long it took and any rating; Open shows the run in full.

What the log keeps

Whether runs are logged, whether provider responses and prompts are kept, the retention periods, and whether the housekeeping task that applies them is on, disabled, in preview, or not installed.

Reading one run

Open shows Run # and its number, with: when, status and error, feature, role, service and endpoint, model, stop reason, tokens, cost, duration, the provider's request id, the request's size, what record it was about, the background job if there was one, who it was made for, and any rating. Below that, What the caller was handed shows the answer and the warnings, and What went over the wire shows the provider's response and the request, but only where the log settings kept them.

A run may say its text is not there for a reason other than an empty answer: the retention sweep blanked it, it was too large to keep, or the feature logged it without bodies on purpose. Scanned documents read by AI are always logged that way, because the transcription already lives with the document under its own access rules.

Using it

1. Open AI Usage and read the four tiles for the last 30 days.
2. If Not usable is high, group by Feature to see which feature fails, then open a failing row's calls.
3. Open one failing run and read its error and warnings.
4. If the cause looks like configuration, change it on AI Settings and use Test on the role.
5. Group by Model or Day to watch cost after a change.

What goes wrong, and how to tell

• "Run logging is switched off": no calls are being recorded, the figures stop at the moment it was turned off, and background AI calls cannot hand their answers back. Turn Log every call back on in AI Settings.
• "Nothing to list: .airuns does not exist": the run log has not been installed on this database. The test call still works; ask your hosting administrator to apply the platform update.
• Cost shows not priced: the model is not in the price table, or the provider reports no token counts. Add a price override if you need an estimate.
• A run says "There is no AI run #...": it does not exist in this installation's log, or it belongs to another tenant.

Worked example

An architect notices the List-price estimate has doubled month on month. Grouping by Feature shows the ERP Feed Queue's suggestions have tripled; grouping by Day shows the rise started on the day a large bank statement was imported. Opening a few runs shows each is a normal, successful suggestion. Nothing is broken: the team simply used Suggest all on many more rows, and the finance manager is shown the figure.

Recommendations

• Look at Not usable weekly; every unusable call was still paid for.
• Group by feature before by model: features are what people can change.
• Treat cost as an estimate and reconcile with the provider's own bill monthly.
• Keep run logging on, and decide deliberately whether prompts are kept.

Testing a Role

The test call sends one tiny, fixed prompt through a role's real configuration and shows exactly what came back. It is the deliberate first live call on an installation: the way to prove that a key, a model id, an AWS permission or a Region works before a user finds out that it does not.

Where to find it

Architect Panel → Activity:

• AI Usage — the Test a role card

Architect Panel → Configuration:

• AI Settings — where you fix what a failed test points at

What it sends, and what it costs

Test asks the model to reply with the single word OK, allows only a short answer (longer for an OpenAI model on Bedrock, whose reasoning counts against the limit) and makes no retries. It uses the role's own provider, model, effort and fallback model, so it exercises what the role's real callers get.

It is a real call and it is billed, though the amount is tiny. It is logged like any other call, under the feature admin.testcall and against the architect who pressed the button. It can only be made from the screen itself: a link cannot trigger it.

The roles listed

The card lists builder and every role the AI configuration names a provider entry for, with the service and model it resolves to, inherits builder where it has no provider of its own, and configured or not configured for its provider. A role whose provider is not configured can still be tested: it fails with not_configured before anything leaves the server.

Running a first live check

1. On AI Settings, enter the provider's key and a default model, and point the roles you need at it.
2. Open AI Usage. Every role you intend to use should say configured.
3. Press Test beside builder, then beside each role with its own provider or model.
4. For each, check the green panel ("The ... role answered.") and that Endpoint and Model are the ones you expected.
5. Open the run from the Run line and confirm it was logged with its cost.

Reading the result

Under the panel is a table: Service, Endpoint, Model (the one that actually answered), Provider request id (what the provider's support can look a call up by), Duration, Tokens, Cost, Stop reason, Attempts and Run. A failure adds Retryable and the HTTP status, and the red panel names the error code, the provider's own message and a hint.

What each error means

• not_configured: the role's provider has no usable credentials.
• auth: the provider, or AWS, refused the key.
• permission: the key is valid but not allowed to use this model. On Bedrock the hint lists the AWS permissions needed.
• not_found: the model id is unknown to the provider, or on Bedrock not enabled in this account and Region.
• invalid_request or bad_request: the provider rejected the request itself, for example a Bedrock geographic profile outside your Region, or an effort level the model does not take. The message is the provider's own wording.
• rate_limited or overloaded: the provider is busy or the account is over its limit. Nothing is wrong with the configuration; try again shortly.
• timeout or connection: nothing usable came back in time. Check the server can reach the provider: outbound HTTPS, proxy, DNS.
• budget_exceeded: nothing was sent, because an AI spend limit refused the call. The message says which limit.
• sdk_missing: a software component is missing on the server; your hosting administrator needs to complete the platform install.

What it does not prove

• Structured answers. It sends no schema, so it cannot show that a feature's structured answers will work.
• Streaming. It streams only when the role's effort is xhigh or max. A missing Bedrock streaming permission shows up as a warning on the first real long answer instead.
• Tools and documents. It sends no document and runs no tools.

Worked example

After moving an installation to Amazon Bedrock, an architect tests builder and gets permission. The hint says the AWS user needs permission to invoke the model in this Region. The cloud team adds it, the architect presses Test again, and the role answers with the expected endpoint and model. They copy the provider request id into the change ticket as evidence, and test the other roles before telling users AI is back.

Recommendations

• Test after every change to keys, models, permissions or Region.
• Test every role that has its own provider or model, not only builder.
• Keep the provider request id of a failing call for the provider's support.
• Do not treat a passing test as proof of everything: watch the first real runs on AI Usage too.

The AI Run Log and Retention

Every call the platform makes to an AI provider writes one row to the AI run log: which feature asked, which model answered, how many tokens it used, what it cost at list price, how long it took and whether it worked. The log is what AI Usage reads and what AI credit is charged from, and three settings decide how much of each call's content it keeps.

Where to find it

Architect Panel → Activity:

• AI Run Log — the raw rows, newest first, read-only
• AI Usage — the same log summarised, with a readable view of each run

Architect Panel → Configuration:

• AI Settings — the section The run log

Architect Panel → Automation:

• Tasks — the AI Run Log Housekeeping task, which applies the retention settings

One row per call

Retries, repairs and a fallback to a second model are added into the same row, so the count of rows is the count of calls. Status is ok, error, invalid (the answer failed its checks) or refused (the model declined). Each row also records the Requester the call was made for, and, where a feature says so, the Subject datastore and Subject row it was about. The log is written by the platform; nobody edits it, and Super Administrators get read access only.

What is kept

• Always: feature, role, service, model, status, error, token counts, estimated cost, duration, the provider's request id, and a fingerprint of the request that lets identical calls be matched without keeping either.
• Output: what the feature was handed back. Kept unless the feature opts out.
• Responses: the provider's raw answers, kept while Keep the response is on (the default). They are the only record of what a model actually said.
• Request: the prompt, including any document text, kept only while Keep the request is on. It is off by default, because prompts are personal data held outside the access rules of the records they came from.

Keys never reach the log. A body larger than about 2 MB is replaced by a note of its size. Scans read by AI keep no text in the log at all, only their cost.

The run log settings

• Log every call: on by default. Off means no row per call, no figures on AI Usage, and background AI calls with no way to hand their answer back. While AI billing is on it is forced on.
• Keep the request: off by default.
• Keep the response: on by default.
• Blank the bodies after (days): default 90. After this, a run's request, response and output are blanked and the usage row is kept, so cost history survives. 0 keeps the bodies.
• Delete runs after (days): default 0, which keeps runs for ever. Otherwise rows are deleted outright after this many days.

An answer a background job has not yet collected is never blanked or deleted until that job expires.

Switching retention on

The two retention figures do nothing until the AI Run Log Housekeeping task runs, and it ships disabled and in preview mode. In preview it only reports what it would blank and delete, even when scheduled.

1. Set the two figures on AI Settings and Save.
2. In Tasks, find AI Run Log Housekeeping and switch it on. It runs daily.
3. Use the row's Preview action and read what it would blank, delete and spare.
4. When the numbers are what you expect, take the task out of preview mode on its row.
5. Check What the log keeps on AI Usage: the warning that "these numbers currently mean nothing" should be gone.

The same task also clears the stored request of finished background AI jobs that no worker tidied up.

What goes wrong, and how to tell

• AI Usage says the housekeeping task is disabled, in preview mode, or not installed: nothing is being removed yet, whatever the settings say.
• A run on AI Usage says "The answer was removed by the retention sweep": its bodies were blanked by retention. The usage and status remain.
• The log grows faster than expected: Keep the request is on and features are sending documents. Turn it off unless you are diagnosing a problem.
• A background AI feature finishes but shows nothing: Log every call is off. Turn it back on.

Worked example

A data protection review asks how long AI prompts and answers are kept. The architect confirms Keep the request is off, sets Blank the bodies after (days) to 30 and leaves Delete runs after (days) at 0, so cost history stays. They enable the housekeeping task, preview it, see it would blank several thousand older runs and delete none, and take it out of preview. The review is answered: prompts are not stored, answers are kept for 30 days, and usage figures are kept indefinitely.

Recommendations

• Leave Keep the request off except while diagnosing a fault, then switch it off again.
• Blank bodies, keep rows: cost and failure history are worth keeping; content rarely is.
• Preview the housekeeping task before trusting it, then take it out of preview.
• Never switch run logging off on an installation that uses background AI work or AI billing.

Background AI Work

Some AI work takes minutes rather than seconds: generating a video, or a long check that a custom app runs over a set of documents. Rather than hold a person's browser open, the platform can run that work in the background and let the screen collect the answer when it is ready. That depends on one task being switched on, and on the AI run log.

Where to find it

Architect Panel → Automation:

• Tasks — the Background Jobs task, the worker that runs queued work

Architect Panel → Data:

• Background Jobs — every queued job, its status and how it ended

Architect Panel → Activity:

• AI Usage — each background call's run, marked with its background job

What runs in the background

• Video generation. A video takes from seconds to several minutes at Google, and something has to keep asking for it and then save it.
• AI calls a custom app queues. An app's own feature can hand an AI call to the worker and be told when it has finished.
• Calls finished after the page has its answer. A feature can also start a call, answer the browser straight away, and finish the call in the same server process. That kind needs no worker.

No standard platform screen queues AI calls of its own today; the sign-up wizard, AI Builder, App Code and Report Builder assistant all answer while the person waits.

Switching the worker on

1. Open Tasks and find Background Jobs. It ships disabled.
2. Switch it on. It runs every minute, picking up queued work within a tick or two.
3. Check the task engine itself runs on its schedule: the Tasks screen's Overview shows when it last ran.
4. Make sure Log every call is on in AI Settings → The run log. The run log is how a background answer comes back: with logging off, a background AI call finishes with nothing to hand back, and without the run log installed at all it refuses to queue.

How a person sees progress

The screen that started the work asks for its status every few seconds. A job is queued, running, ready, failed or cancelled, and can carry a short message. When it is ready, the screen collects the answer. A person can only ever see their own jobs; anyone else's answers exactly as if it did not exist.

A provider error is a finished job, not a failed one: the answer that comes back says what went wrong. A job fails only when the call could not be made at all.

Cancelling

Cancelling stops a queued job before it runs. A call already running stops within seconds if it is streaming; otherwise it finishes and is paid for. Either way a paid-for answer is attached to the cancelled job and can still be collected.

What is kept, and for how long

• The job row expires after a week. The answer outlives it in the AI run log, where a feature can still collect it by its run number.
• The request (prompts, document references) is cleared from the job as soon as the job is decided. Whatever the run log keeps is governed by its own settings.
• Permissions: a background call reads stored documents with the security groups the person had when they started it, which matters only if access is removed in the minutes before it runs.

What goes wrong, and how to tell

• Jobs sit at queued for ever: the Background Jobs task is disabled, or the task engine is not running.
• A video started from code or an AI tool is refused with "the background worker is not running": the worker is not on, and the platform will not start a paid video that nothing would collect. Switch on Background Jobs.
• A video started from a browser or MCP client says to keep the window open, or to keep checking: the worker is off, so only that person's polling will save the video. If they stop, it is still charged.
• A job says ready but there is no answer: run logging was off, or the run could not be written. Turn Log every call on.

Worked example

A team's custom app checks each submission pack against a checklist, which takes two or three minutes a time. The architect switches on Background Jobs and confirms the task engine ran within the last minute. A caseworker presses Check, sees "Queued", then "Checking the submission...", and two minutes later the findings appear on the record. On AI Usage the run shows its cost and the background job it belonged to, and the job row disappears a week later while the run stays.

Recommendations

• Switch on Background Jobs on any installation that generates video or runs long AI work.
• Keep run logging on; background answers depend on it.
• Watch the Background Jobs datastore for jobs that stay queued.
• Tell people that cancelling a running call may still cost money, and that its answer can still be collected.