Loading

Testing a Role

The test call sends one tiny, fixed prompt through a role's real configuration and shows exactly what came back. It is the deliberate first live call on an installation: the way to prove that a key, a model id, an AWS permission or a Region works before a user finds out that it does not.

Where to find it

Architect Panel → Activity:

  • AI Usage — the Test a role card

Architect Panel → Configuration:

  • AI Settings — where you fix what a failed test points at

What it sends, and what it costs

Test asks the model to reply with the single word OK, allows only a short answer (longer for an OpenAI model on Bedrock, whose reasoning counts against the limit) and makes no retries. It uses the role's own provider, model, effort and fallback model, so it exercises what the role's real callers get.

It is a real call and it is billed, though the amount is tiny. It is logged like any other call, under the feature admin.testcall and against the architect who pressed the button. It can only be made from the screen itself: a link cannot trigger it.

The roles listed

The card lists builder and every role the AI configuration names a provider entry for, with the service and model it resolves to, inherits builder where it has no provider of its own, and configured or not configured for its provider. A role whose provider is not configured can still be tested: it fails with not_configured before anything leaves the server.

Running a first live check

  1. On AI Settings, enter the provider's key and a default model, and point the roles you need at it.
  2. Open AI Usage. Every role you intend to use should say configured.
  3. Press Test beside builder, then beside each role with its own provider or model.
  4. For each, check the green panel ("The ... role answered.") and that Endpoint and Model are the ones you expected.
  5. Open the run from the Run line and confirm it was logged with its cost.

Reading the result

Under the panel is a table: Service, Endpoint, Model (the one that actually answered), Provider request id (what the provider's support can look a call up by), Duration, Tokens, Cost, Stop reason, Attempts and Run. A failure adds Retryable and the HTTP status, and the red panel names the error code, the provider's own message and a hint.

What each error means

  • not_configured: the role's provider has no usable credentials.
  • auth: the provider, or AWS, refused the key.
  • permission: the key is valid but not allowed to use this model. On Bedrock the hint lists the AWS permissions needed.
  • not_found: the model id is unknown to the provider, or on Bedrock not enabled in this account and Region.
  • invalid_request or bad_request: the provider rejected the request itself, for example a Bedrock geographic profile outside your Region, or an effort level the model does not take. The message is the provider's own wording.
  • rate_limited or overloaded: the provider is busy or the account is over its limit. Nothing is wrong with the configuration; try again shortly.
  • timeout or connection: nothing usable came back in time. Check the server can reach the provider: outbound HTTPS, proxy, DNS.
  • budget_exceeded: nothing was sent, because an AI spend limit refused the call. The message says which limit.
  • sdk_missing: a software component is missing on the server; your hosting administrator needs to complete the platform install.

What it does not prove

  • Structured answers. It sends no schema, so it cannot show that a feature's structured answers will work.
  • Streaming. It streams only when the role's effort is xhigh or max. A missing Bedrock streaming permission shows up as a warning on the first real long answer instead.
  • Tools and documents. It sends no document and runs no tools.

Worked example

After moving an installation to Amazon Bedrock, an architect tests builder and gets permission. The hint says the AWS user needs permission to invoke the model in this Region. The cloud team adds it, the architect presses Test again, and the role answers with the expected endpoint and model. They copy the provider request id into the change ticket as evidence, and test the other roles before telling users AI is back.

Recommendations

  • Test after every change to keys, models, permissions or Region.
  • Test every role that has its own provider or model, not only builder.
  • Keep the provider request id of a failing call for the provider's support.
  • Do not treat a passing test as proof of everything: watch the first real runs on AI Usage too.