Calling models

Calling a model

Every action's context has ai. ai.generateText takes a prompt (with an optional system) or messages, and answers with the model's text.

import { applyParams, save, type ActionRun } from "@inferencebureau/apps-sdk";

/** Flags a review for a person to read when a model thinks it's abusive. */
export const run: ActionRun = async ({ params, record, ai }) => {
  applyParams(params, record);
  const { text } = await ai.generateText({
    model: "openai/gpt-5-mini",
    system: 'Answer {"abusive": true} or {"abusive": false}.',
    prompt: String(record.body),
    json: true,
  });
  record.status = JSON.parse(text).abusive ? "flagged" : "pending";
  await save(record);
};

The answer also has model (the one that answered, which differs when a Directives fallback was used), finishReason, usage (inputTokens, outputTokens) and requestId. A refusal (a model your Directives don't allow, a spent budget, a low balance) throws an ApiError with the gateway's message.

With the AI SDK

The client is also the settings for @inferencebureau/ai-sdk-provider: createInferenceBureau(ai) gives the AI SDK's generateText, streamText, structured output and tools, on chat completions or Anthropic's Messages (ib.messages(…)). Install ai and @inferencebureau/ai-sdk-provider in the app.

import { createInferenceBureau } from "@inferencebureau/ai-sdk-provider";
import { generateText } from "ai";
import type { ActionRun } from "@inferencebureau/apps-sdk";

export const run: ActionRun = async ({ params, ai }) => {
  const ib = createInferenceBureau(ai); // the action's ai is the provider's settings
  const { text } = await generateText({
    model: ib("anthropic/claude-sonnet-5"),
    prompt: `Write a two-line description of ${String(params.title)}.`,
  });
  return { description: text };
};

Any client built on the OpenAI SDK works the same way, with ai.baseURL as its base URL, ai.apiKey as its key and ai.headers sent along: createOpenAI({ baseURL: ai.baseURL, apiKey: ai.apiKey, headers: ai.headers }).

Outside actions

In the app's routes and pages' server code, createAiClient() makes the same client from the environment's variables, as createClient() does. Its calls are logged on their own rather than under a run.

// web/app/api/chat/route.ts
import { createAiClient } from "@inferencebureau/apps-sdk";
import { createInferenceBureau } from "@inferencebureau/ai-sdk-provider";
import { streamText } from "ai";

const ib = createInferenceBureau(createAiClient());

export async function POST(request: Request) {
  const { messages } = await request.json();
  const result = streamText({ model: ib("openai/gpt-5-mini"), messages });
  return result.toTextStreamResponse();
}

Embeddings

ai.embed(text or texts, { model? }) returns one vector per text, in order, with openai/text-embedding-3-small unless you name another. Vector fields with a source embed themselves; this is for anything else, such as a search by text the app builds itself.

const [vector] = await ai.embed("waterproof hiking boots");
const similar = await api.product.search("embedding", vector, { first: 5 });

Billing and Directives

• Calls made with the environment's token are requests on the team's Apps AI key, a key Inference Bureau makes in the team's default project the first time an app calls a model. They're charged at the gateway's price, like the app's embeddings, and counted as the app's under Plans and usage.
• The team's Directives, budgets and balance apply as they do to any key, and the requests appear in the project's Logs.
• Revoking the Apps AI key turns the client off for every app, with a 403; the app's embeddings are on a key of their own and carry on.
• A call made with an API key of your own (a script with createAiClient({ token: process.env.IB_API_KEY, app, environment })) is billed to that key instead.

Logs

Each call is an ai event in the app's Logs (type:ai), with the model, input and output tokens, and its estimated cost at the gateway's price. An action's calls sit under its run. The settled charge is on the request in the project's Logs, by the event's requestId.

Testing

In tests, the test app answers model calls itself. app.ai.on(match, answer) answers prompts containing match (or matching a RegExp) with answer, a string or a function of the call, through generateText and the AI SDK alike, streamed or not. app.ai.calls lists what was asked. ai.embed uses the test app's embed option.

import { expect, it } from "vitest";
import { loadApp } from "@inferencebureau/apps-sdk/test";

const app = await loadApp();

it("flags an abusive review", async () => {
  app.ai.on("idiot", '{"abusive": true}'); // prompts containing "idiot"
  const review = await app.run("review.create", { params: { body: "Only an idiot buys this", rating: 1 } });
  expect(review.record).toMatchObject({ status: "flagged" });
  expect(app.ai.calls[0]!.model).toBe("openai/gpt-5-mini");
});