AI Inference
Run AI inference through the Strongly AI Gateway. All inference endpoints proxy to the configured AI Gateway backend and support both synchronous and streaming responses. These endpoints are OpenAI-compatible, allowing drop-in replacement for existing OpenAI SDK integrations.
Responses from these endpoints are the AI Gateway's own response body and status, passed through as is: they are not wrapped in the standard data / meta envelope. The X-Request-Id header is still set.
POST /api/v1/ai-gateway/chat/completions
Create a chat completion
Generates a model response for the given conversation. Compatible with the OpenAI Chat Completions API format.
Scope: ai-gateway:inference
Headers Forwarded:
| Header | Description |
|---|---|
X-User-Id | Authenticated user ID |
X-Request-Id | Request trace ID |
X-Organization-ID | Organization context for multi-tenancy |
Authorization | Bearer token passed to the upstream provider |
Request Body:
{
"model": "gpt-4",
"messages": [
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user", "content": "Explain Kubernetes in one sentence." }
],
"stream": false,
"max_tokens": 1024,
"temperature": 0.7,
"top_p": 1.0,
"stop": ["\n"]
}
| Field | Type | Required | Description |
|---|---|---|---|
model | string | Yes | Model ID or name registered in the AI Gateway |
messages | array | Yes | Conversation messages array |
messages[].role | string | Yes | Message role: system, user, or assistant |
messages[].content | string | Yes | Message content |
stream | boolean | No | Enable SSE streaming (default: false) |
max_tokens | integer | No | Maximum tokens to generate |
temperature | number | No | Sampling temperature (0.0 - 2.0) |
top_p | number | No | Nucleus sampling threshold (0.0 - 1.0) |
stop | string or string[] | No | Stop sequence(s) |
web_search_options | object | No | Search the web before answering. {} for the defaults; max_uses caps the searches. OpenAI, Anthropic and Gemini models use their vendor's own search; other models use Strongly's search when they can call tools. See Web Search. |
Response: 200 OK
{
"id": "chatcmpl-abc123",
"model": "gpt-4",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Kubernetes is an open-source container orchestration platform."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 25,
"completion_tokens": 12,
"total_tokens": 37
},
"created": 1706000000
}
POST /api/v1/ai-gateway/completions
Create a text completion
Generates a completion for the given prompt. Useful for non-conversational text generation tasks.
Scope: ai-gateway:inference
Headers Forwarded:
| Header | Description |
|---|---|
X-User-Id | Authenticated user ID |
X-Request-Id | Request trace ID |
X-Organization-ID | Organization context for multi-tenancy |
Authorization | Bearer token passed to the upstream provider |
Request Body:
{
"model": "gpt-6-luna",
"prompt": "Write a SQL query that selects all users where",
"stream": false,
"max_tokens": 256,
"temperature": 0.5
}
| Field | Type | Required | Description |
|---|---|---|---|
model | string | Yes | Model ID or name registered in the AI Gateway |
prompt | string | Yes | Text prompt to complete |
stream | boolean | No | Enable SSE streaming (default: false) |
max_tokens | integer | No | Maximum tokens to generate |
temperature | number | No | Sampling temperature (0.0 - 2.0) |
Response: 200 OK
{
"id": "cmpl-abc123",
"model": "gpt-6-luna",
"choices": [
{
"text": " status = 'active' ORDER BY created_at DESC;",
"index": 0,
"finish_reason": "stop",
"logprobs": null
}
],
"usage": {
"prompt_tokens": 12,
"completion_tokens": 14,
"total_tokens": 26
},
"created": 1706000000
}
POST /api/v1/ai-gateway/embeddings
Generate embeddings
Creates an embedding vector for the given input text. Supports single strings or batches of strings.
Scope: ai-gateway:inference
Request Body:
{
"model": "text-embedding-ada-002",
"input": "Kubernetes pod scheduling explained"
}
Or batch input:
{
"model": "text-embedding-ada-002",
"input": [
"Kubernetes pod scheduling explained",
"Docker container networking basics"
]
}
| Field | Type | Required | Description |
|---|---|---|---|
model | string | Yes | Embedding model ID or name registered in the AI Gateway |
input | string or string[] | Yes | Text to embed (single string or array of strings) |
Response: 200 OK
{
"data": [
{
"embedding": [0.0023064255, -0.009327292, 0.015797347, "..."],
"index": 0
}
],
"model": "text-embedding-ada-002",
"usage": {
"prompt_tokens": 6,
"total_tokens": 6
}
}
POST /api/v1/ai-gateway/tokenize
Count a prompt's tokens with a model's own tokenizer, and give that model's context length.
Self-hosted (vLLM-served) models only; any other model is refused with 400.
Scope: ai-gateway:inference
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
model | string | Yes | The model |
prompt | string | Yes | The text to count |
Response 200 OK: the model server's answer, as it gives it: count, the tokens and
max_model_len when the server returns them.
GET /api/v1/ai-gateway/audio/speech/voices
The text-to-speech voices you can give POST /api/v1/ai-gateway/audio/speech, each with its
id, name, provider and gender.
Scope: ai-gateway:inference
| Parameter | Type | Required | Description |
|---|---|---|---|
provider | string | No | Only this provider's voices |
Response 200 OK
POST /api/v1/ai-gateway/generations
Start generating an image, a video or music with a generation model. The generation runs as a job: the response gives its id to follow.
Scope: ai-gateway:inference
Request Body
| Field | Type | Required | Description |
|---|---|---|---|
model | string | Yes | A generation model |
prompt | string | Yes | What to generate |
type | string | No | image, video or music (default from the model) |
Response 200 OK: the generation job, with its id and status.
DELETE /api/v1/ai-gateway/generations/:id
Cancel a generation job.
Scope: ai-gateway:inference
| Parameter | Type | Required | Description |
|---|---|---|---|
id | string | Yes | The generation job |
Response 200 OK
SSE Streaming Format
When stream is set to true on the chat completions or text completions endpoints, the response uses Server-Sent Events (SSE) instead of returning a single JSON object.
Response Headers:
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
Stream Chunks:
Each chunk is delivered as a data: event followed by a JSON object and two newlines:
data: {"id":"chatcmpl-abc123","model":"gpt-4","choices":[{"index":0,"delta":{"role":"assistant","content":"Kubernetes"},"finish_reason":null}],"created":1706000000}
data: {"id":"chatcmpl-abc123","model":"gpt-4","choices":[{"index":0,"delta":{"content":" is"},"finish_reason":null}],"created":1706000000}
data: {"id":"chatcmpl-abc123","model":"gpt-4","choices":[{"index":0,"delta":{"content":" an"},"finish_reason":null}],"created":1706000000}
data: {"id":"chatcmpl-abc123","model":"gpt-4","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"created":1706000000}
data: [DONE]
| Field | Description |
|---|---|
delta.role | Present only in the first chunk |
delta.content | Token content (may be empty in the final chunk) |
finish_reason | null during streaming, stop or length on the final content chunk |
[DONE] | Signals the end of the stream |
The chunks above are from chat completions. Text completion chunks carry the generated text in each choice's text field instead of delta.
When consuming the stream, concatenate all delta.content values to reconstruct the full response. The usage field is not included in streamed chunks.