InferenceProxy

Location: iris/inference_proxy/ ⧉ Language: Go Role: User-facing API gateway for inference requests

InferenceProxy is the entry point for all user inference requests in the Iris system. It authenticates users, validates deployment ownership and health, publishes requests to RabbitMQ, and returns results from Redis in either synchronous or asynchronous modes.

Architecture Overview

User
 │  POST /v1/chat/completions
 │  Authorization: Bearer <token>
 │  X-Deployment-ID: <id>  (or use `model` field in body)
 │  X-Timeout-S: 30        (optional, 10–60s)
 │  X-Async: true          (optional)
 │  Body: OpenAI payload
 ▼
InferenceProxy
 ├─ Auth check        Redis auth:<sha256(token)>  → Supabase on miss
 │                    JWT: /auth/v1/user  |  API key (lk_*): Postgres
 ├─ Deployment check  1. read deploy:meta:<deployment_id>
 │                       Miss → acquire lock → GET /inference/get from Gateway
 │                            → write deploy:meta + deploy:healthy_replicas
 │                       Hit  → proceed to health check
 │                    2. read deploy:healthy_replicas:<deployment_id>
 │                       Miss (TTL expired) → acquire lock → refresh from Gateway
 │                    Verifies: deployment exists, owned by user
 ├─ Health check      count from deploy:healthy_replicas
 │                    count > 0 → proceed
 │                    count == 0 → 503 Service Unavailable
 ├─ Publish           RabbitMQ  infer_req.<deployment_id>
 │
 ├─ [SYNC]   poll Redis infer_resp:<request_id> with timeout
 │            → pass-through status_code | 408 on timeout | 500 on error
 │
 └─ [ASYNC]  write infer_resp:<request_id> = {"status":"pending"}
              → 202 Accepted { request_id, status: "pending" }

User (async follow-up)
 │  GET /v1/chat/completions/{request_id}
 ▼
InferenceProxy
 └─ read Redis infer_resp:<request_id>
         pending → 202 | completed → 200 | expired → 408 | absent → 404

Quick Start (OpenAI SDK)

InferenceProxy exposes an OpenAI-compatible API. Use the standard OpenAI Python SDK — no custom headers needed:

from openai import OpenAI

client = OpenAI(
    api_key="lk_...",                                      # from API Keys page
    base_url="https://prod.iris.lyceum.technology/v1",     # see Per-Environment URLs below
)

resp = client.chat.completions.create(
    model="<deployment_id>",   # used as deployment ID by the proxy
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Per-Environment URLs:

Environment	`base_url`
Development	`https://devel.iris.lyceum.technology/v1`
Integration	`https://integration.iris.lyceum.technology/v1`
Production	`https://prod.iris.lyceum.technology/v1`

Important: The base_url must point to the Iris proxy ({env}.iris.lyceum.technology), not the Gateway API (api.lyceum.technology). The Gateway is for deployment management (create/stop/list), not for inference requests.

API Reference

POST /v1/chat/completions

OpenAI-compatible chat completions endpoint.

Headers:

Header	Required	Description
`Authorization`	Yes	`Bearer <jwt>` or `Bearer lk_<64-hex>`
`X-Deployment-ID`	No	UUID of target deployment. Falls back to `model` field in request body if missing. At least one must be provided.
`X-Timeout-S`	No	Request timeout (10–60s, default 60)
`X-Async`	No	Set to `true` for async mode

Request Body: Standard OpenAI chat completions JSON. The model field serves double duty: if X-Deployment-ID is not set, model is used as the deployment ID. Before forwarding to the model container, the proxy overwrites model with the deployment's actual HuggingFace model ID (e.g., meta-llama/Llama-3.2-1B).

Success Response (Sync):

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1234567890,
  "model": "meta-llama/Llama-3.2-1B",
  "choices": [...]
}

Success Response (Async):

HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "request_id": "uuid",
  "status": "pending"
}

GET /v1/chat/completions/{request_id}

Check status of async request.

Response:

202 Accepted: Still processing
200 OK: Completed, return result
408 Timeout: Request expired
404 Not Found: Unknown request_id

Error Handling

Scenario	Status Code	Response
Missing/invalid `Authorization`	401	`{"detail": "Not authenticated"}`
Missing `X-Deployment-ID` and `model`	400	`{"detail": "X-Deployment-ID header or model field required"}`
Invalid request body	400	`{"detail": "Invalid request body"}`
`X-Timeout-S` out of range	400	`{"detail": "X-Timeout-S must be between 10 and 60 seconds"}`
Deployment not found	404	`{"detail": "Deployment not found"}`
Deployment not owned by user	404	`{"detail": "Deployment not found"}` (masked as not found to prevent enumeration)
No healthy replicas	503	`{"detail": "No healthy replicas"}`
RabbitMQ publish failed	500	`{"detail": "Queue unavailable"}`
Sync poll timeout	408	`{"request_id": "...", "error": "request timed out"}`
Worker marked expired	408	`{"error": {"message": "request timed out", "type": "timeout_error", ...}}`
Model inference failed	pass-through	`{"error": {"message": "...", "type": "server_error", ...}}` (status code from worker, fallback 502)

Configuration

Environment Variables

Variable	Default	Description
`RABBITMQ_HOST`	`localhost`	RabbitMQ server hostname
`RABBITMQ_PORT`	`5672`	RabbitMQ server port
`RABBITMQ_USERNAME`	`guest`	RabbitMQ username
`RABBITMQ_PASSWORD`	`guest`	RabbitMQ password
`RABBITMQ_VHOST`	`/`	RabbitMQ virtual host
`REDIS_URL`	`redis://localhost:6379`	Redis connection URL
`REDIS_PASSWORD`	(empty)	Redis AUTH password
`API_BASE_URL`	required	Gateway base URL (e.g., `http://gateway:8000`)
`INFERENCE_PROXY_SERVICE_TOKEN`	required	Bearer token for Gateway `/inference/get` calls
`DEPLOYMENT_SCALER_SERVICE_TOKEN`	(empty)	Bearer token accepted from DeploymentScaler requests
`SUPABASE_HTTP_URL`	required	Supabase HTTP API URL (for JWT validation)
`SUPABASE_SERVICE_KEY`	required	Supabase service role key
`SUPABASE_PSQL_URL`	required	Postgres connection URL for API key validation
`LYC_ENVIRONMENT`	`public`	Supabase schema/environment name (used for API key table lookup)
`POLL_INTERVAL_S`	`0.5`	Redis polling interval for sync mode
`DEFAULT_TIMEOUT_S`	`60.0`	Default request timeout
`AUTH_CACHE_TTL_S`	`300`	Auth token cache TTL (5 minutes)
`DEPLOY_HEALTHY_REPLICAS_ZERO_TTL_S`	`10`	Health cache TTL when count=0
`DEPLOY_HEALTHY_REPLICAS_TTL_S`	`600`	Health cache TTL when count>0 (10 minutes)
`INFERENCE_DLX`	`infer.dlx`	Dead-letter exchange name
`INFERENCE_DLQ`	`infer.dlq`	Dead-letter queue name
`INFERENCE_DLX_ROUTING_KEY`	`dead`	DLX routing key
`LOG_LEVEL`	`info`	Log level (`debug`, `info`, `warn`, `error`)
`PORT`	`8080`	HTTP server listen port

Performance Considerations

Caching Strategy

Auth Cache (5 min TTL):

Reduces Supabase API calls by 95%+
Balances security (token revocation lag) with performance

Deployment Metadata (permanent):

Written once at deployment creation
Eliminates Gateway calls for established deployments

Health Cache (adaptive TTL):

10s when count=0: Fast re-check during replica startup
10min when count>0: Stable state, minimal Gateway load

Connection Pooling

RabbitMQ: Single shared connection per instance, auto-reconnect
Redis: Connection pool managed by go-redis client
HTTP: Keep-alive connections to Gateway and Supabase

Horizontal Scaling

InferenceProxy is stateless and scales horizontally via Cloud Run:

Auto-scaling based on request rate
Shared Redis and RabbitMQ eliminate instance-specific state
Per-deployment locking prevents cache stampede

2026-06-042026-06-10