Gateway - Iris Integration
Location: /app/src/app/api/v2_streaming/external/compute/inference/lifecycle/dedicated.py ⧉
Language: Python (FastAPI)
Role: Central orchestrator and source of truth for Iris deployments
Gateway is the existing Lyceum API server extended with Iris-specific endpoints. It manages the deployment lifecycle, maintains CloudSQL as the source of truth, provisions replicas via Streamer, and handles callbacks from Execlet nodes.
Responsibilities
- Deployment Lifecycle: Create, stop, pause, and resume deployments
- Replica Management: Provision, scale, and track replica status
- Database: Maintain
dedicated_deployments and deployment_replicas tables
- Cache Seeding: Write initial deployment metadata to Redis
- Docker Compose Job Rendering: Renders docker-compose template ⧉ with model container (vLLM) and InferenceWorker sidecar configuration
- Streamer Communication: Submits rendered docker-compose jobs to Streamer, which assigns them to Execlet nodes for execution
- Callback Handling: Receive status updates from Execlet nodes
API Endpoints
External (User-Facing)
| Endpoint |
Method |
Caller |
Purpose |
/api/v2/external/inference/create |
POST |
User (lyceum-cli or direct API) |
Create a new dedicated deployment |
/api/v2/external/inference/stop |
DELETE |
User |
Stop a deployment and abort all replicas |
/api/v2/external/inference/get |
GET |
User, InferenceProxy, DeploymentScaler |
Retrieve deployment details and replica status |
/api/v2/external/inference/list |
GET |
User, DeploymentScaler |
List deployments (user's own or all for service tokens) |
/api/v2/external/inference/replica/{replica_id}/logs |
GET |
User |
Get container error logs for failed replicas |
Internal (Service-to-Service)
| Endpoint |
Method |
Caller |
Purpose |
/api/v2/internal/inference/scale |
POST |
DeploymentScaler |
Add or remove replicas from a deployment |
/api/v2/internal/inference/report_replica_status |
POST |
MetricServer |
Report replica health status changes |
/api/v2/internal/inference/pause |
POST |
DeploymentScaler |
Pause a deployment (scale to zero) |
/api/v2/internal/inference/resume |
POST |
InferenceProxy, DeploymentScaler |
Resume a paused deployment |
/api/v2/internal/inference/report_crashed_replicas |
POST |
DeploymentScaler |
Mark replicas as failed due to health check timeout |
Callbacks (from Execlet)
| Endpoint |
Method |
Caller |
Purpose |
/api/internal/inference/replica/progress/{replica_id} |
POST |
Execlet |
Report replica status transitions (pulling → initing → running → failed) |
/api/internal/inference/replica/complete/{replica_id} |
POST |
Streamer |
Handle VM-level crashes (when Execlet connection lost) |
Configuration
Key Environment Variables
| Variable |
Purpose |
VLLM_IMAGE |
vLLM Docker image tag for model containers |
INFERENCE_WORKER_IMAGE |
InferenceWorker Docker image tag |
METRIC_SERVER_URL, METRIC_SERVER_SERVICE_TOKEN |
MetricServer connection and auth |
DEPLOYMENT_SCALER_SERVICE_TOKEN |
Auth token for DeploymentScaler |
INFERENCE_PROXY_SERVICE_TOKEN |
Auth token for InferenceProxy |
EXECLET_HTTP_URL |
Streamer API URL for job submission |
REDIS_URL, REDIS_PASSWORD |
Redis connection for cache seeding |
RABBITMQ_* |
RabbitMQ connection details for docker-compose rendering |
LYC_DB_IRIS_* |
CloudSQL database connection |