Skip to content

Testing Locally

This guide covers running Iris components locally for development and testing using the local_test_linux.sh script.

Prerequisites

Hardware Requirements

  • GPU Machine: Required for testing model inference (e.g., H100, A100)
  • Minimum: 1 GPU, 16 GB RAM
  • Recommended: 2+ GPUs, 32 GB+ RAM

Required Tools

Install the following tools before running local tests:

  1. NVIDIA Driver - Latest stable version
  2. CUDA Toolkit - Version compatible with your driver
  3. Docker - Version 20.10+
  4. NVIDIA Container Toolkit - For GPU support in Docker
  5. Supabase CLI - For local database
  6. AWS CLI (s3) - For model downloads
  7. Go - Version 1.21+ (for building Execlet/Streamer)

Setup Supabase

Initialize local Supabase database with Lyceum schema:

cd supabase

# Start Supabase containers
supabase start

# Initialize both schemas (public and integration)
./scripts/init_db.sh public
./scripts/init_db.sh integration

# Apply incremental migrations
./scripts/apply_changes.sh public
./scripts/apply_changes.sh integration

cd ..

What it does:

  • Starts PostgreSQL, Auth, Storage, and other Supabase services
  • Creates public and integration schemas
  • Applies all database migrations
  • Seeds initial data

Supabase Access:

  • API URL: http://localhost:54321
  • Studio URL: http://localhost:54323
  • DB URL: postgresql://postgres:postgres@localhost:54322/postgres

For more details, see supabase/README.md ⧉.

Start All Components

Run the local test script to start Gateway, InferenceProxy, Streamer, Execlet, MetricServer, DeploymentScaler, and Croesus:

./local_test_linux.sh \
  --rebuild-execlet \
  --rebuild-inference \
  --hardware-profiles gpu.h100,gpu.a100 \
  --reset-croesus-db \
  --reset-iris-db

Script Options

Option Description
--env ENV Environment/schema to use (integration or public). Default: integration
--gateway-docker Run Gateway in Docker (default: run locally)
--gateway-local Run Gateway locally (default)
--skip-supabase Skip starting Supabase
--skip-streamer Skip building and starting Streamer
--skip-execlet-workers Skip building and starting Execlet Workers
--skip-inference-proxy Skip building and starting InferenceProxy
--skip-metric-server Skip building and starting MetricServer
--skip-deployment-scaler Skip building and starting DeploymentScaler
--skip-croesus Skip building and starting Croesus BillingServer
--skip-gateway Skip starting Gateway
--reset-iris-db Drop and recreate Iris database (re-run migrations)
--reset-croesus-db Drop and recreate Croesus database (re-run migrations)
--rebuild-execlet Force rebuild Execlet Docker images (ignore cache)
--rebuild-inference Force rebuild InferenceProxy/Worker/MetricServer/DeploymentScaler images
--rebuild-croesus Force rebuild Croesus Docker image (ignore cache)
--hardware-profiles LIST Comma-separated hardware profiles for workers (e.g., cpu,gpu.h100). Default: cpu
--export-env Export env vars to .env_local_test (shell) and .env_local_test_debug (.env)

Generated Files

  • .env_local_test - Shell script with environment variables (use source .env_local_test)
  • .env_local_test_debug - .env format for debugging

Setup Test User (Optional - First Time Only)

Note: This step is only needed if you haven't created a test user before. User data is persisted in local Supabase, so you only need to run this once.

Open a new terminal and setup environment variables:

# Source generated environment variables
source .env_local_test

# Setup test user (first time only)
python app/scripts/setup_test_user.py

What it does:

  • Creates a test user in Supabase Auth
  • Generates an API key (lk_*)
  • Outputs user ID and API key

Output example: 

Created test user:
  Email: [email protected]
  User ID: 550e8400-e29b-41d4-a716-446655440000
  API Key: lk_abc123def456...

If you already have a test user:

  • Skip this step and use your existing credentials
  • Or re-run to generate a new API key if you forgot the old one

Test Dedicated Inference

Use the test script to create a deployment and send inference requests:

# Test dedicated inference deployment
python app/scripts/test_dedicated_inference.py

What it does:

  1. Creates a dedicated deployment via Gateway API
  2. Waits for replica to become healthy
  3. Sends test inference requests to InferenceProxy
  4. Validates responses
  5. Cleans up deployment

Example output: 

Creating deployment...
Deployment ID: dep-abc123
Waiting for replica to start...
Replica healthy: rep-def456
Sending inference request...
Response: {"choices": [{"message": {"content": "Hello!"}}]}
Test passed!

Cleanup

Stop Local Test Script

By default, press Ctrl+C to stop the local_test_linux.sh process:

# In the terminal running local_test_linux.sh
^C  # Press Ctrl+C

What it does: - Automatically stops related Docker containers (InferenceProxy, MetricServer, DeploymentScaler, Croesus, Execlet workers) - Does not stop basic tools like Supabase

Stop Dedicated Deployments

Important: If you created a dedicated deployment via test_dedicated_inference.py or Gateway API, the docker-compose job will not be stopped automatically.

Option 1: Use test script cleanup (Recommended)

# The test script automatically cleans up at the end
python app/scripts/test_dedicated_inference.py

Option 2: Manual cleanup

# List all running containers
docker ps

# Find and stop deployment containers (look for vllm and inference_worker containers)
docker stop <container-id>