Skip to content

Deploying InferenceProxy

This guide covers deploying InferenceProxy to Cloud Run in different environments.

Prerequisites

  • GitHub Access: Write permissions to trigger GitHub Actions
  • GCP Access: Cloud Run Admin role for target project
  • gcloud CLI: Installed and authenticated

Deployment Steps

1. Build Docker Image

Trigger the GitHub Action to build and push the InferenceProxy Docker image:

Location: .github/workflows/docker_iris.yml ⧉

Via GitHub UI:

  1. Click "Run workflow"
  2. Select branch (usually main)
  3. Select target gcp project
  4. Ensure "Build inference_proxy image" is checked
  5. Click "Run workflow" button

What it does:

  • Builds Docker image for InferenceProxy
  • Pushes to Google Artifact Registry
  • Tags as latest and with commit SHA

2. Deploy to Cloud Run

Once the Docker image is built, deploy to the target environment using gcloud:

Development Environment

gcloud run deploy inference-proxy \
  --project devel-466814 \
  --region us-central1 \
  --image us-central1-docker.pkg.dev/devel-466814/inference/inference_proxy:latest

URL: https://devel.iris.lyceum.technology

Production Environment

gcloud run deploy inference-proxy \
  --project production-492618 \
  --region europe-west3 \
  --image us-central1-docker.pkg.dev/production-492618/inference/inference_proxy:latest

URL: https://prod.iris.lyceum.technology

Verification

1. Check Deployment Status

# Production
gcloud run services describe inference-proxy \
  --project production-492618 \
  --region europe-west3 \
  --format="get(status.url,status.conditions)"

2. Test Inference Request

DEPLOYMENT_ID="<your-deployment-id>"
TOKEN="<your-auth-token>"

# Development
curl -X POST https://devel.iris.lyceum.technology/v1/chat/completions \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"${DEPLOYMENT_ID}\",
    \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}]
  }"

# Production
curl -X POST https://prod.iris.lyceum.technology/v1/chat/completions \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"${DEPLOYMENT_ID}\",
    \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}]
  }"

Rollback

If the deployment fails or has issues, rollback to the previous revision:

# List revisions
gcloud run revisions list \
  --service inference-proxy \
  --project production-492618 \
  --region europe-west3

# Rollback to previous revision
gcloud run services update-traffic inference-proxy \
  --to-revisions <PREVIOUS_REVISION>=100 \
  --project production-492618 \
  --region europe-west3

Troubleshooting

Deployment Fails

Check image exists: 

gcloud artifacts docker images list \
  us-central1-docker.pkg.dev/production-492618/inference/inference_proxy \
  --project production-492618

Service Returns 503

  • Check Redis connection: Ensure REDIS_URL and REDIS_PASSWORD are correct
  • Check RabbitMQ connection: Verify RABBITMQ_HOST is reachable
  • Check Gateway availability: Test API_BASE_URL endpoint

Configuration Updates

To update environment variables or other configuration:

gcloud run services update inference-proxy \
  --project production-492618 \
  --region europe-west3 \
  --set-env-vars "POLL_INTERVAL_S=0.05,DEFAULT_TIMEOUT_S=30"

Important: Configuration changes trigger a new deployment.