Skip to content

Deploying InferenceWorker

This guide covers deploying InferenceWorker Docker images. InferenceWorker runs as a sidecar alongside model containers on Execlet VMs.

Prerequisites

  • GitHub Access: Write permissions to trigger GitHub Actions

Deployment Steps

Build Docker Image

Trigger the GitHub Action to build and push the InferenceWorker Docker image:

Location: .github/workflows/docker_iris.yml ⧉

Via GitHub UI:

  1. Click "Run workflow"
  2. Select branch (usually main)
  3. Select target GCP project
  4. Ensure "Build inference_worker image" is checked
  5. Click "Run workflow" button

What it does:

  • Builds Docker image for InferenceWorker
  • Pushes to Google Artifact Registry as inference_worker:latest
  • Tags with commit SHA for version tracking

Image locations:

  • Development: us-central1-docker.pkg.dev/devel-466814/inference/inference_worker:latest
  • Production: us-central1-docker.pkg.dev/production-492618/inference/inference_worker:latest

How It Gets Deployed

InferenceWorker is deployed automatically as part of the VM provisioning process:

New VMs

When Hydra provisions a new VM:

  1. Hydra creates VM based on job requirements
  2. Ansible runs VM initialization scripts
  3. VM pulls inference_worker:latest from Artifact Registry
  4. Gateway submits docker-compose job to Streamer
  5. Execlet runs docker-compose up with InferenceWorker sidecar
  6. InferenceWorker starts consuming from RabbitMQ

Result: New VMs automatically use the latest InferenceWorker image.

Existing VMs

VMs that are already running will continue using their current InferenceWorker image until:

  • The replica is stopped and a new one is created
  • The VM is terminated and replaced (manual or auto-scaling)
  • The deployment is recreated

Note: There is no automatic rolling update for InferenceWorker on existing VMs.

Verification

1. Check Image Exists

Verify the new image was pushed to Artifact Registry:

# Development
gcloud artifacts docker images list \
  us-central1-docker.pkg.dev/devel-466814/inference/inference_worker \
  --project devel-466814

# Production
gcloud artifacts docker images list \
  us-central1-docker.pkg.dev/production-492618/inference/inference_worker \
  --project production-492618

2. Check New Replica Uses Latest Image

After creating a new deployment or scaling up:

# SSH to Execlet VM
ssh <execlet-vm-ip>

# Check running containers
docker ps | grep inference_worker

# Check image tag
docker inspect <container-id> | grep Image

Expected: Image tag should match the commit SHA or latest from the recent build.

Rollback

If the new InferenceWorker image has issues:

Rollback Image in Artifact Registry

1. Find previous working image:

gcloud artifacts docker images list \
  us-central1-docker.pkg.dev/production-492618/inference/inference_worker \
  --project production-492618 \
  --format="table(version,createTime)"

2. Tag previous version as latest:

# Pull old version
docker pull us-central1-docker.pkg.dev/production-492618/inference/inference_worker:<old-sha>

# Retag as latest
docker tag \
  us-central1-docker.pkg.dev/production-492618/inference/inference_worker:<old-sha> \
  us-central1-docker.pkg.dev/production-492618/inference/inference_worker:latest

# Push
docker push us-central1-docker.pkg.dev/production-492618/inference/inference_worker:latest

Troubleshooting

Worker Not Reporting Metrics

Check worker logs:

# SSH to VM
ssh <execlet-vm-ip>

# View worker logs
docker logs <inference-worker-container-id>

Common issues:

  • METRIC_SERVER_URL or METRIC_SERVER_SERVICE_TOKEN incorrect
  • MetricServer unreachable from VM network
  • Worker crashed during startup (check exit code)

Worker Not Consuming Messages

Check RabbitMQ connection:

# Check worker logs for connection errors
docker logs <inference-worker-container-id> | grep -i rabbitmq

Common issues:

  • RABBITMQ_HOST or credentials incorrect
  • Firewall blocks connection to RabbitMQ

Best Practices

Emergency Hotfix

For critical bugs requiring immediate rollback:

  1. Rollback latest tag in Artifact Registry (see Rollback section)
  2. Notify team to remove VM in PI
  3. Auto-scaler will gradually replace replicas as traffic scales
  4. Fix the bug and re-deploy proper fix