Skip to content

Deploying MetricServer

This guide covers deploying MetricServer to Cloud Run in different environments.

Prerequisites

  • GitHub Access: Write permissions to trigger GitHub Actions
  • GCP Access: Cloud Run Admin role for target project
  • gcloud CLI: Installed and authenticated

Deployment Steps

1. Build Docker Image

Trigger the GitHub Action to build and push the MetricServer Docker image:

Location: .github/workflows/docker_iris.yml ⧉

Via GitHub UI:

  1. Click "Run workflow"
  2. Select branch (usually main)
  3. Select target GCP project
  4. Ensure "Build metric_server image" is checked
  5. Click "Run workflow" button

What it does:

  • Builds Docker image for MetricServer
  • Pushes to Google Artifact Registry
  • Tags as latest and with commit SHA

2. Deploy to Cloud Run

Once the Docker image is built, deploy to the target environment using gcloud:

Development Environment

gcloud run deploy metric-server \
  --project devel-466814 \
  --region us-central1 \
  --image us-central1-docker.pkg.dev/devel-466814/inference/metric_server:latest

Production Environment

gcloud run deploy metric-server \
  --project production-492618 \
  --region europe-west3 \
  --image us-central1-docker.pkg.dev/production-492618/inference/metric_server:latest

Verification

1. Check Deployment Status

# Production
gcloud run services describe metric-server \
  --project production-492618 \
  --region europe-west3 \
  --format="get(status.url,status.conditions)"

2. Test Prometheus Metrics Endpoint

# Production
curl https://metric-server-378619998622.europe-west3.run.app/prom

Expected: Prometheus text-based metrics format with lyceum_inference_* metrics.

Rollback

If the deployment fails or has issues, rollback to the previous revision:

# List revisions
gcloud run revisions list \
  --service metric-server \
  --project production-492618 \
  --region europe-west3

# Rollback to previous revision
gcloud run services update-traffic metric-server \
  --to-revisions <PREVIOUS_REVISION>=100 \
  --project production-492618 \
  --region europe-west3

Troubleshooting

Deployment Fails

Check image exists: 

gcloud artifacts docker images list \
  us-central1-docker.pkg.dev/production-492618/inference/metric_server \
  --project production-492618

Service Returns Errors

  • Check Redis connection: Ensure LYC_REDIS_ADDR and LYC_REDIS_PASSWORD are correct
  • Check Gateway availability: Verify API_BASE_URL is reachable
  • Check service token: Ensure METRIC_SERVER_SERVICE_TOKEN is valid (used for both metrics auth and Gateway calls)

No Metrics Appearing

Check InferenceWorkers are reporting:

  • Workers should POST to /metrics every 30s
  • Check worker logs for connection errors
  • Verify METRIC_SERVER_URL and METRIC_SERVER_SERVICE_TOKEN in worker config

Configuration Updates

To update environment variables or other configuration:

gcloud run services update metric-server \
  --project production-492618 \
  --region europe-west3 \
  --set-env-vars "LYC_STALE_THRESHOLD=60s,LYC_EVICTION_THRESHOLD=1h"

Important: Configuration changes trigger a new deployment.