MetricServer

Location: iris/metric_server/ ⧉ Language: Go Role: Telemetry broker between InferenceWorkers and Prometheus

MetricServer acts as a central metrics collection point that receives raw metrics from all InferenceWorkers, maintains replica health state in Redis, exposes a Prometheus-compatible /prom endpoint, and reports health changes back to the Gateway.

Architecture

InferenceWorker (each replica)
      │
      │ POST /metrics
      │ (every 30s)
      │
      ▼
MetricServer
      │
      ├─► Redis
      │   └─ Update replica health state
      │       + last report timestamp
      │
      ├─► Gateway
      │   └─ POST /report_replica_status
      │       (when health changes)
      │
      └─► In-Memory Prometheus Registry
          └─ Update gauges, histograms, counters
              │
              │ GET /prom
              │ (scrape every 15s)
              ▼
          Prometheus
              │
              └─► DeploymentScaler
                  (queries for scaling decisions)

Key Responsibilities

1. Metrics Ingestion (`POST /metrics`)

Receives metrics payloads from InferenceWorkers containing inference records, health snapshots, and resource metrics. Updates Redis with replica health state and records metrics in Prometheus registry. Notifies Gateway when health status changes.

2. Health State Management (Redis)

Maintains replica health state in Redis (replica_health:<replica_id>) with health status, last report timestamp, and deployment metadata. Replicas are healthy if healthy == true in latest report and reported within stale threshold.

3. Staleness Checking (Background Loop)

Runs every STALENESS_CHECK_INTERVAL (default 10 minutes). Marks replicas unhealthy if no report within STALE_THRESHOLD (60s). Evicts replicas from Redis after EVICTION_THRESHOLD (1h) to prevent memory leaks.

4. Prometheus Exposition (`GET /prom`)

Exposes metrics in Prometheus text-based format:

Metric	Type	Labels	Purpose
`lyceum_inference_latency_millis`	Histogram	`deployment_id`, `user_id`, `hf_model_id`	P50, P95, P99 latency calculations
`lyceum_inference_requests_total`	Counter	`deployment_id`, `user_id`, `hf_model_id`, `status_code`	RPS calculations, error rates
`lyceum_inference_tokens_total`	Counter	`deployment_id`, `user_id`, `hf_model_id`, `token_type`	Token usage tracking
`lyceum_inference_instance_health`	Gauge	`deployment_id`, `replica_id`, `user_id`, `hf_model_id`	Per-replica health status (1=healthy, 0=unhealthy)
`lyceum_inference_instance_last_report_age_seconds`	Gauge	`deployment_id`, `replica_id`	Detect reporting delays
`lyceum_inference_kv_cache_usage_percent`	Gauge	`deployment_id`, `replica_id`	Monitor KV cache pressure
`lyceum_inference_gpu_utilization_percent`	Gauge	`deployment_id`, `replica_id`, `gpu_index`	GPU utilization tracking
`lyceum_inference_queue_depth`	Gauge	`deployment_id`, `replica_id`	Queue backlog monitoring

5. Health Reporting to Gateway

Calls POST /api/v2/internal/inference/report_replica_status when replica health changes. Gateway updates deployment_replicas table and triggers cache refresh in InferenceProxy. Only reports when status toggles (healthy ↔ unhealthy).

Configuration

Environment Variables

Variable	Default	Description
`LYC_REDIS_ADDR`	`localhost:6379`	Redis server address
`LYC_REDIS_PASSWORD`	(empty)	Redis AUTH password
`LYC_API_ADDR`	`:8081`	HTTP listen address
`LYC_STALE_THRESHOLD`	`60s`	Duration after which non-reporting instance is unhealthy
`LYC_PROM_SCRAPE_INTERVAL`	`15s`	Prometheus scrape interval (for shutdown grace tuning)
`LYC_GRACEFUL_SCRAPE_DELAY`	`20s`	Duration to wait before shutting down API to allow final Prometheus scrape
`LYC_STALENESS_CHECK_INTERVAL`	`10m`	How often to check for stale instances
`LYC_EVICTION_THRESHOLD`	`1h`	Remove instances from Redis after being unhealthy this long
`METRIC_SERVER_SERVICE_TOKEN`	(empty)	Bearer token for `POST /metrics` auth and Gateway API calls; if unset, auth and Gateway reporting disabled
`API_BASE_URL`	(empty)	Gateway URL for health reporting; if unset, reporting disabled
`DEFAULT_TIMEOUT_S`	`60.0`	HTTP timeout for Gateway calls (in seconds)

Note: Gateway reporting requires both API_BASE_URL and METRIC_SERVER_SERVICE_TOKEN to be set. The same service token is used for both incoming metrics auth and outgoing Gateway calls.

Command-Line Flags

Flags take precedence over environment variables (for LYC_* variables only):

./metric_server \
  -redis-addr=redis:6379 \
  -redis-password=secret \
  -api-addr=:8081 \
  -stale-threshold=60s \
  -graceful-scrape-delay=20s \
  -prom-scrape-interval=15s

Operational Considerations

High Availability

Multiple MetricServer instances can run simultaneously
All state in Redis (no local state)
Prometheus scrapes all instances (aggregates automatically)

Failure Modes

Redis Down: - Metrics ingestion fails (returns 500) - Workers retry on next flush interval - Prometheus scraping continues (serves cached metrics)

Gateway Down: - Health reporting fails - Logged but doesn't block metrics ingestion - Retries on next health change

Prometheus Down: - MetricServer unaffected - Metrics continue to accumulate - DeploymentScaler cannot scale (no metrics)

2026-06-042026-06-10