Skip to content

Redis Cache Strategy

Iris uses Redis for multiple caching layers with different update patterns. This document describes all Redis keys used in the system.

1. Response Bus

Key Pattern: infer_resp:<request_id>

Example: infer_resp:a1b2c3d4-e5f6-7890-abcd-ef1234567890

Value: JSON object with inference response

Writer:

  • InferenceProxy writes {"status":"pending"} in async mode (TTL = request's timeout_s)
  • InferenceWorker overwrites with final result after model inference completes

Reader: InferenceProxy polls this key for results

TTL: REDIS_RESPONSE_TTL_S (default 3600s / 1 hour) when written by worker; timeout_s when written as async pending sentinel

Source:

2. Auth Cache

Key Pattern: auth:<sha256(token)>

Example: auth:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08

Value: user_id string

Writer: InferenceProxy on JWT/API key validation

Reader: InferenceProxy on every request

TTL:

  • JWT: Lesser of AUTH_CACHE_TTL_S or token's remaining lifetime
  • API key: Lesser of AUTH_CACHE_TTL_S or key's expires_at

Updated When:

  • Cache miss (token not in Redis)
  • After successful validation via Supabase or database

Source: iris/inference_proxy/cache.go ⧉

3. Deployment Metadata Cache

Key Pattern: deploy:meta:<deployment_id>

Example: deploy:meta:550e8400-e29b-41d4-a716-446655440000

Type: Redis hash

Fields: deployment_id, user_id, hf_model_id, hardware_profile, min_replicas, max_replicas

Writer:

  • Gateway seeds on /inference/create
  • InferenceProxy refreshes on cache miss

Reader: InferenceProxy on every request

TTL: None (permanent until deployment deleted)

Updated When:

  • Deployment creation (Gateway)
  • Cache miss - InferenceProxy calls Gateway /inference/get and writes result

Source:

4. Deployment Health Cache

Key Pattern: deploy:healthy_replicas:<deployment_id>

Example: deploy:healthy_replicas:550e8400-e29b-41d4-a716-446655440000

Type: Redis hash

Fields: count (int), last_update (unix timestamp)

Writer:

  • Gateway seeds count=0 on /inference/create
  • InferenceProxy refreshes on TTL expiration

Reader: InferenceProxy on every request

TTL:

  • DEPLOY_HEALTHY_REPLICAS_ZERO_TTL_S (default 10s) when count=0
  • DEPLOY_HEALTHY_REPLICAS_TTL_S (default 600s / 10min) when count>0

Updated When:

  • Deployment creation with count=0 (Gateway)
  • TTL expiration - InferenceProxy calls Gateway /inference/get, counts healthy replicas, writes new count

Rationale: Shorter TTL when count=0 forces frequent checks during replica startup; longer TTL when healthy reduces load.

Source: iris/inference_proxy/cache.go ⧉

5. Replica Health State

Key Pattern: instances:<replica_id>

Example: instances:rep-550e8400-e29b-41d4-a716-446655440000

Type: Redis hash

Fields: replica_id, deployment_id, hf_model_id, user_id, status, last_report

Writer: MetricServer on receiving health reports

Reader: MetricServer health check loop

TTL: None (explicit eviction after 1h of staleness)

Updated When:

  • InferenceWorker sends health snapshot every 30s → MetricServer updates Redis
  • MetricServer marks status=unhealthy if last_report age > 60s (stale threshold)
  • MetricServer removes key if unhealthy for > 1h (eviction threshold)

Source: iris/metric_server/internal/redis_cache/redis_cache.go ⧉

6. All Instances Set

Key Pattern: instances:all

Type: Redis set

Value: Set of all replica_id strings

Writer: MetricServer (adds on upsert, removes on eviction)

Reader: MetricServer for listing all tracked replicas

TTL: None (persistent)

Source: iris/metric_server/internal/redis_cache/redis_cache.go ⧉

Cache Update Patterns

Write-Through (Gateway → Redis)

  • Deployment metadata (deploy:meta:*) is written immediately on deployment creation
  • Health cache (deploy:healthy_replicas:*) is seeded with count=0

Write-Behind (InferenceWorker → Redis)

  • Response bus (infer_resp:*) is written after model inference completes
  • Replica health (instances:*) is written every 30s by MetricServer

Cache-Aside (InferenceProxy → Redis)

  • Auth cache (auth:*) is populated on cache miss after validation
  • Deployment metadata and health are refreshed on TTL expiration

TTL-Based Invalidation

  • Auth cache uses token lifetime or configured TTL
  • Health cache uses shorter TTL when count=0 (10s) vs count>0 (10min)
  • Replica health uses explicit eviction rather than TTL