Skip to content

Deployment and Replica Status Lifecycle

This document explains the status values and state transitions for deployments and replicas in the Iris inference system.

Deployment Status

Deployment status is stored in dedicated_deployments.status and tracks the overall lifecycle of a deployment.

Status Description Transitions To Triggered By
created Active deployment; replicas may be pending, provisioning, or running paused, failed, stopped Gateway on /inference/create after submitting initial replicas
paused Scaled to zero (no replicas) created DeploymentScaler when idle timeout reached
failed Permanent failure (all replicas failed fatally) (terminal) Gateway callback when last non-terminal replica fails with fatal error (image pull failure, container init failure)
stopped User-terminated deployment (terminal) User via Gateway /inference/stop; all replicas aborted

Source:

Notes:

  • failed is terminal — user must create a new deployment with corrected configuration
  • stopped is terminal — deployment cannot be restarted
  • paused is resumable — InferenceProxy automatically triggers resume on first inference request

Replica Status and State Machine

Replica status is stored in deployment_replicas.status and tracks the lifecycle of each individual replica (VM + docker-compose job).

Status Values

Status Description Terminal?
pending Replica record created, job submitted to Streamer No
pulling_docker_image Execlet is downloading Docker images No
initing_docker_container Execlet is starting containers (before healthy) No
running Containers healthy and accepting requests No
failed Replica failed (see Failure Reasons below) Yes
stopped User-terminated or aborted by Gateway Yes

Source: shared_models/src/lyceum/shared_models/inference.py ⧉

State Machine

                    STARTING              STARTING               RUNNING
                  "Pulling image"    "Model server..."         "healthy"
                        │                   │                      │
  pending ─────────────►│                   │                      │
                        ▼                   │                      │
             pulling_docker_image ──────────┼──────────────────────┤
                                            ▼                      │
                              initing_docker_container ────────────┘
                                                                   │
                                                                   ▼
                                                                running
                                                                   │
                                                      (crash, OOM, SYSTEM_FAILURE)
                                                                   │
                                                                   ▼
                                                                failed


  Any non-terminal state can also transition to:
    - stopped  (user calls /inference/stop; Gateway writes status then aborts)
    - failed   (see Failure Transitions table for specific conditions)

Triggers:

  • pending → pulling_docker_image: Execlet progress callback with message "Pulling image..."
  • pulling_docker_image → initing_docker_container: Execlet progress callback with message "Model server is starting..."
  • initing_docker_container → running: Execlet progress callback with "healthy" in message
  • Any → stopped: User calls Gateway /inference/stop → Gateway writes status='stopped' → Gateway aborts job
  • Any → failed: See Failure Transitions below

Failure Transitions and Fatality

Replicas can fail for different reasons, and some failures are fatal (indicate permanent misconfiguration) while others are non-fatal (transient infrastructure issues).

From State Execlet Status Fail Reason Fatal? Description
pulling_docker_image FAILED image_pull_failed Yes Bad HuggingFace token, private model without access, or wrong image tag
initing_docker_container FAILED container_init_failed Yes Application crashes during startup, OOM during init, or bad configuration
running SYSTEM_FAILURE container_crashed No Container exited unexpectedly after becoming healthy (crash, OOM, etc.)
Any VM_CRASH container_crashed No Execlet process died (VM crash, network partition, etc.) — transient infrastructure failure

Fatal Failures:

  • If a replica fails with a fatal reason and no other replicas remain in non-terminal state, the deployment is automatically marked as status='failed'
  • This prevents DeploymentScaler from spawning new replicas that will fail the same way
  • User must create a new deployment with corrected configuration (valid HF token, accessible model, etc.)

Non-Fatal Failures:

  • DeploymentScaler will eventually spawn replacement replicas when current_replicas < min_replicas
  • These are treated as transient issues that auto-scaling can recover from

Current Replicas Count:

  • Incremented by +1 when replica transitions to running
  • Decremented by -1 when replica transitions from running → failed (crashed)
  • Not decremented when replica fails before reaching running (never counted)

Source: