Skip to content

RabbitMQ Queue Naming

Each deployment has a dedicated RabbitMQ queue for inference requests.

Queue Pattern

Pattern: infer_req.<deployment_id>

Example: infer_req.550e8400-e29b-41d4-a716-446655440000

Source Code

InferenceProxy

iris/inference_proxy/config.go ⧉

func (s *Settings) QueueName(deploymentID string) string {
    return "infer_req." + deploymentID
}

InferenceWorker

iris/inference_worker/src/config.py ⧉

@property
def INFERENCE_QUEUE(self) -> str:
    return f"infer_req.{self.DEPLOYMENT_ID}"

Queue Properties

  • One queue per deployment (not per replica)
  • All replicas consume from the same queue (competing consumers pattern)
  • Queue is created when first replica connects
  • Queue persists until deployment is deleted
  • Dead-letter queue (DLQ): All inference queues share a single DLQ named infer.dlq for failed messages

Message Flow

  1. InferenceProxy publishes inference requests to infer_req.<deployment_id>
  2. All InferenceWorkers for that deployment consume from the same queue
  3. RabbitMQ distributes messages based on worker availability and prefetch limits
  4. Failed messages are routed to the shared DLQ infer.dlq

This competing consumers pattern provides natural load balancing without requiring an external load balancer.