Replicas

A replica is one running copy of the job's model. All replicas serve requests through a single endpoint. When you create a Managed Inference Job, you choose how many replicas to run.

Running more than one replica gives the endpoint two benefits.

• Capacity: more replicas serve more traffic, because CosmicAC balances requests across every healthy replica.
• Resilience: if a replica fails, the endpoint keeps serving on the remaining replicas at reduced capacity.

A round-robin load balancer spreads requests across the healthy replicas, and drops an unhealthy one from rotation. It holds no sticky sessions, so consecutive requests from one caller can land on different replicas.

CosmicAC doesn't scale the count automatically, and a job keeps the count it started with. The endpoint's model health reflects the health of its replicas.

Multi-node replicas

A replica claims the GPUs that its model needs. When a model needs more GPUs than one of your nodes holds, the replica runs on more than one node from one rack. Each of those nodes runs one VMI and contributes an equal number of GPUs.

Those VMIs serve as one replica. The first VMI runs the model server and answers requests from the load balancer. The others add their GPUs to that server. CosmicAC connects them over InfiniBand and an overlay network.

Run cosmicac jobs detail to see the node count and node names for each replica.

Only a vLLM replica can be multi-node. A Parakeet replica always runs on one node.

A multi-node replica needs the overlay networking add-ons on your cluster. See Requirements.

Replicas and nodes scale independently. The replica count sets how many copies of the model serve the endpoint. The model's GPU count sets how many nodes each copy takes.

You don't choose the node count. You set a GPU count for one replica. That count must match the per_replica_gpu_count in the model master. See GPU configuration.

Next steps

• Model Health for how CosmicAC rates the replicas behind an endpoint
• Create a vLLM Managed Inference Job Using the Web UI to set a replica count
• Model Masters for where a model's GPU count comes from
• Managed Inference architecture for how a replica runs on the cluster