Managed Inference architecture
How a Managed Inference Job runs on your cluster, and how CosmicAC authenticates and routes an inference request.
A Managed Inference Job runs an open source model inside a VMI, using vLLM for language models or Parakeet for speech-to-text. It exposes the model through cosmicac-proxy-inference as an OpenAI-compatible endpoint, which authenticates requests and balances load. You reach the model through that endpoint from any OpenAI-compatible client, or by running inference directly with cosmicac-cli.
The following diagram shows a multi-node replica on two nodes. A replica that fits on one node runs a single VMI.
The inference agent
Every VMI in a replica runs cosmicac-wrk-agent-inference, the inference agent. The agent runs the model server in a second container, built from the vLLM or Parakeet runtime image, and restarts that container if it exits.
The agent and the model server are separate processes, so every VMI runs one of each. The agent also captures the model server's output, which is why that output appears in the job's logs.
How a job starts
When you create a Managed Inference Job from cosmicac-ui or cosmicac-cli, cosmicac-app-node authenticates the request and forwards it to cosmicac-wrk-ork. cosmicac-wrk-ork allocates GPUs for each replica. The orchestrator then hands the job to cosmicac-wrk-server-k8s-nvidia, which creates the Kubernetes resources. Kubernetes runs each replica as a pod containing a VMI with cosmicac-wrk-agent-inference inside.
A multi-node replica runs one pod and one VMI on each of its nodes. The first VMI runs the model server and answers requests. The others add their GPUs to that server. Ray forms the cluster over the overlay network. The NVIDIA Collective Communications Library (NCCL) carries the GPU-to-GPU traffic over InfiniBand.
As a replica spins up, the cosmicac-wrk-agent-inference on its serving node registers in the DHT table, so cosmicac-proxy-inference can discover the replica.
How CosmicAC serves a request
Serving traffic follows a separate path from job creation. A client sends a request to the inference endpoint over the OpenAI-compatible API, or you run inference from cosmicac-cli. cosmicac-proxy-inference authenticates the request, searches the DHT table by topic to discover a model server, and balances load across the running servers. cosmicac-wrk-agent-inference then runs the request with the job's runtime and returns the response.
Next steps
- Managed Inference Job for what the job is and when to use one
- Replicas for how many copies of the model serve an endpoint
- Architecture for the components and how CosmicAC reaches your cluster