Architecture

CosmicAC is a self-hosted platform that runs GPU workloads on your Kubernetes cluster. This page explains the components involved, how they connect to your cluster, and how each job type runs. For deployment steps, see Installation.

Deployment architecture

Setting up your cluster is separate from deploying CosmicAC. You bring a Kubernetes cluster that already has its GPU nodes and KubeVirt configured. The CosmicAC components then connect to that cluster and run your workloads on it.

cosmicac-wrk-server-k8s-nvidia connects to your cluster's Kubernetes API and creates the resources each job needs. For each job request it builds a graph of Kubernetes resources and applies them through the API server. For every workload it provisions a GPU Virtual Machine Instance (VMI), a KubeVirt resource. CDI imports the VMI root disk from a registry image, and the VMI claims one or more whole GPUs through PCI passthrough. Multi-node workloads also claim InfiniBand.

Your cluster's requirements fall into two tiers:

• Base platform: the Kubernetes, GPU, virtualization, storage, and registry foundation that every workload needs.
• Overlay networking: an add-on tier that connects a workload's nodes over an OVS/VXLAN subcluster, and provides a WireGuard gateway for workloads that need an isolated network.

A multi-node vLLM Managed Inference replica needs the subcluster. See Requirements.

CosmicAC documents the cluster requirements, not the steps to build the cluster. See Requirements.

Racks, nodes, and GPUs

Your GPU capacity has three nested levels:

• Rack: the GPU capacity behind one worker, listed under a single location. Despite the name, a rack is an entry in your CosmicAC network, not physical hardware. The worker reports the nodes and GPUs of the cluster it connects to, and cosmicac-wrk-ork tracks them as available capacity.
• Node: a machine in your cluster that holds a pool of GPUs. More than one job can run on the same node, each claiming different GPUs from its pool.
• GPU: a single accelerator card, such as an H100. A job claims one or more whole GPUs through passthrough.

A default deployment registers one rack during task bootstrap. To register, list, or remove racks yourself, see Manage racks in your CosmicAC network.

Every VMI runs inside a Kubernetes pod on a node. Because a node holds a pool of GPUs, one node can host many jobs at once, each in its own pod. The GPU Container architecture diagram shows a node with its pods.

When you create a job, CosmicAC finds a rack in the location you chose, then reserves GPUs on nodes in that rack.

For a Managed Inference Job, how many nodes each replica takes depends on its GPU count:

• Single node: the replica's GPU count fits one node, so it runs in a single VMI.
• Multi-node: the replica's GPU count is more than one node holds, so it runs one VMI on each node.

Only a vLLM Managed Inference replica can be multi-node. A multi-node replica takes its nodes from one rack. Each of those nodes contributes the same number of GPUs. CosmicAC joins the replica's VMIs over InfiniBand and an overlay network.

Every other job runs on one node. If a GPU Container Job asks for more GPUs than a node holds, CosmicAC fails the job.

CosmicAC components

These components make up CosmicAC. Most run outside your cluster as part of the self-hosted platform, and the per-job agents run inside each job's VMI.

• cosmicac-ui: web interface that provides a browser dashboard for creating and managing jobs.
• cosmicac-cli: command-line interface that submits jobs, manages resources, and connects to containers from your terminal.
• cosmicac-app-node: app server that serves the HTTP API, authenticates requests, and routes commands to the orchestrator.
• cosmicac-wrk-ork: orchestrator that allocates resources, distributes jobs across the cluster, and routes requests to the workers.
• cosmicac-wrk-server-k8s-nvidia: server worker that connects to your cluster's Kubernetes API and provisions the VMIs.
• cosmicac-proxy-inference: inference proxy that authenticates Managed Inference requests, balances load, and routes them to model servers.
• cosmicac-wrk-agent-instance: per-job agent that runs inside a GPU Container Job's VMI and accepts shell sessions over hyperswarm-ssh.
• cosmicac-wrk-agent-inference: per-job agent that runs inside a Managed Inference Job's VMI, serves the model with the job's vLLM or Parakeet runtime, and registers itself in the DHT table.
• cosmicac-wrk-monitor: observability aggregator that collects logs and metrics from the other services, exposes them for your Prometheus to scrape, and pushes log lines to your Loki.
• redis: in-memory data store that cosmicac-app-node uses for caching and runtime state, with persistence enabled.
• caddy: web entry point that serves the UI and reverse-proxies API and inference traffic on port 5173.

Caddy proxies /api to cosmicac-app-node and /inference to cosmicac-proxy-inference.

Holepunch stack

Inside CosmicAC, the components connect to each other over the Holepunch peer-to-peer stack rather than through a central server. Components address each other directly, so there is no central broker to route, bottleneck, or expose internal traffic.

• Hyperswarm: peer-to-peer networking that lets components find and connect to each other directly, without a central broker.
• hyperswarm-rpc: internal RPC that carries calls between cosmicac-app-node, cosmicac-wrk-ork, and the workers.
• hyperswarm-ssh: an implementation that gives cosmicac-cli shell access to a running GPU Container Job over Hyperswarm.
• HRPC + Hyperswarm: bidirectional RPC that streams inference responses from the models.
• DHT table: distributed hash table where Managed Inference model servers register, and cosmicac-proxy-inference discovers them by topic.
• HyperDB + Autobase: distributed database that stores usage metrics and job metadata.

Next steps

• GPU Container architecture for how a GPU Container Job runs
• Managed Inference architecture for how a Managed Inference Job runs
• Observability architecture for how logs and metrics reach your stack
• Installation