The open AI-native stack on Kubernetes: From open models to distributed inference and agents

Detailed summary

KubeRay provides an open-source, Kubernetes-native operator that manages the complete lifecycle of Ray clusters, jobs, and services (RayCluster, RayJob, and RayService custom…

Over the past decade, Kubernetes established itself as the standard for orchestrating microservices and containerized applications. Today, we're witnessing the next major architectural shift in distributed computing: AI workloads moving from isolated research prototypes into large-scale, multi-tenant production fleets. In this post, we'll walk through the open-source stack that powers this shift and show you the sample manifests that you can try on your own Kubernetes clusters. You can also join us at KubeCon + CloudNativeCon North America 2026 to see these projects in action through live demos and technical sessions.

For modern engineering teams, production AI is rapidly moving beyond generic calls to closed, black-box APIs. Organizations are realizing that their long-term competitive advantage lies in domain specialization. Platform engineers, ML researchers, and AI builders want the autonomy to run high-performing open-weight models (such as the Gemma family). They want to adapt them to proprietary enterprise datasets by using reinforcement learning (RL) and serve them with predictable latency, high throughput, and maximum hardware efficiency.

Whether it's scientific initiatives training domain models like DolphinGemma (an LLM helping scientists study how dolphins communicate) on acoustic data, legal platforms evaluating complex contracts, or engineering teams building autonomous coding agents, the objective is the same: achieving frontier-level task performance at a fraction of the serving cost and latency.

Yet, scaling RL post-training, stateful distributed inference, and autonomous agent execution on traditional container infrastructure introduces substantial hurdles:

At KubeCon + CloudNativeCon North America 2026, Google Cloud is addressing these challenges by introducing and expanding a unified, open-source AI-native stack across Kubernetes and Google Kubernetes Engine (GKE) based on these components:

While supervised fine-tuning (SFT) has become standard…

Histori…

Opening of the article, excerpted from Google Finance. To read the rest, open the article on the publisher's site.