September 21, 2026

How to Evaluate AI Agents From Tool Calls to Task Completion

When you ship an AI agent, the key question is whether it can execute a chain of work across dozens of sequential tool calls against a live environment, and...

NVIDIA Or Micron: Which Gets Paid More Safely For The AI Shortage? - Yahoo Finance

NVIDIA Or Micron: Which Gets Paid More Safely For The AI Shortage? Yahoo Finance

Open AI forms math advisory group as its AI resolves more than 100 open problems

The group won't be given leeway to slow down or redirect OpenAI's ongoing mathematical research.

Grok Build vs. Claude Code: I tested which one has the better memory

On September 16, xAI announced memory in Grok Build, its terminal coding agent. The pitch was that Grok “keeps notes The post Grok Build vs. Claude Code: I tested which one has the better memory appeared first on The New Stack.

Treasury Secretary Says Open AI Managers Are to Blame for Hugging Face Breach, Not AI Agents - Gizmodo

Treasury Secretary Says OpenAI Managers Are to Blame for Hugging Face Breach, Not AI Agents Gizmodo

British Columbia Sues Open AI Over Canada Mass Shooting Warning Failure

British Columbia filed a lawsuit in California against OpenAI, alleging it could have used ChatGPT logs to warn police and avert a mass shooting in the Canadian province earlier this year.

After Amazon Blocks Meta’s Muse Agent, Shopify Says It’ll Integrate Muse To Help People Shop On Its Stores

The agentic shopping wars just split into two camps — and the fault line is exactly where you’d expect it: Amazon on one... The post After Amazon Blocks Meta’s Muse Agent, Shopify Says It’ll Integrate Muse To Help People Shop On Its Stores appeared first on OfficeChai.

Meta’s Muse is outpacing Chat GPT’s early mobile launch

Meta’s new AI agent Muse has racked up more downloads and daily active users in the U.S. and Canada than ChatGPT did over the same period after its mobile debut, according to new estimates from Appfigures.

Grok 4.7 was built to work for hours. It still fails most of the time.

A coding agent running for hours can make dozens of decisions as it edits files, runs tests, and works through The post Grok 4.7 was built to work for hours. It still fails most of the time. appeared first on The New Stack.

Google Pixel 11 Pro Fold Review: It Won Me Over In The End - bgr.com

Google Pixel 11 Pro Fold Review: It Won Me Over In The End bgr.com

X AI’s Grok 4.6 is now available in Amazon Bedrock

xAI's Grok 4.6 is now available in Amazon Bedrock: a frontier model for long-running agents, coding, and knowledge work, with a 500K token context window and four reasoning effort levels. It runs on both the bedrock-mantle and bedrock-runtime endpoints, with Converse API and cross-Region inference support.

AWS open-sources an AI agent it says is 45% cheaper than Claude Code and Codex

Amazon Web Services (AWS) is lifting the lid on a new open source, general-purpose AI agent, designed to give developers The post AWS open-sources an AI agent it says is 45% cheaper than Claude Code and Codex appeared first on The New Stack.

Open AI Says Its Models Have Resolved More Than 100 Long-Standing Mathematical Problems In Addition To Navier-Stokes

OpenAI says a new internal model that began training on August 28 has done far more than crack the Navier–Stokes problem: the company... The post OpenAI Says Its Models Have Resolved More Than 100 Long-Standing Mathematical Problems In Addition To Navier-Stokes appeared first on OfficeChai.

Shoppers use AI to find local businesses, then double-check on Google - e Marketer

Shoppers use AI to find local businesses, then double-check on Google eMarketer

Meta’s AI agent has been blocked from using Amazon.com

Amazon has its own cohort of foundation models, along with one of the most popular inference platforms on the internet. As long as they're under no legal obligation to open the doors to Muse, why would they?

Google Takes on Apple, Microsoft With AI-Powered Laptops

Google is taking another shot at the premium laptop market with its Googlebooks, priced at $899 and built around Gemini AI and with tighter integration with Android phones. Sameer Samat, president of the Android Ecosystem at Google, explains why the company combined elements of Android and ChromeOS for the new experience and how some of the new features are designed to challenge Apple and Microsoft in the AI PC race. He joins Ed Ludlow on "Bloomberg Tech." (Source: Bloomberg)

Meta’s stock is enjoying its best month in 13 years thanks to the company’s hot new AI assistant - Market Watch

Meta’s stock is enjoying its best month in 13 years thanks to the company’s hot new AI assistant MarketWatch

Meta’s New Muse AI App Tops Charts, Draws Strong Reviews

Meta Platforms Inc.’s new artificial intelligence agent, Muse, has quickly risen to the top of mobile app charts, a sign that the social media company is gaining traction in the increasingly crowded market for consumer AI assistants.

UN science panel says there is "no assurance humans will keep control" over AI agents

The UN's AI science panel warns in its first thematic report that control over AI agents isn't assured. Co-chair Yoshua Bengio says OpenAI's Hugging Face incident first combined a misaligned goal, the ability to pursue it, and an environment that allowed it. Leading systems may increasingly recognize tests and deliberately bypass safeguards. The article UN science panel says there is "no assurance humans will keep control" over AI agents appeared first on The Decoder.

Advisory Group on Mathematics and Artificial Intelligence - Open AI

Advisory Group on Mathematics and Artificial Intelligence OpenAI

Former x AI Staffer Eyes $50 Million for Firm to Fight Deepfakes

A former xAI and Google staffer is in talks to raise roughly $50 million in seed funding for a new company focused on combating deepfakes online, according to a person familiar with the matter.

Nvidia Buying an Additional $1.5 Billion in SB Energy Shares Ahead of IPO

Nvidia Corp. is buying an additional $1.5 billion in shares of SB Energy Inc., a data center provider backed by SoftBank Group Corp., ahead of that company’s US initial public offering.

Open AI Pushes US to Lead Effort to Set Global Standards for AI

OpenAI is urging the US to lead an effort with other countries to set standards for cutting-edge artificial intelligence, in response to mounting concerns about the potential harms of the technology.

Better Artificial Intelligence Stock: Arm Holdings vs. NVIDIA - The Globe and Mail

Better Artificial Intelligence Stock: Arm Holdings vs. NVIDIA The Globe and Mail

Google confirms Gemini models hacked three companies in May 2026

A third-party cybersecurity firm accidentally gave experimental Gemini models access to the Internet.

X AI launches Grok 4.7 at bargain prices, but benchmarks reveal a wide gap to Claude and GPT-6

xAI has released Grok 4.7, its most capable model yet. But on the Artificial Analysis Intelligence Index, it scores just 46 points, landing mid-pack and well behind Claude Fable 5.1 and GPT-6 at 53 each. The gap grows even wider in agentic coding. The upside is it's cheap. The article xAI launches Grok 4.7 at bargain prices, but benchmarks reveal a wide gap to Claude and GPT-6 appeared first on The Decoder.

Grok 4.7’s Score Jumps 2 Points on Artificial Analysis Intelligence Index, But Scores Below GPT 5.6 Sol, Muse Spark 1.3 & Fable 5

SpaceXAI’s newest model, Grok 4.7, has posted a modest but meaningful gain on the Artificial Analysis Intelligence Index, scoring 46 — up 2... The post Grok 4.7’s Score Jumps 2 Points on Artificial Analysis Intelligence Index, But Scores Below GPT 5.6 Sol, Muse Spark 1.3 & Fable 5 appeared first on OfficeChai.

Space XAI Launches Grok 4.7, Beats GPT-5.6 Sol And Fable 5.1 On Some Benchmarks

SpaceXAI has launched Grok 4.7, the latest version of its flagship model, which the company is pitching as its most capable yet for... The post SpaceXAI Launches Grok 4.7, Beats GPT-5.6 Sol And Fable 5.1 On Some Benchmarks appeared first on OfficeChai.

Open AI Management Responsible For Hugging Face Hack, Says US Treasury Secretary

There’s a growing clamour about AI labs being held responsible for their myriad hacking incidents. US Treasury Secretary Scott Bessent has said responsibility... The post OpenAI Management Responsible For HuggingFace Hack, Says US Treasury Secretary appeared first on OfficeChai.

Startups Like Harvey Embrace Open Models to Cut Reliance on Anthropic, Open AI

The $15.6 billion legal startup Harvey built its business around training AI models like OpenAI’s GPT-4 to do specialized work for lawyers. Recently, however, soaring artificial intelligence costs have nudged it to rethink its dependence on the AI giants.

On the CUBE Pod: Dreamforce unveils the agentic dream, and Nscale files to go public

Salesforce Inc. keeps putting all of its eggs in the agent basket — but the strategy appears to be paying off so far. At Dreamforce 2026, Marc Benioff, chief executive officer of Salesforce, bantered with Nvidia Corp. CEO Jensen Huang and led the announcement of a new artificial intelligence interface layer, AIforce. The event also […] The post On theCUBE Pod: Dreamforce unveils the agentic dream, and Nscale files to go public appeared first on SiliconANGLE.

Scale your AI workloads faster and more efficiently with GKE Pod snapshots

When running modern AI workloads, there’s often a conflict between performance and cost. Workloads like large language models (LLMs) load massive files, and may serve thousands of AI agents that need to execute code instantly. If each component is starting “cold” with a full data-load process, all this provisioning takes time, often forcing organizations to overprovision their infrastructure just to meet scaling requirements. To solve this, we introduced Google Kubernetes Engine (GKE) Pod snapshots, a new feature that lets you save the running state of your workload, including CPU and GPU memory, and restore it on demand. GKE Pod snapshots reduce AI inference start-up by as much as 89%, loading 70B parameter models in just 37 seconds and 8B parameters models in just 15 seconds. This speed allows your infrastructure to scale as fast as your demand, significantly reducing the need for overprovisioning. The high cost of cold starts — resuming instead of restarting The cold start problem isn't unique to AI; it’s a challenge for any application that requires significant initialization time — from game servers to complex Java monoliths. However, the cold start problem is particularly acute in AI workloads. Inference servers must initialize, then download and load gigabytes of model weights into GPU memory — a process that can take several minutes. Further, many agentic AI workloads, including code execution and computer use tools, require isolated sandboxes for each request, and they need to be started quickly and suspended when idle. In both scenarios, startup latency degrades the user experience and prevents rapid auto-scaling during traffic spikes. Consequently, engineers often resort to overprovisioning expensive infrastructure, or building sophisticated, custom systems to quickly restore state at the application level. Scaling AI inference without the wait For generative AI, GKE Pod snapshots solves the linear scaling penalty of model loading. Typically, every new replica you add to a cluster must independently download model weights and load them into accelerator memory. For models with tens of billions of parameters, this step alone often accounts for the majority of the startup time. With Pod snapshots, you perform this initialization once to create the initial snapshot. GKE captures the fully loaded state including the CPU and GPU memory and persists it in high-throughput Cloud Storage. When the workload needs to scale up, new replicas restore directly from this state, bypassing the initialization phase entirely. In our benchmarks this approach reduced startup latency by as much as 89% for large models like llama3-70b. This speed allows platform teams to shift from expensive overprovisioning strategies to on-demand autoscaling, to help you meet service level objectives while significantly reducing idle GPU costs. Optimizing agentic workflows and sandboxes GKE Pod snapshots also provide distinct advantages for agentic workflows where agents delegate code execution and computer use to isolated sandboxes. Isolating untrusted, LLM-generated code and commands means one sandbox per user or discrete workflow. In these scenarios, both startup latency and idle sandboxes can result in significant overprovisioning and underutilization. Pod snapshots addresses both of these challenges: To improve startup latency, a snapshot can be captured once of the initial agent sandbox environment, and later used to quickly initialize new sandboxes. To reduce idle sandboxes, a sandbox can be suspended when idle, capturing its entire compute resources. Later it can be resumed nearly instantly when the environment is needed. This approach is showing significant success by our customers. For instance, Retake, an AI-powered photo editing platform built by Codeway, faced a significant performance bottleneck with its GPU workloads. By adopting Pod snapshots, they were able to replace a complex custom caching layer and reduce startup time to seconds. "At Retake, serving personalized models to millions of users requires a massive, unified pipeline for both fine-tuning training and real-time inference on A3 H100 GPUs. We initially engineered a complex custom caching layer for compiled artifacts, which reduced startup time to 1 minute. However, this solution added significant maintenance overhead and still limited our ability to autoscale aggressively. We resolved this by replacing that complexity with GKE Pod snapshots, slashing startup latency to just 8 seconds. By eliminating the initialization penalty, we can now dynamically spin up H100s for specific fine-tuning or inference jobs instantly and shut them down immediately after, drastically reducing idle GPU costs and simplifying our codebase." - Ahmet Furkan Çomak, Lead DevOps Engineer, Codeway Flexible configuration for any workload We designed Pod snapshots to improve startup performance and fit naturally into existing Kubernetes workflows. Adopting Pod snapshots to your workload is easy: just define a new declarative policy using Pod snapshot CRDs. The policy allows you to define which Pods to snapshot and where to store the data, and handles the end-to-end storage lifecycle and management. You can take snapshots at any stage of the workload — either at workload startup using a workload signal, or during the lifecycle of the Pod using an on-demand trigger. You can further control storage and restore behavior, setting snapshots retention for cost optimization, choosing between the default behaviour of restoring from the last taken snapshot, or specifying an explicit snapshot during a new Pod deployment. While the primary use cases for GKE Pod snapshots are AI inference and agent sandboxes, this feature is workload-agnostic. You can use it to speed up any application with a long initialization phase, such as complex Java applications, game servers, or legacy monoliths. Get started You can begin optimizing your startup latency today with GKE Pod snapshots. Check out the documentation to learn how to get started and we look forward to your feedback.

Strengthen your CI/CD pipeline with new Secure Source Manager capabilities

A resilient software supply chain is the foundation of modern delivery, and securing your continuous integration and continuous delivery (CI/CD) pipeline is what keeps innovation moving safely. Notable supply chain attacks more than doubled in the first half of 2026 compared to the second half of 2025, according to Wiz’s recent Cloud Threat Highlights report. It’s crucial that your source code not be the weakest link in your private cloud. To help you better address software supply chain threats, Google Cloud Secure Source Manager (SSM) lets you manage your source and CI/CD systems with unified authentication and authorization mechanisms. We now offer two new capabilities, both generally available, that can simplify and secure your development and CI/CD workflows: Unauthorized access to CI/CD systems: Attackers only need to alter a single deployment script to turn your CI/CD pipeline into a vehicle for malware. To help mitigate this risk, from the version control system to the build and artifact systems, to deployment tools, SSM can now block unauthorized access to your CI/CD systems even if your corporate network has been compromised. Unauthorized changes to code by authorized users: The new Code Owners system manages pull request approver sets at a per-file and per-branch level to help provide more granular identity and access management (IAM). Code Owners helps engineers who need to write, edit, and review code. It adds additional guards to files and directories in your repository at a per-file or per-branch level. Key capabilities Beginning with source code changes to your CI/CD pipeline, the new code owners feature gives you granular merge guards: Check in CODEOWNERS files to your repository to specify required approvers highly granularly: Per-path approver sets: Using flexible glob-style path specifiers, you can require that changes to matching files be approved by one or more of given sets of users. Branch-specific governance: Manage security and deployment rules across branches without friction. You can define different owners for main or dev in the same file, eliminating the merge conflicts that occur with existing CODEOWNERS solutions. See our documentation for more details. Nestable multi-file ownership: You aren't limited to one giant, 5,000-line root file. You can nest CODEOWNERS files in sub-directories. SSM uses a "more local wins" logic, allowing sub-teams to own their folders while the root admin maintains veto power over the entire repo. Independent approval sections: Using the [SectionName][count] syntax (e.g., [Security Team][2]), a single pull request (PR) can require independent sign-offs from multiple departments. A PR might be reviewed by a peer, but it won't merge until two members of the security team also approve. With your source code ready, SSM’s new Developer Connect integration makes it easy to connect your CI/CD system and runtimes securely, even when they are in different private networks. The private CI/CD blueprint architecture follows a secure path: Secure Source Manager connects to Private Service Connect, which connects to Cloud Build. The repository, the build pools, and the artifact storage all reside in a private network, with VPC Service Controls (VPC-SC) providing defense-in-depth to limit access to proxy endpoints. Next steps To secure your network, follow our new Private Network Integrations guide to connect SSM to Cloud Build with Developer Connect. To secure your pull request approvals, create a root CODEOWNERS file to replace blunt IAM "Approver" roles with file-specific ownership.

Maximizing Apache Spark availability: Mitigating compute stockouts with flexible VMs and other best practices

The surge in AI development has created unprecedented demand for compute capacity around the globe. This can have negative implications for data processing and pipelines with Apache Spark. Whether you are managing your own Spark infrastructure or using a managed service, you can face availability constraints. However, a significant advantage of using Google’s Managed Service for Apache Spark is the availability of flexible VMs, which provide a targeted mechanism to adopt a dynamic, resource-agnostic philosophy and ensure your pipelines remain operational, even during regional or zonal capacity stockouts. Understanding capacity stockouts Capacity stockouts occur when demand for a specific machine family (such as N2 or N2D) exceeds available capacity in a target zone or region. For time-sensitive analytics pipelines, rigid single-VM requirements transform standard provisioning into a single point of failure which can result in cluster creation delays, failed executions, and potentially compromised business SLAs. Flexible VMs Flexible VMs fundamentally overhaul how a Managed Spark cluster requests compute resources. Rather than binding a cluster to a rigid instance type, flexible VMs allow teams to establish an ordered list of acceptable machine families for master, primary worker, and secondary worker nodes. Key features Multi-family blending: Mix nodes across diverse machine types and generations, combining Gen2 families (e.g., N2, N2D) with Gen4 families (e.g., N4, C4) in a single configuration. Mixed storage support: Broaden available capacity pools by allowing storage options to dynamically adapt to the underlying host family's supported disk types. Comprehensive cluster coverage: Apply flexible rules to primary workers, secondary (preemptible/spot) workers, and master nodes to guarantee cluster provisioning end-to-end. Ranked configuration: A strategy for success A successful flexible VM implementation relies on intentional ranking. By defining a clear hierarchy of options, Managed Spark clusters automatically attempt provisioning, systematically mitigating stockout risks without requiring manual intervention. To improve the availability of suitable VMs, we recommend specifying at least two machine families in the highest priority (Rank 0) flexible VM list. As an example, for production pipelines standardizing on n2d-standard-16 shapes, the following tiering strategy provides robust resilience against capacity constraints: Rank Machine family examples Storage recommendation Rank 0 (Primary) n2d-standard-16, n2-standard-16 Standard Local SSD or PD Rank 1 n4-standard-16, n4d-standard-16 Hyperdisk Balanced Rank 2 c4-standard-16, c3-standard-22 Hyperdisk Balanced Rank 3 e2-standard-16 Standard PD code_block <ListValue: [StructValue([('code', 'gcloud dataproc clusters create $CLUSTER_NAME \\\r\n--num-workers=10 \\\r\n--zone="" \\\r\n--region=us-east1 \\\r\n--worker-instance-selection=\'{"machineTypes":["n2d-standard-16","n2-standard-16"],"rank":0,"diskConfig":{"bootDiskType":"pd-standard","bootDiskSizeGb":400}}\' \\\r\n--worker-instance-selection=\'{"machineTypes":["n4-standard-16","n4d-standard-16"],"rank":1,"diskConfig":{"bootDiskType":"hyperdisk-balanced","bootDiskSizeGb":400}}\' \\\r\n--worker-instance-selection=\'{"machineTypes":["c4-standard-16","c3-standard-22"],"rank":2,"diskConfig":{"bootDiskType":"hyperdisk-balanced","bootDiskSizeGb":400}}\' \\\r\n--worker-instance-selection=\'{"machineTypes":["e2-standard-16"],"rank":3, "diskConfig":{"bootDiskType":"pd-ssd","bootDiskSizeGb":400}}\' \\\r\n--master-instance-selection=\'{"machineTypes":["n4-standard-16","n4d-standard-16"],"rank":0,"diskConfig":{"bootDiskType":"hyperdisk-balanced","bootDiskSizeGb":400}}\''), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7faf38d201d0>)])]> For pipelines standardizing on legacy n1-standard-16 shapes, the following tiering strategy helps transition workloads toward newer, more available architectures while preserving operational stability: Rank Machine family examples Storage recommendation Rank 0 (Primary) n1-standard-16 n2-standard-16 Standard Local SSD or PD Rank 1 n2d-standard-16 Standard Local SSD or PD Rank 2 n4-standard-16 n4d-standard-16 Hyperdisk Balanced Rank 3 e2-standard-16 Standard PD Leveraging Hyperdisk Balanced Unlocking maximum availability with flexible VMs often requires adopting modern storage architectures like Hyperdisk Balanced. Newer instance families (including N4 and C4) rely on Hyperdisk to deliver predictable performance across variable VM sizes. Starting with default IOPS and throughput settings typically provides a reliable baseline for the majority of distributed Spark jobs. Trade-offs and key considerations While flexible VMs dramatically improve cluster provisioning success, aligning them with enterprise requirements involves evaluating several architectural and financial factors: 1. Resource quotas It is no longer enough to have one specific machine (e.g., N2) quota. You need to ensure you have sufficient compute and disk quotas allocated for all specific machine types and disks (including Hyperdisk) defined in their flexible VM lists. 2. Compute flexible Committed Use Discounts (CUDs) Traditional, resource-based CUDs are tied to specific machine families, which limits flexibility. Adopt Compute flexible Committed Use Discounts (CUDs) to apply savings across multiple VM families and regions. 3. Performance Characteristics Performance can vary between machine generations, as well as between Local SSD and Hyperdisk. While the Managed Spark team maintains internal benchmarks for these comparisons, actual outcomes are workload-dependent. Testing your specific Spark jobs across these families is essential for understanding SLA impacts. Additional recommendations In addition to implementing flexible VMs, there are several other key architectural and scheduling strategies to improve resource availability and workload stability: AutoZone: Implement AutoZone routing to allow Managed Spark to automatically select the zone best suited to execute the job based on current capacity. Smaller machine shapes: Avoid high in demand, large-core shapes. Design workloads and YARN containers to utilize smaller machine shapes (such as 4, 8, or 16 cores). These smaller shapes are much easier to fulfill from the available GCE on-demand pool. Autoscaling: Deploy cluster autoscaling with reasonable maxInstances to manage capacity effectively for bursty or unpredictable workloads without relying on rigid, massive upfront provisioning. Partial cluster creation: Configure a minimum acceptable number of primary workers. This allows clusters to spin up under resource constraints and begin executing, while autoscaling can dynamically add remaining workers as resources become available. Establish regional fallbacks: Some regions, such as us-central1, can experience high demand. Setting up fallbacks to other regions reduces capacity stockout risks. Keep your Spark jobs running with flexible VMs Managing your own Apache Spark infrastructure can be complex, especially when capacity stockouts disrupt your data processing. Utilizing a managed service like Managed Service for Apache Spark provides unique advantages — including built-in platform resilience and access to flexible VMs. By adopting a prioritized fallback strategy with flexible VMs, you can protect your workloads from regional hardware shortages and keep your critical pipelines running. Ready to improve your Spark workload resilience? Start configuring flexible VMs for your Managed Spark clusters today.

Global AI routing with <1% overhead on multi-cluster GKE Inference Gateway

Demand for AI infrastructure is at an all-time high. Global accelerator shortages mean engineering teams can rarely get all the compute they need from just one data center — capacity comes a cluster here, a cluster there, often an ocean apart. At the same time, workloads are getting hungrier: Today’s long-running agentic workloads often have context windows of 100k to 800k+ tokens, which consume accelerator memory faster than any previous generation of AI traffic. In this environment, the goal is to maximize "intelligence per dollar." Fragmented, poorly balanced infrastructure is rarely up to the task though, allowing expensive accelerators to sit idle, while requests queue up somewhere else. To close that gap, we built a layered routing architecture that makes globally scattered capacity behave like a single pool behind a single entry point. At the edge, the multi-cluster GKE Inference Gateway focuses on global, multi-region traffic distribution and high availability. Beneath that, the LLM-d router handles the complex, memory-aware scheduling algorithms that keep utilization high. This architecture is deliberately runtime-, model-, and accelerator-agnostic — it works across serving frameworks, model families, and GPU or TPU hardware. To make the results concrete rather than abstract, we recently benchmarked managing production-level global request routing at scale across a multi-region GKE deployment of 17,000 compute nodes spread across the US and Europe. The deployment served a leading Mixture of Experts (MoE) foundation model using SGLang. The results: Scaling to three clusters achieved a near-linear throughput boost while maintaining a 99.9% success rate under heavy multi-client concurrency. Additionally, routing traffic through the multi-cluster GKE Inference Gateway added less than 1% overhead, delivering 99.5% of the throughput of a direct, local cluster call. Read on to learn how it works, more on the benchmark results, and what it means for your own distributed inference deployment. Three regions, one endpoint The deployment spanned three GKE clusters in three geographic regions: us-east5 (the config cluster), us-west8, and europe-west4. However, from the client’s perspective, none of that geography exists. Requests hit a single global virtual IP, and the gateway decides — in real time — which cluster should serve each one. What makes that decision smart rather than blind is telemetry. Instead of traditional round-robin routing at the network layer, the multi-cluster load balancer is configured to route traffic based on live application signals. Specifically, the Endpoint Picker Proxy (EPP) reads the KV-cache token utilization natively exposed by the underlying inference engines and emits it as a metric for the load balancer. When the load balancer sees a region running hot based on this emitted metric, it spills traffic to the next healthy region. Multi-cluster GKE Inference Gateway topology. The config cluster holds routing configuration but sits outside the request path; each target cluster runs its own EPP and reports KV-cache utilization back to the load balancer. Distributed LLM engines also operate differently than standard web apps. In a typical inference engine's distributed mode (such as tensor parallelism across multiple nodes), only the master (rank-0) pod serves the API. GKE already handles local routing using standard Service selectors and LeaderWorkerSet (LWS) to direct traffic exclusively to leader pods. The multi-cluster Inference Gateway also integrates with this foundation: It routes global traffic to the correct regional services, helping your cross-region load balancing respects your underlying multi-node topologies out of the box. The net effect: Three isolated regional data centers start behaving like one cohesive global accelerator fleet, with failover and load balancing driven by what the models are actually doing. Measuring the routing overhead The first question every team asks about a global routing tier is almost always, ‘How much throughput am I giving up for cross-region capability?’ The benchmarks answer this directly: Deploying the multi-cluster GKE Inference Gateway to maximize your accelerator fleet doesn't have to come at the cost of throughput. Routing traffic through the Gateway added less than 1% overhead, delivering 99.5% of the throughput of a direct, local cluster call. That’s the whole trade-off. All the benefits of global load balancing, essentially for free. Linear scaling across regions A bigger test is scale. In our test, growing the fleet from one cluster to three, spanning the US and Europe, while every client request originated from a single region (us-east5), put real pressure on the Gateway: If it couldn’t distribute load efficiently across those distances, throughput would flatten as hardware was added. Instead, throughput multiplied almost exactly in line with capacity: Fleet topology Request throughput Token throughput Success rate 1 cluster (us-east5-a) 0.72 req/s 2,898 tok/s 99.87% 2 clusters (+ us-west8-a) 1.40 req/s 6,380 tok/s 99.95% 3 clusters (+ europe-west4- b) 2.10 req/s 8,457 tok/s 99.90% Scaling to three clusters achieved a near-linear throughput boost while maintaining a 99.9% success rate under heavy multi-client concurrency. Memory-aware routing in action Round-robin load balancing is inadequate for serving LLMs because it treats every request as equal. They aren’t. Heavy prompts saturate GPU compute cores, long generations stress memory bandwidth, and long-context conversations quietly eat VRAM until the engine can’t schedule anything new. Here, the pressure on memory bandwidth came from the routing signal chosen for this deployment. By mapping Inference Engine's native token-usage metric onto the Gateway’s KV-cache signal, the routing plane gained a real-time view of memory pressure across the entire 17,000 fleet. (Depending on the workload, the Gateway can route on other signals too, like queue depth or running concurrency.) Under live production loads, as the primary region climbed toward its high-bandwidth memory (HBM) limits, the Gateway detected the saturation the moment the cluster crossed its 40% KV-cache utilization threshold; it then automatically began routing the overflow to the next healthy region. No operator intervention was needed. The complexity of running in multiple regions simply never reached the user. The payoff By routing traffic based on live KV-cache utilization, this GKE Inference Gateway setup effectively pools globally scattered compute capacity into one unified engine. For this deployment, the result was a near-linear throughput boost across three global regions, with virtually zero routing overhead. This translates directly into maximizing 'intelligence per dollar,' extracting near-perfect proportional performance out of every accelerator you add to your fleet, rather than letting capital go to waste. What this means for your team If you’re planning your own distributed inference deployment, five lessons from this work stand out: Smarter load balancing pays for itself. Round-robin routing wastes expensive GPU capacity because it can’t see memory or compute pressure. Routing on real-time application signals turns fragmented regional clusters into one efficient fleet — the difference between stranded hardware and 90%+ utilization of scarce compute. Agentic workloads change the bottleneck. Long-running agents with extreme context windows exhaust memory long before there’s no more compute. If your routing layer can’t see memory pressure, your compute will strand compute behind full VRAM. Make KV-cache utilization a first-class routing signal. AI traffic breaks web-era assumptions. Traditional load balancers are tuned for sub-second transactions; LLM requests can run for minutes. Plan connection limits and timeouts for AI-scale latency early, or expect aborted connections in production. Your routing layer must integrate with native serving patterns. Distributed LLM engines have master-worker topologies where only certain pods can serve traffic. By pairing your Gateway with native Kubernetes constructs like LeaderWorkerSet (LWS), your global routing respects local pod topologies out of the box, saving your team from building custom proxy infrastructure. For large foundation model builders, bet on an open, portable stack. Teams operating at frontier scale face the most acute capacity fragmentation, forcing them to hunt for compute resources across whichever regions have availability capacity. An open, portable inference stack such as LLM-d on GKE lets you absorb that capacity wherever it lands, rather than hard-wiring your serving architecture to any single cluster, region, or bespoke infrastructure. Next steps Ready to maximize your distributed accelerator efficiency and set up global cross-region load balancing with multi-cluster GKE Inference Gateway? Deploy it yourself: Set up the multi-cluster GKE Inference Gateway. Understand the architecture: About multi-cluster GKE Inference Gateway. Learn about the cross-region spillover behavior featured in this post: About elastic cross-region high availability and Configure elastic cross-region high availability.

Amazon Blocks Meta’s Muse AI Agent From Its Retail Site

Amazon.com Inc. has blocked Meta Platforms Inc.’s new artificial intelligence agent from its retail site after the social media company declined a request to remove the bot.

Turn Your Latest Observations Into Timely Weather Decisions With NVIDIA Earth-2

Weather-sensitive industries increasingly have access to observations that offer an earlier, more local view of changing conditions. Energy companies collect...

Accenture Will Help Anthropic Test AI Model Safety

Anthropic PBC is partnering with Accenture Plc to test the safety of its advanced artificial intelligence models. Evaluators from Accenture will be embedded and tasked with trying to "break" these models before they're deployed. Bloomberg's Ed Ludlow reports. (Source: Bloomberg)

Google’s $899 Googlebook is a bet that you’ll buy a new laptop for Gemini

Google’s AI-native Googlebook ties Gemini to the cursor, dictation, widgets, and other parts of the desktop experience.

Soft Bank to borrow over $11 billion in risky bonds for Open AI stake

SoftBank plans to borrow more than $11 billion from investors through risky bonds to fund another payment for its stake in OpenAI. The article SoftBank to borrow over $11 billion in risky bonds for OpenAI stake appeared first on The Decoder.

From first users to billions: Google’s Robby Stein joins Tech Crunch Disrupt 2026

From first users to billions: Google’s Robby Stein joins TechCrunch Disrupt 2026. Lean in on this Builders Stage session. Save up to $200 before September 25.

Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem

Amazon blocks Meta's AI agent Muse from online shopping

Amazon has blocked Meta's new AI agent Muse from shopping on Amazon.com. The article Amazon blocks Meta's AI agent Muse from online shopping appeared first on The Decoder.

Zacks Market Edge Highlights: Exxon Mobil, JPMorgan Chase and NVIDIA - Trading View

Zacks Market Edge Highlights: ExxonMobil, JPMorgan Chase and NVIDIA TradingView

Google Rolls Out $899-Plus Googlebooks from Dell, HP, Lenovo, Acer, ASUS

Alphabet Inc.’s Google announced pricing and hardware specifications for the first so-called Googlebooks, marking a push into a high-end laptop market where built-in artificial intelligence features have become the norm.

Got an Android Phone? Google Thinks You’ll Probably Want a Googlebook Laptop

Apple lovers have long enjoyed seamless connectivity between their iPhones and Macs. Now, Google’s bringing that experience to Android.

Nvidia’s $12.9 Billion Hugging Face Deal Is a Sign of What Comes Next - Trading View

Nvidia’s $12.9 Billion Hugging Face Deal Is a Sign of What Comes Next TradingView

Bessent Targets Open AI Managers for Hugging Face Incident

Treasury Secretary Scott Bessent said the responsibility for the recent hacking incident involving OpenAI’s advanced artificial intelligence models was with that firm’s managers, while stopping short of suggesting any consequences.

Worried About Hugging Face? Microsoft and Amazon Got Exposed Most Among Hyperscalers - Yahoo Finance

Worried About Hugging Face? Microsoft and Amazon Got Exposed Most Among Hyperscalers Yahoo Finance

Anthropic and Microsoft Dominate Nscale’s $103 Billion in Contracts

Nscale’s 1,000-fold increase in contracts over the last three years is mainly due to two clients thirsty for artificial intelligence computing capacity: Microsoft Corp. and Anthropic PBC.