Thought Leadership
An Inference Factory is an integrated stack that runs trained AI models in production inside an organization’s own data center. Scality introduced one on October 8, 2026. This article explains what the term means, how the technology works, which competitors exist and what Gartner, Forrester and IDC have to say.
What Is an Inference Factory?
The term Inference Factory is not a formal standard but a marketing label that vendors fill in differently. It derives from the “AI factory” that NVIDIA popularized. According to the NVIDIA glossary, this is specialized computing infrastructure covering the entire AI lifecycle, from data ingestion through training and fine-tuning to high-volume inference. Its product is intelligence, measured in token throughput.
An Inference Factory concentrates on the last step. Inference is the phase after training in which a model processes inputs and produces outputs, as NVIDIA describes it. An Inference Factory bundles four building blocks: models, a serving layer that distributes requests across GPUs, a control plane for access, billing and scheduling, and fast storage for models, data and intermediate state. The factory idea is decisive here: the stack is delivered and maintained as a verified whole, rather than organizations assembling individual parts themselves.
Why Inference Is Moving to the Center
The analyst figures show the shift. Gartner expects worldwide spending on AI-optimized infrastructure as a service of USD 42.3 billion in 2026, an increase of 96 percent. According to Gartner, as reported by it-daily.net, USD 23.3 billion of this goes to inference and USD 19 billion to training. Inference accounts for 55 percent in 2026 and is set to reach 59 percent in 2027. Gartner names agentic AI among the drivers, because multistep, autonomous workflows increase the compute load.
For organizations that use AI in production, this raises three questions. First, cost: those who pay per token pay more with every additional workflow, and spending is hard to cap. Second, control: cloud providers can change model versions and quantization, which can affect workflows built on top. Third, data sovereignty: where are prompts, documents and code processed, and which jurisdiction applies? Scality makes these arguments in its announcement. Surveys point the same way: according to the Broadcom Private Cloud Outlook 2026, 56 percent of the surveyed enterprises run or plan to run production AI inference on private cloud, while the share of public cloud as the primary environment fell from 56 to 41 percent within a year. 1,800 IT decision-makers were surveyed. Because Broadcom itself sells private cloud software, the study should be read as interest-driven.
The Technology: Prefill, Decode and KV Cache
To understand what an Inference Factory does, it helps to look at how a request is processed. It has two phases. During prefill, the model processes the entire prompt. This is compute-intensive and determines the time to first token. During decode, the model generates the answer token by token. According to the slides of the DistServe study, this phase is memory-bound.
During both phases the model builds up an intermediate store called the key-value cache, or KV cache. It holds the context processed so far. With long documents, reasoning models and agents, it quickly outgrows the limited high-bandwidth memory (HBM) of the GPU. If context is lost, the GPU has to compute it again. This is where disaggregation comes in: prefill and decode run on separate GPU pools, and a shared cache keeps the context available.
The idea is well supported scientifically. The study DistServe, presented at OSDI 2024, separates prefill and decode and thereby served up to 7.4 times more requests or met 12.6 times tighter latency targets than comparison systems, keeping more than 90 percent of requests within the limits. The serving system Mooncake from Moonshot AI adds a distributed KV cache. According to its authors, this lets the Kimi service handle 75 percent more requests under real load. Both results come from the authors’ research and production environments and cannot simply be transferred to every organization.
The Scality AI Inference Factory in Detail
Scality, known for object storage, markets the AI Inference Factory as a supported alternative to cloud AI services for enterprises, government agencies and neo-cloud providers. According to the announcement, the stack consists of four parts:
- validated open-weight models that are maintained as part of the stack,
- disaggregated inference serving, in which prefill and decode scale separately,
- a control plane for authentication, billing, routing and scheduling against SLA targets,
- Scality AI Data Infrastructure (ADI) as storage for models, enterprise data and inference state.
ADI serves as the shared cache layer. It extends the KV cache beyond GPU memory and, according to the vendor, provides a cache at multi-petabyte scale that GPUs can access with latency in the same order of magnitude as GPU memory. A single namespace distributes data across TLC flash, HDD and optionally tape. Data transfer runs directly between storage server and GPU via RDMA.
Scality reports several measurements from its own tests. These are vendor claims; independent benchmarks are not available:
- Loading Gemma-3 27B takes 1.9 seconds, nearly ten times faster than from local NVMe.
- Restoring a 14,000-token context from ADI yields a time to first token of 166 milliseconds, 83 milliseconds slower than from HBM.
- Retrieving the KV cache is 14 times faster than recomputation at 14,000 tokens and up to 72 times faster at 439,000 tokens.
- A KV cache roughly 80 times larger than the memory of a single GPU holds up to 1,000 parallel sessions without recomputing.
- 97 percent of network bandwidth is reached between GPUs and storage.
According to Scality, the software base is open as well. Every delivered component is provided as open code, so customers can trace how inference state is stored and transferred and submit contributions for review. The stack is compatible with frameworks such as OpenCode, Hermes, Goose, LangGraph and Pydantic AI. It supports validated models from Mistral, Gemma, gpt-oss, Qwen, Kimi, GLM and DeepSeek on standard servers from Dell, HPE, Lenovo and Supermicro. The solution is available as a software license or as a fully managed service. Scality showed a reference architecture on October 8 at Scality Day in Paris.
The Competition at a Glance
Scality is not alone. The market for KV cache offloading and disaggregated serving is contested, and most storage vendors connect to the NVIDIA Dynamo inference framework. Dynamo separates prefill and decode, has a KV-aware router and a distributed KV cache manager, and transfers data via the NIXL library. According to NVIDIA, Dynamo can offload the KV cache to CPU RAM, local SSDs or network storage. According to Blocks & Files, vendors including Cloudian, DDN, Dell, Hitachi Vantara, HPE, IBM, NetApp, Pure Storage, VAST Data and WEKA support this technology.
The difference lies less in the basic idea than in the scope of delivery. Most competitors position themselves as a data platform or cache layer within a stack that customers assemble themselves around Dynamo, vLLM or LMCache. Scality, by contrast, sells the validated overall stack including maintenance. Those who prefer to steer the integration work themselves have more room with the alternatives. Those who want to outsource it find a package at Scality.
What Do Gartner, Forrester and IDC Say?
The analysts view the trend toward local inference positively overall, but call for differentiation.
Gartner: In the study “How to Determine Infrastructure Requirements for On-Premises Generative AI” (March 5, 2025, ID G00825318), Gartner expects more than 20 percent of enterprises to run AI workloads for training or inference locally by 2028, up from around 2 percent in early 2025. The quote is relayed via a DDN blog post; the study itself is paywalled. A Gartner abstract on on-premises AI infrastructure points out that low GPU utilization and weak cost efficiency often occur in LLM inference there. In June 2026, Gartner also devoted a briefing to the topic “The Case for On-Premises AI Inference”.
Forrester: In its Predictions 2026, Forrester predicts that private AI factories will reach 20 percent adoption and on-premises servers will capture 50 percent share. How both are defined is stated in the paywalled report. According to IT Brief, Forrester also expects at least 15 percent of enterprises to seek private AI options instead of placing corporate data in public cloud models.
IDC: Nataliya Yezhkova, Vice President Storage and Data Management, points out in the Scality announcement that IDC research rates local inference as practicable for a growing number of enterprise and public sector workloads. The prerequisite, she says, is an infrastructure that can store and serve model state efficiently and at scale. The statement comes from a vendor announcement, and the underlying IDC study is not named there.
Futurum Group: The January 2026 report AI Inference: Enterprise Infrastructure and Strategic Imperatives no longer sees cloud-first as the default. Local inference offers the lowest cost per inference for stable, high-volume workloads.
A Critical View: What Organizations Should Watch
The announcement offers many arguments but leaves questions open. The measurements come exclusively from Scality, and the accompanying blog post with the test details could not be retrieved when this article was written. The announcement gives neither prices nor license terms, and it does not explain under which license the “open code” is released. The term is not equivalent to open source in the sense of the Open Source Initiative. The approach also requires GPU servers and a fast RDMA network, which demands investment and operational know-how. The Gartner abstract on weak GPU utilization shows that local inference is only economical if the hardware stays busy.
The surveys deserve a second look as well: the Cloudian survey of 203 IT decision-makers, according to which 93 percent have already repatriated AI workloads or are evaluating it, was commissioned by a storage vendor. The data show a clear shift of inference, but no complete retreat from the cloud. Hybrid models are realistic, in which critical, stable workloads run locally and load peaks run in the cloud.
Conclusion
The Inference Factory describes less a new product than an architectural idea: run models locally, separate prefill and decode, and offload the KV cache to fast, shared storage. Research supports the principle, and analysts see growing importance for on-premises inference. Scality’s distinguishing feature is the delivered and maintained overall stack. Whether the approach pays off depends on your own utilization. Organizations should therefore test the vendor figures in a proof of concept with their own models and load profiles.
Q&A: Frequently Asked Questions About the Inference Factory
What is an Inference Factory?
An Inference Factory is an integrated stack of models, serving layer, control plane and storage that runs AI models in production in an organization’s own data center. The term is not a standard but derives from the AI factory.
What is the difference between training and inference?
During training, a model learns from data. During inference, it applies what it has learned to new inputs. According to Gartner, spending on inference exceeds spending on training for the first time in 2026.
What is a KV cache and why does it matter?
The KV cache stores the context of a request processed so far. If it is offloaded and reused, the GPU does not have to recompute the context. This saves compute time, especially with long contexts and agents.
What does disaggregated serving mean?
Prefill and decode run on separate GPU pools and scale independently. The DistServe study shows that this can enable up to 7.4 times more requests at the same latency targets.
Who offers comparable solutions?
Besides Scality, NVIDIA with Dynamo, WEKA, DDN, VAST Data and open projects such as LMCache and Mooncake, among others, work on KV cache offloading and disaggregated serving.
Is local inference always worthwhile?
No. According to Futurum Group, it pays off mainly for stable, high-volume workloads. Gartner points to the risk of low GPU utilization. A hybrid model is often sensible.
