AI Inference Doesn't Need New Data Centers

On July 24, 2026, Verizon's CEO told analysts that the company had converted a central office into an edge data center for AI inference as a trial and that the available capacity sold out in 24 hours. He said Verizon is now doing the same to many more of them and he described what customers were buying as power-ready, permitted locations.

I came up in this industry when the central office was where the network lived. If you've spent time in one, you know what I mean when I mention the wet cell battery room, the overhead ladder rack, the cable vault, the generator out back that gets tested once in a while (one hopes). For twenty years those buildings have been emptying out as PSTN switches retire, copper is decomm'd and the industry mostly stopped being interested in them. Now the largest US telco says it can't convert them into mini AI data centers fast enough. Something changed and I don't think it's gotten a lot of attention.

The AI infrastructure buildout keeps getting covered as a compute story. Chips, gigawatts, data centers the size of towns. But underneath all that, a networking story is forming and to me it's a story of where inference actually runs in the future, how requests get to it, what moves between sites while an answer is being generated and who operates the fabric in the middle. This piece is my attempt to lay that out, end to end. The central offices, the CDNs, the exchange layer and what a truly distributed inference internet could look like using the infrastructure we already have. Maybe we don't have to build all these new data centers that nobody wants. Maybe we can reuse what's already built. I think this is the most important infrastructure shift of my career in the wake of the biggest technological, and perhaps societal, shift the world has ever seen at the hands of AI.

Past attempts: CORD and MEC

We've been here before. Twice, at least.

Around 2016, AT&T and the open networking crowd launched CORD, Central Office Re-architected as a Datacenter. Their own paper counted 4,700 AT&T central offices, some holding up to 300 unique hardware appliances, and proposed turning each one into a small cloud pod of commodity servers and white-box switches. Around the same time, the wireless industry spent the better part of a decade on multi-access edge computing (MEC), with the same basic promise of compute near the user. Both had real engineering behind them. Both mostly fizzled though. Michael Huerta, whose startup is now building an inference grid, put it plainly this spring: Edge computing 1.0 and 2.0 "whiffed at every turn."

Why? CORD and MEC were right about the building potential but were wrong about the tenant. They asked the telco to rip out working gear and re-platform its own network functions and they never found a paying customer once it was built. A classic "build it and they will come," but they never came.

The difference now is that AI inference is a workload in search of infrastructure. The tenant comes with its own budget and the budget is huge. Cloudflare said in August that automated and AI traffic now makes up more than half of everything crossing its network and that daily AI agent requests grew 1,700% in a year. Akamai's cloud infrastructure business grew 39% last quarter with every GPU it owns sold out, while its legacy delivery revenue fell 6%. That's the traditional CDN era ending and the edge inference era starting for Akamai within the same earnings report.

The AI workload WANTS to move

Content delivery networks (CDNs) worked because most web content is static and could be infinitely copied ahead of time. Popular bytes got replicated toward users via caching and peering, hit ratios were tuned, margins were collected. An AI answer is a completely different beast because it is dynamic, computed at request time for one person or agent with one context. You can't easily put that in a static content cache server and replicate it all over the world. Some of it is still cached (semantic caching of repeated prompts, prefix caching inside the serving stack, etc.), but the majority that has to be computed is where the cost and latency sensitivity live. That residual grows as answers get personalized and the agents get more chatty.

Agents are one of the least appreciated parts of this calculation. A human in a chat pays the time-to-first-token cost once when they prompt an LLM for an answer. An agent working a coding task fires 50 to 200 model calls and each one pays the cost of the full network path again and again. Voice AI agents run on a total budget around half a second. When every call and series of interdependent calls pays the cost of the path latency, proximity stops being a nice-to-have and starts compounding in cost.

AI Schedulers as Routers?

Production AI Large Language Model (LLM) inference, the part where answers from the model are created, splits into two phases:

• Prefill - This is the model reading your prompt. It processes every input token in one big parallel pass and builds the key-value (KV) cache, which is the model's working memory of what you said. It's compute-bound so the GPU runs flat out doing matrix math as fast as it can and the phase ends when the first output token is produced. GPU speed is absolutely key for this phase. Time to first token is mostly prefill time.
• Decode - This is the model writing the answer. It generates one token at a time and for each new token it has to read the entire KV cache back out of memory. It's memory-bandwidth-bound so the GPU is mostly waiting on memory, not computing, which is why decode leaves GPUs 25 to 30 percent utilized while prefill runs them near 90 percent. The speed at which memory gets this data out is key at this phase. Inter-token latency, the pace at which words appear, is decode time.

Every serious serving stack now separates them, because keeping them together means a long prompt stalls the conversation in flight. The catch is that the KV cache prefill builds has to cross a network to reach the decode hardware before the first token can appear.

That cache is not small. For example, with Llama-3-70B it runs about 320 KB per token. A 128K-context request drags roughly 40 GB of it. Split across four GPU pairs that's about 10 GB each, and on a 25 Gbps link every one of those transfers takes about 3.2 seconds, sitting directly inside the time-to-first-token budget. An August paper on topology-aware KV transfer (TopKV) works the more ordinary case: a 70B model at 4K context moves 1.3 GB per request, which at production scale exceeds 100 GB/s in aggregate, and the author argues that existing serving systems assume uniform bandwidth between any two GPUs, which stops being true the moment you leave the rack.

The proposed fixes read like problems we've solved before in network engineering. NetKV, published in June, proposes that the network operator publish a topology tier map, per-tier bandwidth, latency and a live congestion signal built from ordinary switch telemetry (INT, sFlow, SNMP counters), with KV transfer flows marked with DSCP so they can be told apart from background traffic. The scheduler consumes that and places decode accordingly, cutting mean time to first token by up to 21% in their tests. Prefill-as-a-Service, published in April, goes further and ships long prefills to a compute-dense cluster in another data center over commodity Ethernet, keeping short and cache hit requests local. The authors note that the newer hybrid-attention architectures shrink the KV cache enough to make cross-datacenter transport practical. And NVIDIA's AI Grid control plane already routes inference on latency, sovereignty, cost and KV-cache hit rate.

Path attributes, DSCP marked flows, path selection and a control plane picking egress based on policy. The inference scheduler is turning into a router with new attributes and the state it's routing is measured in gigabytes per request. As models get distributed, the network becomes the constraint and the tools for managing it are from our toolboxes as network practitioners.

A tale of two CDNs

Akamai is buying GPUs and can't buy enough. Its inference cloud runs a fleet of NVIDIA RTX PRO 6000 GPUs in a first wave of cities on a network of 4,300 points of presence, it just signed a $600 million four-year deal with a US tech company for robotics infrastructure, it has booked more than $2.8 billion in multi-year infrastructure commitments this year and management now guides overall growth to accelerate in 2027 on the strength of it. CEO Tom Leighton's framing is that training happens in a few giant places and running the models everywhere is the next problem.

Cloudflare said something different. Matthew Prince told analysts the company won't compete in the capital-intensive GPU race and would rather sell work done on its network than commodity compute. Cloudflare's play is the AI gateway, the routing, the security and increasingly the payments layer for agent traffic. Its gross margin fell 320 basis points on the shift in traffic mix, which is what it looks like when your network starts carrying machine-to-machine load that behaves nothing like a web page.

Somebody has to own the gear and somebody also has to route the requests. Neither company has the wrong approach and the split between them is the first outline of the distributed world: compute owners and route operators, with an interconnection layer between them that doesn't fully exist yet.

So is a Central Office (CO) a good place for AI inference?

So where does the hardware for inference go once you accept it can't all go into 500 MW campuses that nobody wants in their backyard?

The candidate estate is enormous and it's emptying out.

• BT and Openreach run about 5,600 exchange buildings in the UK and serve the whole fiber footprint from roughly 1,000 of them; the rest are scheduled to close through the 2030s.
• AT&T had 4,700 central offices when CORD was written (fewer now, most likely).
• Frontier, now part of Verizon, markets edge colocation at more than 2,500 sites including its COs.
• Ziply, now owned by Bell Canada, is converting 200 plus former exchanges for colo.
• NVIDIA's own pitch to operators counts roughly 100,000 distributed network data centers worldwide with spare room for more than 100 gigawatts of AI capacity over time.
• Bill Chang, who runs Singtel's infrastructure unit, describes it as a power grid: AI data centers are the generators, the fixed network is transmission and edge locations are the substations. Converted central offices are the most obvious substations we have.

I want to be upfront about COs as I've been in enough of them in my career. A CO was built for a -48V DC battery plant, telephone switch heat and telecom floor loading, with air moving lower-front to upper-rear under Telcordia standards written decades ago. By contrast, a rack of modern inference servers wants clean three-phase AC, front-to-back airflow and 10 to 15 kW of cooling per rack before you even think about density. The pragmatic design is to stop chasing density by spreading racks across the cheap empty floor in contained pods, add rear-door heat exchangers for the tenant who insists on 30 kW and put a packaged chiller or dry cooler in the generator yard. Roughly 100 to 500 kW of inference per building is credible with an electrical upgrade. Megawatts need a different building entirely.

But what a CO does give you is a head start on the four things that take years to get elsewhere: a utility entrance, a fiber vault with diverse routes, a generator and a grandfathered zoning classification.

Not In My Back Yard (NIMBY)

The AI buildout has been underwritten on chips, power and capital. It turns out there's a fourth element and it's now the scarce one. Data Center Watch counted at least 75 projects worth about $130 billion blocked or delayed in the first quarter of 2026 alone, matching all of 2025 in three months, with organized opposition groups more than doubling to 800+ across 49 states. The three states with the most opposition groups are Maryland, Ohio and Texas. I live in Cleveland, Ohio. I didn't need the report to know about this because I see it in my own community, which has a data center construction moratorium. There's now a federal moratorium bill in Congress, a statewide one made it to a governor's desk in Maine before being vetoed and polling in Texas finds most residents don't want a data center nearby.

Notice Verizon's exact words for what its customers were seeking: power-ready and permitted. A few hundred kilowatts in a windowless building that has been utility-zoned since Eisenhower was president is a different conversation than a new substation and a rezoning hearing in front of hundreds of concerned people. Distribution doesn't magically make the electrons free and it doesn't dodge ratepayer politics. There are still things to work out. But this converts one huge impossible fight into a lot of smaller conversations, mostly about buildings that have been chugging along unnoticed for forty years.

So who runs all of this?

Follow all of this to its conclusion and you get inference running in many places owned by many parties: telco COs, CDNs, regional colos, neocloud racks a private link away and, at the far end of the distribution curve, SPAN's XFRA nodes, which put sixteen GPUs and a battery next to a residential electrical panel, draw about 19 kW each and are aiming at 80,000 nodes by 2027 if the current 100-home proof of concept holds up. Capacity that spread out is going to have some unique uses, such as resource overflow during a burst, a long prefill shipped to whoever has compute dense silicon idle or a deterministic path bought across somebody else's footprint. This is the new traffic engineering plus settlement and we already have a word for it. It's peering all over again.

The early movers are out there. DE-CIX launched what it calls an AI Internet Exchange a year ago. Megaport, which runs a network fabric across 1,100 data centers, bought a bare-metal GPU provider and an internet exchange operator, raised A$827 million to build a distributed inference cloud and now expects revenue to roughly double next year. A two-year-old startup called Zero Latency is building what it openly calls a CDN for inference, with AT&T, Zayo and Crown Castle on its partner list, and the fiber owners in that story mostly want to carry inference point to point and monetize the strands.

What nobody has written down yet is the economics, and this is where the peering analogy earns its keep. Settlement-free peering works when both sides get roughly equal value from the exchange and imbalance turns it into paid peering. Inference capacity exchange starts out asymmetric, because the party short on GPUs gets more from the deal than the party long on them, so compute peering probably starts paid and stays paid, with settlement-free arrangements reserved for mutual overflow between operators of similar size, the inference version of backup transit. Prefill-as-a-Service is a ready-made billable unit for exactly that. And the video world already built a federation template with Open Caching which lets a content owner reach caches inside many ISPs through one API. Swap the cache for a GPU pod and the standard is most of the way to an inference exchange.

The open question is the same one that shaped the internet the first time. Does a neutral operator run the fabric where inference capacity and inference traffic meet, the way exchanges did for packets, or does every relationship get negotiated bilaterally? I personally think the neutral operator argument wins, because the parties here trust each other even less than networks did in 1995. If that's right, the internet exchange gets a second act too.

What a distributed inference world looks like

What I'm talking about is a hierarchy, not necessarily a dispersion. At the top, a small number of AI factories train frontier models and serve the heaviest inference, because those workloads have the best economics at scale. Below them, regional and metro hubs: converted central offices, regional colos, CDN inference sites, holding small and mid-size models and the decode side of disaggregated serving. Below those are edge PoPs running gateways, caches and small models.

A request enters at a gateway near the user. If it's a cache hit, it never touches a GPU. If it's a short request, it runs in the metro hub. If it carries a long context, the prefill may ship to a compute-dense cluster while decode stays close, with the KV cache moving on marked flows across paths the scheduler picked from live telemetry. Capacity that isn't there locally gets bought across an exchange fabric, settled the way transit and peering settle today. Sovereignty and regulation fall out of the topology naturally, because the data can be kept in the region. And when the upstream is cut, the answers keep coming, because the model is in the building on your street. Someone raised exactly that objection to CORD in 2017, asking what happens to the home when the fiber goes down. In the inference version, local survivability is a marketable feature.

The physical picture may be less glamorous than the campus renderings for some but personally, I like it more. A mid-century brick building with a new dry cooler humming in the generator yard. Twelve contained racks on a floor that used to hold a 5ESS. N x 400G waves to the metro exchange. A colo'd tenant whose engineers badge in unescorted at 2 a.m. It's an elegant rebirth of this infrastructure that has been built to stand the test of time, structures that were made in an era when craftsmanship meant something.

Where this "whiffs"

Light Reading's read on NVIDIA's AI grids was that the realistic version is a sparse set of refurbished central offices, not a dense mesh. I agree with that take. The trouble comes if demand lands lower than the queues suggest, because interconnection queues overstate real load and ERCOT has already torn up one 2030 forecast mid-cycle. If that happens, centralized capacity absorbs the work and the substations stay dark.

The cheapest inference is still a cache hit and semantic caching pulls the other way from GPUs everywhere. Frontier gravity is real. Akamai's own COO says the big factories keep the best tokenomics for training and frontier work. Cloudflare's margin compression shows that carrying this traffic is not free even for the gateway players and Megaport's stock fell on its results because investors looked at the capex before the contracts. Nobody has published a retrofit cost per usable kilowatt for a CO conversion, which means every CO business case right now rests on an engineering estimate.

So here are some things that would prove me right or wrong within a couple of years. If a public exchange could somehow publish an inference traffic share on its fabric. A real paid compute-peering agreement appears between an inference supplier and a route operator. Liquid-cooled density actually lands in CO-class sites at scale. And Verizon, or someone, discloses how big "sold out" actually is. Until then, treat this as a thesis with early evidence.

What to do about it

If you run networks for a living, learn the AI serving stack's network surface now. KV transfer, cache-aware routing, DSCP classes and the telemetry interfaces schedulers consume are becoming the conversation. That space between the GPU people and the network people is where our profession gets hired into this wave, and the talent pool there is tiny right now.

If you run central offices, inventory them before someone else does. Power entrance, fiber diversity, floor loading, cooling path, zoning and the union and access rules that will decide whether a neocloud or frontier lab tenant gets access. Verizon's trial sold out in a day. Yours might too, but you'd better be able to talk the talk when they come knocking.

The central office was where the network lived when I started and I think it's about to be where the answers live. The buildings are there, the tenant finally exists and the routing problem is a challenge that will be solved. If you're inside an operator, a colo or an exchange and you're looking at the same stuff I've detailed here, I'd like to compare notes so please reach out.