Forty-five billion. That is how many AI inference tokens AT&T’s production systems process every single day. To put that number in perspective: it is 45 times the volume AT&T was generating in September 2024, when the company’s daily token count was approximately one billion. The growth is not from adding new AI experiments. It is from running AI in production at industrial scale, across network operations, customer service, fraud detection, employee tools, and engineering workflows, on a continuous basis. AT&T’s experience with that scale — and specifically what it had to build to make it economically sustainable — is one of the most detailed public accounts of what enterprise AI infrastructure actually looks like when it stops being a pilot and becomes a utility.
AT&T detailed this infrastructure in a blog published on its official newsroom on July 23 and subsequently covered by RCR Wireless News and TelecomTV on July 27. The accounts describe an AI operating model built around three interlocking decisions: a proprietary AI Gateway that routes inference workloads to the most cost-effective model for each task; an open-source telco-specific large language model called OTel 2.0 that was trained specifically because frontier models do not adequately understand telecom operations; and an AI tokenomics framework that treats inference cost management as a first-order engineering priority rather than a cost centre.
Key Developments
- AT&T processes an average of 45 billion AI inference tokens every day across its production systems — a 45x increase from the approximately 1 billion daily tokens it was generating in September 2024.
- AT&T developed a proprietary AI Gateway that uses cache-aware routing to match each task to the most cost-effective model, reducing AI inference costs by up to 90%, saving millions of dollars in production.
- The company launched OTel 2.0, an open-source telecom-specific large language model post-trained on 400 billion telecom tokens selected from over 1 trillion processed — using AMD GPUs via Microsoft Foundry, in collaboration with GSMA, Red Hat, Dell, and Microsoft Azure.
- AI applications at AT&T span network planning, customer service, fraud prevention, employee onboarding, cell tower installation, and code
The 45 Billion Token Number and What It Means
As AT&T’s chief data and AI officer Andy Markus wrote on the AT&T corporate blog, the company processes an average of 45 billion AI tokens every day across its production applications. That 45x growth from September 2024’s one billion daily tokens reflects AI moving from a research and tooling function into the operational core of how AT&T runs its business. The company has deployed AI across network planning, customer service, fraud prevention, employee onboarding, and cell tower installation, generating more than one trillion tokens per month across its AI workloads as generative AI applications have expanded. The practical challenge that scale creates is straightforward: if AT&T routed all 45 billion daily tokens through its most capable frontier models at standard API pricing, the inference cost would be prohibitive. The entire engineering architecture that AT&T has built around its AI workloads is, at its core, a solution to that economic problem.
The AI Gateway: Cache-Aware Routing at Scale
How the Gateway Works
The AI Gateway is AT&T’s most significant internal AI infrastructure innovation. At 45 billion tokens per day, the company processes an enormous variety of task types: a customer service query about a bill, a network anomaly diagnosis, a fraud flag requiring rapid classification, an employee onboarding question, a code completion in a development environment, a cell tower installation checklist item. Each task has different complexity, different latency requirements, different acceptable error rates, and therefore a different optimal model: the cheapest model that handles the task adequately, rather than the most capable model that handles it perfectly. The AI Gateway evaluates each incoming task against those parameters and routes it to the most cost-effective model — without compromising output quality for the specific task. It can even change models partway through a multi-turn conversation session: at each turn, the gateway weighs speed, cost, and expected output quality, then routes the prompt to the best available model for that specific turn.
Cache-Aware Routing
The ‘cache-aware’ dimension of the routing adds a further layer of efficiency. Many enterprise AI queries are variations on recurring patterns — standard troubleshooting questions, common billing inquiries, repetitive code patterns, standard network configuration queries. A cache-aware router recognises when an incoming prompt is similar enough to a previously processed query that the cached response is adequate, avoiding a full model inference computation entirely. Combined with intelligent model selection for queries that do require fresh inference, the cache-aware gateway produces the 90 percent cost reduction Markus describes: ‘We’re reducing AI costs as much as 90%, already saving millions.’ The specificity of that claim — ‘already saving millions’ — indicates AT&T is measuring the gateway’s economic impact against its production inference baseline, not against a theoretical maximum cost.
OTel 2.0: Why Frontier Models Are Not Enough for Telecom
The Accuracy Problem with General Models
The reason AT&T invested in a telco-specific model is straightforward. Frontier models — GPT-5.6, Claude Fable, Gemini — were not trained on telecom standards documents, network operation protocols, GSMA specifications, or the specific terminology of mobile network architecture. AT&T’s chief data officer identified that frontier models could answer telecom-specific questions accurately only about 60 to 70 percent of the time. In customer service contexts, 30 to 40 percent inaccuracy is commercially destructive. In network operations contexts, it is potentially physically destructive. The company needed a model that ‘speaks telco’ — and no frontier lab had built one.
How OTel 2.0 Was Built
OTel 2.0 is a post-trained version of Google’s Gemma 4 31B-IT model, built within the Open Telco AI initiative that AT&T leads in partnership with GSMA, AMD, Microsoft, Red Hat, Dell Technologies, and others. Building it required solving a data problem that does not exist in other AI domains: there is no telecom equivalent of Wikipedia, no large publicly available corpus of telecom technical text that a model can simply be trained on. The GSMA built the initial training dataset by collecting and processing documents from seven telecommunications standards development organisations — 3GPP, ETSI, GSMA, CAMARA, ITU, O-RAN, and TM Forum — and converting dense technical specifications into material suitable for model training. That initial dataset comprised approximately 15 billion tokens. AT&T then processed more than one trillion tokens in total — including data preparation, synthetic data generation, and filtering stages — to select 400 billion tokens for actual post-training on AMD GPUs through Microsoft Foundry’s Managed Compute infrastructure.
The Results
OTel 2.0 now tops the leaderboard on Open-telco.ai, the evaluation platform for telecom-specific AI models. GSMA’s Louis Powell described it as ‘the best and largest post-trained open model built specifically for the telecoms industry.’ The model’s value is not only in its accuracy on telecom tasks but in its deployment economics: a domain-adapted 31-billion parameter model is significantly smaller than a frontier model, which means it runs on ‘right-sized hardware, either on premises or in the cloud,’ in Markus’s framing. A smaller, faster, more accurate-for-telco model that AT&T can run on AMD hardware deployed through Microsoft Foundry reduces both inference latency and cost per query compared to routing the same question to a frontier model at standard API rates. This connects directly to the AI infrastructure efficiency challenge documented in our reporting on AT&T’s position in the broader AI data center energy demand picture: smaller, domain-specific models that run on more efficient hardware contribute to the same infrastructure efficiency goal as architectural innovations at the hardware level.
The Tokenomics Framework
Markus’s description of AT&T’s ‘AI tokenomics framework’ frames the economic challenge at the level of unit economics: the cost per token generated must be low enough that AI is more economical than the human labour it supplements or replaces, across each application in the portfolio. That framing requires treating inference cost as a product design parameter — a constraint that shapes which models are selected, how caches are built, what task types are handled by the AI Gateway versus routed elsewhere — rather than as a variable cost that grows proportionally with usage volume. The 90 percent cost reduction AT&T has achieved is the practical output of applying that framework consistently across a 45-billion-token-per-day production environment: it is the result of many small optimisation decisions, not a single architectural breakthrough.
Implications for Other Telcos
AT&T’s architecture is designed to be replicable. OTel 2.0 is open source, available for download, and deployable by any telco on AMD or NVIDIA hardware. The AI Gateway routing approach is a software architecture that any sufficiently large enterprise can implement with its own model portfolio. The GSMA’s involvement in building the training dataset and evaluating the model means that the foundational telco AI infrastructure is being built as an industry resource rather than proprietary AT&T technology. As Sebastian Barros noted in his analysis of the AT&T announcement, the most interesting element of AT&T’s 45 billion daily tokens is not the scale of the number but the architecture that makes it economically sustainable: a specialist-model-plus-routing approach that any telco can adopt. The same principle applies beyond telecoms to any sector with sufficient volume of domain-specific AI queries that the economics of building a sector-specific model outweigh the convenience of API access to a frontier general model. The Microsoft Maia 200 inference chip represents the hardware-level complement to what AT&T is doing at the software and model level: both approaches are responses to the same fundamental challenge of making AI inference economically sustainable at industrial scale.
What Happens Next
AT&T has not specified a target for daily token volume beyond the current 45 billion, but the trajectory — from 1 billion to 45 billion in under two years — suggests the ceiling is determined by application coverage rather than infrastructure capacity. As AI extends into more AT&T operational workflows, and as the AI Gateway’s cost optimisation makes each additional application more economically viable, token volume will continue growing. OTel 2.0’s release as an open-source model will accelerate the broader telco industry’s adoption of the same specialist-model approach, which in turn will generate more training data, more evaluation benchmarks, and more community-developed fine-tunes that improve the model’s accuracy further. The Open Telco AI initiative’s collaboration structure — AT&T, GSMA, AMD, Microsoft, Red Hat, Dell — means that future versions of the model will have access to a larger and more diverse training dataset than any single company could generate alone.
Why It Matters
AT&T’s 45 billion daily token disclosure matters because it is the most detailed public account of what enterprise AI actually looks like at industrial scale: not a collection of AI experiments or a set of productivity tools deployed to individual employees, but a continuous, 24-hour-a-day production infrastructure processing the equivalent of hundreds of millions of individual AI requests. The architecture behind that scale — domain-specific open models, intelligent cost-optimising routing, cache-aware inference management — is a template that any sufficiently large enterprise can adapt. And the 90 percent cost reduction AT&T has achieved by applying it is the most concrete public evidence available that AI at industrial scale can be made economically sustainable without simply accepting whatever the frontier API providers charge.
Sources
AT&T corporate blog (about.att.com/blogs/2026/the-tokenomics-equation.html), July 23, 2026. RCR Wireless News, July 27, 2026. TelecomTV, July 27, 2026. GSMA newsroom (gsma.com), OTel 2.0 launch coverage, July 2026. TelecomLead, July 24, 2026. Sebastian Barros Substack (sebastianbarros.substack.com), July 25, 2026.
