Pour a coffee. This one is not a fire drill.
IDC just published a January 2026 whitepaper, sponsored by Google Cloud, summarizing a global survey of 1,317 senior AI decision-makers. The headline framing is heavy: a “TCO crisis,” an “efficiency gap,” a fundamental rethink of how we build AI infrastructure. All true, and all worth understanding. But on a quiet Saturday, the more useful read is that almost every problem in that report has a known fix, and most of those fixes are within reach of any team that already has a working .NET or Python stack.
So let’s walk through what the data actually says, what holds up under scrutiny, and what you can do about it on Monday morning without panic, without a rip-and-replace, and without writing a check that needs a board vote.
Where AI workloads actually live now
The first useful number in the report is the workload split. According to IDC’s survey, inference now accounts for 47% of AI workloads, training 28%, model optimization 16%, and the “innovation workbench” where data scientists prototype 9%.
That is a real and important shift, and it is corroborated outside this single survey. IDC has separately forecast that global investment in inference infrastructure will surpass training infrastructure by the end of 2025. The era of “train a giant model once, talk about it forever” is over. The money is now in what happens after the model exists — every chatbot reply, every search result, every agent step is an inference event, and they add up fast.
Why does this matter for a calm Saturday read? Because inference is the workload most teams can optimize quickly. Training optimizations tend to be expensive science projects. Inference optimizations are mostly engineering: better batching, model quantization, sensible caching, smarter routing. Boring, doable work. That is good news.
The “efficiency gap” in plain English
IDC defines the efficiency gap as the difference between the theoretical performance of your AI stack and what you actually get out of it. They argue, fairly, that a 30% GPU utilization rate means 70 cents of every infrastructure dollar is wasted.
Is that real? Yes. Independent industry data lines up with it. Multiple sources tracking enterprise GPU clusters in 2025 and 2026 report average utilization in the 10–30% range, with even well-run organizations often landing in the 55–65% range. A widely cited Google study of millions of training workloads found jobs spend, on average, around 30% of training time stuck in the input data pipeline. The number IDC quotes is not a worst case. It is roughly the middle of the road.
Now here is the part the report only hints at: this is a young-industry problem, not a permanent one. The reason GPUs sit idle is rarely the GPU. It is the data pipeline, the scheduler, the network fabric, or the orchestration layer that was never designed to feed accelerators at full tilt. Every one of those is a solvable software problem. None of them require buying a new generation of hardware.
Where the money actually goes
IDC asked respondents what contributes most to AI TCO. The answers, in order:
Compute resources — 61%
Data storage and management — 60%
Software licenses and subscriptions — 53%
Data pipeline development and maintenance — 52%
Specialized personnel and skills — 38%
MLOps platforms and tooling — 35%
Notice that the top four are not really about GPUs. Three of the four are data and software problems. That tracks with another finding from the same survey: 47.7% of respondents struggle with data quality, consistency, and governance; 45.6% with storage costs; 44.1% with the time it takes to clean and prepare data.
If you have ever felt like your AI budget is being eaten by something you cannot quite point at, this is usually where it is hiding. Not in the headline GPU bill, but in the long, patient work of moving data around.
What people want to measure but can’t
This part of the report is, honestly, the most interesting. When IDC asked what teams wish they could measure but cannot, the top answers were:
Intelligence per dollar — 53.9%
ROI — 50.8%
Cost per token — 40.7%
Time to insight — 40.1%
GPU utilization — 36.1%
“Intelligence per dollar” is not yet an industry-standard metric. IDC frames it as the total cost per unit of useful intelligence delivered to an end user, and points at LMArena-style rankings combined with public pricing data as a rough proxy. A more rigorous version is “goodput,” specifically the ML Productivity Goodput framework Google introduced in 2024. Goodput measures how much of your system’s capacity is actually being converted into useful work, decomposed into scheduling goodput, runtime goodput, and program goodput.
Pour yourself another cup. The point is not the jargon. The point is that you can start measuring this without buying anything. A modest telemetry layer over your inference endpoints, combined with token accounting and a sensible definition of “good” output for your use case, will get you 80% of the way there. Most of my clients are surprised at how quickly that picture comes into focus once they stop arguing about which exact metric to standardize on and just start logging.
The 1% trick
Here is the math worth holding onto. IDC walks through an example: an application handling 500 million inferences a day at an annual compute cost of $182,500. A 1% efficiency gain saves over $1,800 a year on that one workload. The arithmetic checks out exactly, and the underlying point scales: across dozens of high-volume services, 1% improvements stack into millions.
This is why I never tell clients to “rebuild everything.” Compounding small wins beats one heroic rewrite every time. Quantize the model. Tune the batch size. Add a cache. Pick a better embedding store. Move the right workloads to the right hardware. Each move looks small. The bill at the end of the quarter does not.
Where on-prem and edge still win
The report makes a careful, honest point that’s worth repeating: cloud is the right answer for most AI workloads, but not all. The cases where on-prem or edge still wins are predictable:
Latency-sensitive systems where milliseconds matter — robotics, predictive maintenance, autonomous control.
Regulatory and data sovereignty requirements — defense, finance, healthcare, and anywhere data residency law applies.
Cost of data movement — high-volume video and sensor streams where egress fees alone make cloud-first uneconomical.
Note that autonomous systems are the only category in the survey where on-premises deployment (33.4%) actually exceeds cloud deployment (27.4%). Everything else — content creation, chatbots, document processing, fraud detection — leans cloud, often heavily.
This is why a hybrid architecture is usually the calm answer, not the fashionable one. Keep the workloads in the cloud that benefit from elasticity. Pull the workloads on-prem or to the edge where physics or regulation demands it. Stop trying to force every workload into one pattern.
What to actually do this week
If the report leaves you wanting a checklist instead of more anxiety, here is one. None of it requires a forklift upgrade. All of it is the kind of work my team at Inverted Software does for clients every week.
1. Measure before you optimize
Stand up basic telemetry on your inference workloads. Token counts, latency percentiles, GPU utilization, cache hit rates, error rates. If you are on .NET, OpenTelemetry slots cleanly into ASP.NET Core, and the new Microsoft Agent Framework exposes hooks that make agent-level tracing surprisingly painless. On the data side, SQL Server 2025’s vector storage gives you a single place to log embeddings, retrieval calls, and outcomes without bolting on a separate vector database just to get observability.
2. Consolidate your stack before you scale it
IDC found that 54.3% of organizations are running multiple AI frameworks or hardware platforms in a single workload, and 92% of those report negative efficiency effects. That is the most actionable number in the entire report. If your team is mixing TensorFlow, PyTorch, vLLM, three different vector databases, and two cloud regions for one application, you do not have an AI problem. You have a coordination problem dressed up as an AI problem. Pick a primary path. Keep the others only where they earn their keep.
3. Quantize and right-size your inference
Most production inference workloads can run on a smaller model than the one originally chosen, with quantization down to INT8 or INT4 where the model family supports it, and dynamic batching turned on. For .NET teams, this often means putting Ollama or a hosted Claude/Gemini/OpenAI endpoint behind a Semantic Kernel or Microsoft Agent Framework orchestration layer, with routing logic that sends easy queries to the small model and only escalates the hard ones.
4. Add a thin governance layer
Data quality, lineage, and access controls are not optional. They are also not as scary as they sound. A simple metadata catalog, even one as light as a SQL Server schema plus a few stored procedures, plus consistent tagging on your data pipelines, will cover most of what auditors and your own future self will ask for. Heavy MLOps platforms have their place, but a lot of clients I work with overspend here before they need to.
5. Pick one workload, not ten
This is the calm-Saturday version of the advice. Find the single AI workload in your portfolio with the highest run-rate cost. Instrument it. Optimize it. Measure the savings. Tell the story. Then go back to your roadmap with credibility and a budget unlock. The temptation to fix everything at once is what turns a manageable engineering effort into a stalled program.
How we help at Inverted Software
This is the part where, in the interest of being useful and not just informative, I tell you what we actually do. Inverted Software is the consulting house I run. We are a small team with deep .NET, agentic AI, and cloud experience — the kind of resume that ends up doing real work for Microsoft, Sony, Yahoo, Best Buy, Allegiant, Rockstar, and 2K Games over the years. Here is where we tend to add the most value on the kind of problems this IDC report describes:
AI efficiency audits. A focused two-to-four week engagement where we instrument your existing AI workloads, identify the top three to five sources of waste, and hand you back a prioritized fix list with expected savings. No replatforming required to participate.
Agentic workflow design. Whether you are building on Microsoft Agent Framework, Semantic Kernel, MCP, or a custom orchestrator, we help you design agent systems that are debuggable, governable, and economically sustainable. This is the work I have been doing on Tony, our internal leadership AI, and Axon, our agentic workflow orchestrator, and it transfers directly to client systems.
Hybrid AI architecture. Helping teams decide what stays in the cloud, what runs locally on Ollama or a private inference cluster, and how to route between them. This is where the regulatory, latency, and cost-of-data-movement conversations actually get resolved.
Vector and RAG modernization. Migrating from bolted-on vector databases to integrated solutions like SQL Server 2025’s vector storage, Redis, or ElasticSearch, depending on what the workload actually needs. RAG pipelines that were thrown together in 2023 are usually carrying a surprising amount of cost that comes out easily.
Cross-cloud and on-prem deployment. AWS, Azure, GCP, and on-prem. We are not loyal to a single cloud. We are loyal to the workload running in the right place.
If any of that sounds like a conversation worth having, the easiest way to start is to reply to this post, message me on LinkedIn, or email me through invertedsoftware.com. No sales pitch, no obligation. The first conversation is usually about whether there is even a problem worth solving — and sometimes the honest answer is no, your team has it handled.
The Saturday takeaway
The IDC report is right that there is real waste in how the industry is running AI today. The 30% utilization figure is real. The 92% of fragmented stacks reporting negative efficiency effects is real. The data-quality challenges are real.
But the same report quietly contains the antidote. Inference is now the dominant workload, and inference is the easiest to optimize. Consolidation, measurement, quantization, and right-sizing are well-understood techniques. The companies pulling ahead are not the ones with the biggest GPU clusters. They are the ones that stopped treating every AI decision as an emergency.
There is no rush. Pick one workload. Measure it. Improve it. Then do the next one. That is the whole strategy.
Enjoy the rest of your weekend.
— Gal
Gal Ratner is Chief Architect at Prana Entertainment and founder of Inverted Software. He writes about enterprise AI, .NET, and the architecture of agentic systems at galratner.com.