Writing the Benderson Media article on ZML’s inference tool now.

ZML Free Inference Tool Runs Fast AI on Any Chip

A French startup just handed every AI builder a weapon against NVIDIA’s pricing power. ZML released a free, open-source inference product that runs across AMD, Intel, and custom accelerator chips. AI inference spending is on track to exceed $200 billion annually by 2027 according to Goldman Sachs. If ZML delivers on its promise, a big chunk of that money stops being locked to one vendor.

Why This Moment Matters

For three years, running AI models at speed meant one thing: NVIDIA GPUs. NVIDIA controls roughly 80% of the AI accelerator market according to TechInsights. That monopoly has a price. Companies pay absurd compute bills and they’re locked into one stack. You can’t just swap chips and keep running.

Meanwhile, real competition has been building. AMD’s MI300X chips have been shipping to hyperscalers. Intel’s Gaudi 3 accelerators are in production data centers. Amazon, Google, and Microsoft are all running custom silicon at scale. The hardware side of the AI chip market is genuinely fragmenting according to reports from Semianalysis.

The bottleneck was never the chips. It was the software. Most inference frameworks are written tightly around NVIDIA’s CUDA stack. You can’t easily move a production workload to AMD or Intel hardware without rewriting large parts of your pipeline. That friction keeps companies paying NVIDIA prices even when cheaper options exist. ZML is attacking that friction directly.

What ZML Actually Built

ZML, founded in Paris and operating with a lean engineering team, released a unified inference runtime that abstracts away the chip layer. Their product sits between your AI model and the hardware. You call one API. ZML figures out how to run it fast on whatever chip is underneath, whether that’s NVIDIA, AMD, Intel Gaudi, or custom accelerators.

This is harder than it sounds. Chips have different memory hierarchies, different instruction sets, and different parallelism models. Writing software that runs efficiently across all of them requires deep systems engineering. ZML’s team has published benchmarks showing competitive inference speeds across chip families, though independent verification at scale is still early.

The “free” part is important. They’re releasing this as open source. That’s a deliberate market strategy. Get the runtime embedded everywhere. Build trust with developers. Then monetize the enterprise tier, the managed service, or the professional support. It’s the same playbook Redis used, the same one HashiCorp used. Give away the tool and charge for the infrastructure around it.

I’ve watched this pattern play out enough times to know what it usually means. The companies that get disrupted aren’t the chipmakers. They’re the inference hosting providers that built their business on NVIDIA lock-in. If any chip becomes viable for production inference, your differentiation evaporates.

According to Andreessen Horowitz research published in 2025, inference costs now account for roughly 60% of total AI compute spending for most production applications. That’s the budget ZML is targeting. Even a 20% reduction in inference cost on existing workloads is a significant number for any company running AI at scale.

The contrarian read here is that ZML’s biggest competition isn’t from NVIDIA. It’s from inertia. Most engineering teams don’t want to rethink their infrastructure. They want to ship features. So the real question is whether ZML’s tool is good enough to justify the migration effort, or whether it becomes another promising open-source project that gathers GitHub stars but never hits production.

Builders who want to turn AI capabilities into content output fast should look at tools like InVideo AI while the compute cost gap still exists. The window where you can build AI-powered content products cheaply is narrowing as everyone races toward the same infrastructure plays.

What I Would Do With This

If you’re running an AI-powered product in production today, your first move is not to rip out your infrastructure. It’s to run a benchmark. Pull your most expensive inference workload. Test ZML against your current setup on AMD hardware. Get a real cost number. That number tells you whether this is worth engineering time or not.

If you’re building something new, this matters more. You no longer have to architect around NVIDIA as a hard dependency. You can design for portability from day one and choose your hardware based on availability and price at runtime. That’s real optionality and optionality has monetary value.

For startup founders specifically, this is a reminder that infrastructure commoditization always flows downhill eventually. Inference is becoming a commodity. The money is in what you build on top of it, not in the compute itself. Stop paying premium prices for commodity infrastructure. If AppSumo has deals on AI tooling in your stack, that’s the same principle applied to software. Own the product. Rent the plumbing.

The developer community will be the real proving ground here. If ZML’s runtime gets embedded in popular inference serving frameworks like vLLM or SGLang, adoption accelerates without any sales motion. Watch the GitHub stars and the PR activity over the next 90 days. That’s your signal on whether this reaches critical mass.

One practical step right now: follow ZML’s GitHub, join their Discord if they have one, and test their runtime on a low-stakes workload. Engineering teams that evaluate early have time to plan. The ones who wait until this becomes mainstream are playing catch-up.

The Bottom Line

NVIDIA’s pricing power in inference is built on software lock-in, not chip superiority. ZML just released a tool that attacks the lock-in directly. The rich operator sees this and runs a benchmark this week. The average company waits two years, pays full NVIDIA prices, and then wonders why their margins are thin. The chip that runs your model cheapest is the right chip. ZML is trying to make that choice yours to make.

Frequently Asked Questions

What is ZML and what does its free inference tool do?

ZML is a French AI startup that built a unified inference runtime letting developers run AI models across multiple chip types including NVIDIA, AMD, and Intel Gaudi hardware. The tool abstracts away the chip differences so you write one integration and it handles the hardware layer. It’s free and open source.

Does ZML’s tool actually match NVIDIA inference performance?

ZML has published benchmarks showing competitive speeds on their target chip families, but large-scale independent verification is still early. Performance varies by model architecture and workload type. Running your own benchmark on a real workload is the only way to get a number you can trust.

Who does ZML’s product hurt the most?

The companies most at risk are inference hosting providers and cloud services that built pricing models around NVIDIA GPU scarcity. If chip portability becomes easy, the commodity inference market gets very competitive very fast. NVIDIA itself is more insulated because they still make the fastest chips overall.

Is open-source inference tooling safe to use in production?

Open source can absolutely run in production. The right question is whether the project has active maintainers, a clear support path, and real adoption from companies you trust. Check the GitHub contributor history and whether any serious engineering teams are running it before committing your production stack to it.

How does chip-agnostic AI inference affect AI product costs?

According to Andreessen Horowitz, inference is roughly 60% of total AI compute spend for production applications. If a tool lets you run the same workloads on cheaper hardware without rewriting your code, your savings come directly from that 60% budget. Even a modest efficiency gain on large workloads adds up to real money quickly.