Mozilla has acknowledged that its Firefox AI Runtime — the on-device engine powering features like Smart Tab Groups and alt-text generation — was “not quite at the speed we wanted.”
Translation: it was slow, and users noticed.
So, in typically un-Mozilla fashion, it’s done something about it.
Over the past few months, engineers have replaced the WebAssembly (WASM) build of Microsoft’s ONNX Runtime which powers Firefox’s Transformers.js-based features with a native C++ version that is built into the browser’s codebase.
Early benchmarks show “from 2 to 10× faster inference” with the new native backend, and loses the WASM warm-up overhead that made first-runs of AI features feel sluggish, per a new blog post from Mozilla engineers Tarek Ziadé, Paul Adenot, and Serge Guelton.
Why ditch WebAssembly?
Per a blog post, the inference pipeline was inefficient by design: JavaScript preprocessing, WASM execution, then back to JavaScript for post-processing. Even with “warm caches” (data already loaded in memory) the back and forth was slowing things down.
Mozilla’s in-browser translations use a native C++ approach for its ML-based tasks, so the Firefox AI team decided to follow suit — and it sounds like it’s paid off.
The PDF.js alt-text generation feature “went from from 3.5s to 350ms” on the same hardware, which accounts for a 10x speedup. Meanwhile, (so-called) Smart Tab Grouping “which can be laggy on first run, is now quite snappy.”
The team adds that things still aren’t perfect and “a lot of improvement can and will happen in the future” to improve ML-based features in Firefox.
A major speed boost will come from using GPU-acceleration for tasks, so effort is underway to add support for that. Other tweaks, including multi-threaded operations, and compiled graph caching to boost model startup times, are in the works too.
Until then, Firefox’s “AI” features continue to run on CPU (which is why some Firefox users noticed runaway CPU usage recently).
When? Why, it’s already here…
If you upgraded to Firefox 142 you are benefitting from the new approach, as Firefox’s Smart Tab Grouping feature is said to be using the native background.
Beyond that, Mozilla is “gradually rolling out this new backend to additional features throughout the summer, so all capabilities built on Transformers.js can take advantage of it.”
Whether you want “AI” features in your browser is a separate question, but if they are going to be present, on-device tools offer privacy benefits, and making sure they work as efficiently as possible is…
Well, it’s better than not, I guess.