Removing backgrounds from video in Kdenlive, the free, open-source video editor for Linux, macOS and Windows, is about to get a whole lot easier.
Developers are adding a “modern background removal tool” that uses machine learning to create object masks so you can ‘cut out’ an object, person, or item in a video clip.
Real-time background removal features are common in video conferencing/chat apps like Slack. They allow a user to replace their actual backdrop (like a messy kitchen) with an alternative image or, sometimes, even a video clip.
And image editing tools like GIMP and Photoshop have long been able to ‘detect’ objects to speed up selections and masking.
Video editors often have similar needs. They may need to isolate a moving object, item, person, etc in a clip to apply an effect or colour correction, or ‘cut out’ something to add it to another video clip, animate it, and so on.
That is all possible using masks (and for masking objects which move within a clip, masking with keyframes) — two features Kdenlive (as any good video editor should) already offers.
However, masking is a time-consuming and tedious process. A subject must be traced accurately, adding points and curves with a pen tool, and the mask must be adjusted frame-by-frame if the subject moves (and in a video, they usually do).
Enter machine learning to speed up the process.
Kdenlive + Auto Masking
This year Apple released Final Cut Pro 11 with a magnetic masking feature that takes the monotony out of masking: analyse a clip, select a subject, and machine-learning will mask and track it frame by frame — with the option for users to go in and tighten it up after.
Marvellous stuff – and Kdenlive is getting a similar feature of its own!
Object masking in Kdenlive makes uses of Meta’s Segment Anything Model 2 (SAM 2). This is specifically designed to support ‘visual segmentation in images and videos’ meaning it’s perfect for the job of masking of objects in videos.
Once the feature lands, the flow will go something like this: select a video clip, pick an area to apply ‘background removal’ to, create a new Mask, select the object to keep (i.e, cut out), then click Generate Mask to the ML magic happens.
Results are surprisingly good, even if isolating a subject from a busy background with similar colours, as this GIF of a mask made using this new feature in a Kdenlive 25.04 alpha shows:
You can download a Kdenlive alpha build to try it out right now (be sure to download the 25.04 alpha, not the 24.05 RC). Expect further alpha previews to appear as this, and other new features planned for 2025, get finessed, fleshed out, and primed for stable use.
Aside from the fact it may not work reliably (yet), Kdenlive’s new ‘background removal’ tool (I prefer ‘auto-mask’ as a term) does require initial setup and downloading models (a few GB in size).
It will likely work best (and quick) on a decent rig. Attempting to test the alpha on my slowpoke laptop with a feeble CPU and integrated graphics, the current Kdenlive alpha crawled and then crashed when using object masking.
But this is an exciting development. It shows how machine learning and “AI” can be used to aid human creativity rather than, as is increasingly common, replacing it.