But what are the odds that obscure formats like this will be implemented (and maintained) without security holes? I vaguely remember an exploit where the GNOME file viewer would automatically preview samples of any audio file that (I think) gstreamer could recognise, and one of those formats was something like a NES audio format, whose implementation had some buffer overflows in it. Are there similar concerns with ffmpeg?
> But what are the odds that obscure formats like this will be implemented (and maintained) without security holes?
Extremely high?
I wrote a stand-alone commercial DSP app which seems to still be sold ten years later. There might be security issues, there probably are since it hasn't been recompiled in years as far as I know, but I'm certain they aren't in the DSP parts.
DSP code involves large blocks of numbers which you translate into other numbers. If you're writing a format or facility within ffmpeg, you aren't directly reading or writing to files, or connecting to the Internet. It's probably that the whole API you write to is just blocks of numbers and some sort of data description.
ffmpeg as a whole might have security violations in it but adding new plug-ins or format won't necessarily increase those chances.
The problem is that when you use ffmpeg, you generally delegate the task of figuring out "what the hell is this data" to it - you don't want to figure out yourself if something is MP3, MP3 in an AVI container, VP9 in MKV, ...
So when ffmpeg adds obscure formats that no one ever uses and are mostly toy implementations, the risk is always that someone at the top (say, your Electron app) feeds in user-supplied data, the 10 layers in between pass it along, until ffmpeg at the bottom goes "oh I know this one, it's a Digital Pictures SGA game"!
Now Chromium wisely disables most of this trap code in its copy of ffmpeg from even being compiled in the first place, but that's probably not the case for the ffmpeg copy on your operating system that you might use for some server-side processing task.
Running ffmpeg on untrusted input outside of a sandbox you trust to be secure is an extremely bad idea. It doesn't have a good track record for that, and making it safe to use ffmpeg on untrusted input has never been a priority for the project.
In practice things like video hosting services which use ffmpeg internally tend to disable support for most of the obscure file formats to reduce the potential attack surface.
You can't expect absence of security holes. ffmpeg is an extremely complex piece of software written with unsafe C and unsafe assembler. It's full of security holes, that's for sure. If that's of concern to you, use additional security measures to mitigate that threat.