I didn't notice that iptables-persistent actually saves the currently configured rules periodically. That's both kinda neat and a little scary, and I'm not entirely sure I see much value in persisting dynamic rules; it seems like it'd be easy to end up with a long chain of stale rules that way. Still worth knowing about the automatic persistence, though.

Surely as rules are deleted they disappear from the saved copy?

Or is the issue that fail2ban et al don't remove stale rules?

It does, but I seem to remember it having trouble cleaning up after itself when abruptly terminated and restarted. It's been a few years, though, so that's probably no longer an issue.