I completely agree with this. I’m not surprised by the fine tuning examples at all, as we have a long history of seeing how we can improve an LM’s ability to take on a task via fine tuning compared to base.

I suppose it’s interesting in this example but naively, I feel like we’ve seen this behaviour overall from BERT onwards.