But US citizens living abroad have to file tax returns even when they don't exercise that right. They could live abroad for the remainder of their life and have to file (under IRS rules).

That seems like a very flimsy argument for why US nationals abroad have to file tax returns.

If I have a second house that I never use I still have to pay tax on it. The tax man doesn't care whether I use it or not.

You didn't specify "what rights do they exercise", just what rights do they have.