Yeah, it’s miserable; xterm shouldn’t require Xorg. It should be agnostic to display system and not force the X monoculture on everyone. Classic Microsoft gestapo tactics: shoehorn Xorg dependencies into tons of unrelated apps and thus curtail user freedom to run xterm with a WinForms or Wayland display system. It’s appalling.
From the project documentation: "The xterm program is a terminal emulator for the X Window System."
The application does not require xorg it requires an x11 server.
It just so happens that until recently xorg was the only game in town as far as Linux x11 servers are concerned.