It get exponentially difficult to add more qubits so it's not a given that we will be able to build one large enough to be a real threat to modern cryptography.

“Quantum computers that break diffie hellman as easily as RSA”, where “easily” means “not at all”, do exist.