As you said, the specific danger to cryptography posed by quantum computers is that they're capable of factoring large primes much more quickly than current processors.
The reason this is a problem is that the most widely used cryptographic protocols today (including the one protecting your connection to Cake as you read this sentence) are public key protocols, in which each party possesses a secret private key and a non-secret public key that's safe to share. So Alice can give her non-secret public key to Bob and that allows Bob to encrypt his messages so that only someone who has Alice's private key can decrypt them.
The ability to quickly share a public key like this in order to establish secure communication is really convenient, since it means Alice and Bob don't actually have to share any secrets with each other before they can start encrypting messages. The actual secrets — their private keys — never get transmitted anywhere.
This relies on algorithms that make it easy to calculate a public key if you know the private key, but virtually impossible to calculate the private key if you only know the public key. But quantum computers are very good at exactly the kind of math necessary to calculate private keys from public keys, so eventually it will no longer be safe for Alice and Bob to exchange public keys since anyone with a fast enough quantum computer would then be able to compute their private keys and snoop on their messages.
But don't worry! All hope is not lost.
Quantum computers are very good at certain kinds of algorithms, but they're not good at everything. There are forms of encryption that don't rely on prime factoring or public key exchanges. In fact, one of the very oldest and simplest forms of cryptography — secret key cryptography, in which both Alice and Bob know the secret key they use to encrypt their messages — is no more vulnerable to attack by quantum computers than by any other computer.
The problem is that secret key cryptography is inconvenient (it requires both parties having a shared secret ahead of time), and other quantum-safe cryptographic techniques haven't gotten much attention yet, so they'll need a lot of work and refinement before we'll be able to rely on them.
But that work is happening, and these problems will eventually be solved, and then we'll have brand new problems to worry about. 😄