Over-zealous developers who think it's appropriate to log all function calls with parameters for trace level logs, or a framework with the same opinion automatically applies such tracing and logging over the whole code-base.
No-one notices because no-one uses trace level logging, until one day another developer is tearing their hair out because they can't reproduce a bug that is only occuring on live. It's an urgent bug that needs resolution asap. So this developer turns on the trace level logging and eventually finds and resolves their bug.
Being the careful person they are, they turn off the logging and go away happy.
Meanwhile they've unknowingly produced a few gigabytes of log outputs which happen to include plaintext passwords.
That's just one of many different scenarios where people acting in 'good faith' can still lead to bad outcomes. That is why a "PUNISH THEM!" attitude to this kind of incident is not helpful.
HTTPS form submissions should be encrypted while the data travels between the user's computer and the server, but the server will still need to decrypt them to perform the hashing. It's possible, and probably even common, for inexperienced or forgetful developers to add request logging for debugging or diagnosing service outages without adding extra logic to scrub sensitive fields.
seriously? pretty easily. somebody probably left a debug log message in place or something. guaranteed that this happens all the time and most people don't report it.
I doubt anyone left something that logged the plaintext password. No reasonable architecture necessitates holding onto a plaintext password for more than one line of code.
One possibility is an HTTP server on the request path after TLS termination. But then why is an HTTP server logging the request body?
My guess would be some sort of instrumentation process was blindly reading data in memory without distinguishing what the data was, but produced logs that incidentally included passwords.
In my experience, I've seen both of the following scenarios:
POST request comes in from the client. Full URL and request body is logged. Sometimes for simply troubleshooting, sometimes for security reasons (e.g., wanting to know all data coming in so that it's possible to identify security holes after they've been exploited).
POST request comes in from client. Frontend server makes a GET request to a backend server, and the password ends up in the standard request logs. In one case, I've seen this happen because the developer thought path variables were cool, so every API they wrote looked like /a/b/c/d/e. Sigh.
As developer, I can tell you this happens more often than I'd like to admit.
debug logs is that necessary evil you need to troubleshoot pesky bugs. Unfortunately some of these debug tools need to be turned on in a live environment to capture those logs for debugging. But also Unfortunately, we are humans and we concentrate on fixing the bug and forget to turn off logging or log unnecessary data.
Indeed. This is probably a good reminder for every developer to just go and check through their logs to see what is there. It can be quite a shock sometimes to find how much can get dumped there..
I'm curious if anyone has details on using bcrypt/scrypt at scale. Specifically one way I could see this happening is something like login requests go to a load balancer that puts the requests on a queue to be picked up and validated by some hasher service, and the queue ends up writing the requests to logs to recover from certain kinds of failures.
From the password compromise side, not really, you're just pushing the cost of hashing to your users (and it will impact mobile users more). There's a similar technique on needing proof-of-work on the client to combat DDoS.
From the authorization side, there is a threat, because if your table storing hashes is compromised, attackers just have to supply the stored hash to the auth endpoint and they get to login as anyone.
A combination of hashing on the client side (or immediately once the pw hits the endpoint) with something cheaper followed by a more intense bcrypt/scrypt afterwards might help a bit with the tradeoffs.
Does it buy you anything at that point? A server-side issue, such as this, would still log the thing you need to log in, and a client-side issue would just intercept the hashed form or could derive the hashing mechanism from analysing the client.
At most, it would seem to prevent weak passwords from being passed directly to bcrypt, but salting should solve that in a similar way anyway, and anyone brute-forcing a copy of the database can incorporate the same weak hashing logic
Another comment in this thread mentioned an idea of seeding the hash with a quickly expiring nonce fetched from the server. I think that’s a quite clever approach, similar to CSRF tokens in a sense.
That would effectively create a one time “password” for transmission from browser to database. In a case like this one, where sensitive text transmitted from the client leaked into logs, it would be a non-issue. The sensitive string in the logs is a temporary hash that would be useless shortly after discovery, since it was derived from an expired nonce.
It effectively becomes a real time scrubbing system with 100% coverage, because the passwords are “scrubbed” by design, and do not depend on explicit detection code in some scrubbing mechanism.
That's exactly what it is: incompetence, rank incompetence. Something like nine out of ten people getting paid today as professional software "engineers" should be let go. Dr. Margaret Hamilton figured out most of what we need to do to develop reliable software during and after the Apollo 11 mission. She coined the term "software engineering". Unfortunately, her work suffered from bad languaging and languished.
You'll notice you've been downvoted to hell and the comments in reply to yours are apologists and excuses. Not a coincidence.
It just sounds so incompetent.