I'm optimistic that it's easier to find/solve vulnerabilities via auto pen-testing / patching, and other security measures, than it will be to find/exploit vulnerabilities after - ie defense is easier in an auto-security world.
Does anyone disagree?
This is purely my intuition, but I'm interested in how others are thinking about it.
All this with the mega caveat of this assuming very widespread adoption of these defenses, which we know won't be true and auto-hacking may be rampant for a while.
If you can compromise an employee desktop and put a too-cheap-to-meter intelligence equivalent to a medium-skilled software developer in there to handcraft an attack on whatever internal applications they have access to, it's kind of over. This kind of stuff isn’t normally hardened against custom or creative attacks. Cybersecurity rests on bot attacks having known signatures, and sophisticated human attackers having better things to do with their time.
I've also thought this for scam perpetration vs mitigation. An AI listening to grandma's call would surely detect most confidence or pig butchering scams (or suggest how to verify), and be able to cast doubt on the caller's intentions or inform a trusted relative before the scammer can build up rapport. Security and surveillance concerns notwithstanding.
In general, most modern vulnerabilities are initially identified with fuzzing systems under abnormal conditions. Whether these issues may be consistently exploited can be probabilistic in nature, and thus repeatability with a POC dataset is already difficult.
That being said, most modern exploits are already auto-generated though brute-force, as nothing more complex is required.
>Does anyone disagree?
CVE agents already pose a serious threat vector in and of itself.
1. Models can't currently be made inherently trustworthy, and the people claiming otherwise are selling something.
"Sleeper Agents in Large Language Models - Computerphile"
2. LLMs can negatively impact logical function in human users. However, people feel 20% more productive, and that makes their contributed work dangerous.
3. People are already bad at reconciling their instincts and rational evaluation. Adding additional logical impairments is not wise:
4. Auto merging vulnerabilities into opensource is already a concern, as it falls into the ambiguous "Malicious sabotage" or "Incompetent noob" classifications. How do we know someone or some models intent? We can't, and thus the code base could turn into an incoherent mess for human readers.
Mitigating risk:
i. Offline agents should only have read-access to advise on identified problem patterns.
ii. Code should never be cut-and-pasted, but rather evaluated for its meaning.
iii. Assume a system is already compromised, and consider how to handle the situation. In this line of reasoning, the policy choices should become clear.
> I'm optimistic that it's easier to find/solve vulnerabilities via auto pen-testing / patching, and other security measures, than it will be to find/exploit vulnerabilities after - ie defense is easier in an auto-security world.
I somewhat share the feeling that this is where it's going, but not sure if fixing will be easier. In "meatbag" red vs. blue teams, reds have it easier as they only have to make it once, blue has to always be right.
I do imagine something adversarial being the new standard, though. We'll have red vs blue agents that constantly work on owning the other side.
In open source codebases perhaps, either because big tech would be generous enough to run and generate PRs(if they are welcome ) for those issues.
In proprietary/closed source it depends on ability to spend the money these tools would end up costing.
As there is more and more vibe coded apps there will be more security bugs because app owners just don’t know better or don’t care to fix them .
This happened when rise of Wordpress and other cmses and their plugin ecosystem or languages like early PHP or for that matter even C opened up software development to wider communities.
In many small companies (e.g. startups), the attackers are far more experienced and skilled than are the defenders. For attacking specific targets, they also have the leisure of choosing the timing of the attack - maybe the CTO just boarded a four hour flight?
Does anyone disagree?
This is purely my intuition, but I'm interested in how others are thinking about it.
All this with the mega caveat of this assuming very widespread adoption of these defenses, which we know won't be true and auto-hacking may be rampant for a while.