Politics aside, if there was ever a time for an Apollo-like moonshot program to build secure infrastructure and write secure code, it's now. Thanks to wide dissemination of techniques, Attack As A Service and even AI-assisted hacking, the bar to mount attacks just keeps getting lower. The real answer is to build resilient systems with zero backdoors rather than trying to put the hacking genie back in its proverbial bottle.
Apollo was a scramble to get the absolute minimum tin can to the moon as fast as possible which wouldn't kill the 2-3 men inside it. Building anything secure is a detail-oriented endeavour. It is not solvable with a moonshot.
The question I have for cisco is: why are these special features for resisting cyberattack not standard on EVERY router?
“Cisco’s response involved shipping a large order of modified equipment, specifically designed to maintain accurate time even under radio jamming conditions.”
Probably not something most Cisco customers need or want
Timekeeping upgrades and cold-weather functionality are the flashy headlines that are easy to explain to the masses, but there's plenty in there about russian cyberattacks too. Russia has been attacking ukrane's infra over the internet for over a decade now.
Well, as usual, the answer is a mix of cost and usability. I don't know if people outside of Germany remember it, but there was a big splash when it came out the NSA hacked Merkels phone. Our chancellor! Why doesn't she have a secure phone?! What do our security authorities even do?! Well, the thing is .. she had one. And it probably wasn't hacked. But the usability of these secure phone is so bad (one common thing is that everyone needs one, which has to be compatible with each other) that she usually just used the phone that her party gave her (she was also the head of the party), which was a normal smartphone and the NSA hacked this one.
Same goes for switches etc. There are no real standards, everyone does a bit of their own thing, so you have a bunch of incompatibility. Then you need to configure them special, which takes more time and effort and so on.
And, at the end of the day, there's always the matter of cost. Resisting cyber attacks means probably different chips, which are safe according to e.g. https://en.wikipedia.org/wiki/Tempest_(codename), and the software has to be checked extra and programmed to different standards. Someone has to pay for this, simple as that.
Also, if you are not the US, the US will probably want to have a say in whether Cisco can sell you such machines. Same goes for other companies and their countries.
No, these systems are not secure in any configuration. There are exactly zero large scale commercial IT companies that can deploy systems that can protect against commercially-motivated criminal attackers let alone well-funded intelligence agencies. These companies do not have any super secret secure smartphones, or super secret secure routers, or super secret secure configurations. They are all just plain easily hacked, routinely get hacked, and the government agencies and companies using them get ransacked regularly.
Companies such as Cisco, Microsoft, Apple, etc. are just systemically incapable of deploying or even developing secure systems. They have no knowledge or expertise in that field and for their employees to develop that knowledge would take both prioritization and years to decades of learning and experimentation.
There aren’t any systems that are “secure” or “not secure” in the abstract anywhere in existence. Every system has strengths and weaknesses and is suitable for some purposes and not others, depending on your threat model.
It is perfectly possible to use products from each of the vendors you mentioned to build a high assurance system. It depends on what you build, how you configure it, and what threats you are trying to protect against.
The non-commercial/open source world isn’t exactly a bastion of impeccable security practice, either. You can counter every Solar Winds or Double Pulsar anecdote with a Heartbleed or Log4J anecdote.
But, if you look behind the headlines of every major breach, for every 1 company that got popped by a zero-day, 99 got popped by either social engineering or improper configuration/outdated software.
Why do they have poor configs and outdated software? They’re short-staffed and can’t make changes due to fear of outages. That’s a business culture problem, not a technology problem.
> Companies such as Cisco, Microsoft, Apple, etc. are just systemically incapable of deploying or even developing secure systems. They have no knowledge or expertise in that field and for their employees to develop that knowledge would take both prioritization and years to decades of learning and experimentation.
Each of these vendors employs many widely known and respected security researchers. I’ll grant their product teams can be hit or miss, but to say they have no security expertise at all is just false.
Yeah, great, name one actual high assurance system any of those vendors have actually deployed or that anybody has ever deployed using those products. Something that can protect against something simple like a team of 10 FTEs for 3 elapsed years and a competent audit verifying that where their system is actually integral to the security properties of the system. None of this, I heard from a friend of a friend that their internal security is super awesome, but they have no evidence of it and none of their team would ever claim it in public.
Now you are probably going to say something like 30 FTE-years is too much. That is only like 10 M$. That is less than the ransom Caesars Palace paid out, which is a pittance compared to how much they would be willing to pay out to avoid disruption. Being unable to make such a attack unprofitable means your security is inadequate to defend medium to large sized companies who are routinely attacked by commercial-motivated hackers. Let alone systems with actual high assurance requirements like fighter jets where multi-billion dollar attacks are more cost-effective than the missiles otherwise needed.
The unary thinking where systems are all "not secure" and thus it does not matter whether it actually works against the commercially-motivated criminal hackers who will target your systems is tiresome. "More" or "less" security does not matter, "adequate" security that protects against the current and predictable future threat landscape of commercially-motivated hackers with multi-million dollar budgets who can get multi-million dollar payouts is what matters and these companies do not reach even that basic bar.
These widely known and respected security researchers have never made any system that can protect against the modern threat landscape of commercially-motivated criminals let alone state actors. For that matter, most of them probably think that is just impossible. Excuse me if I think they have no meaningful security expertise given that they have never actually secured a system against standard attacks. They do hire some pretty good offensive researchers, but that has about as much to do with security expertise as gunmaker expertise has to do with bulletproof vest expertise.
So you can not even point to any independently verifiable systems or even evidence that supports your claim that products from those vendors can be used to make high assurance systems. In the entire world, not even a single one.
But, despite the fact that every competent offensive software specialist claims that everything is easily hacked and demonstrate that repeatedly on the software made by the giant commercial IT vendors, we should just take it on faith that those same vendors can totally make high assurance systems even though they have failed every time they tried in the past, but this time for sure they figured it out.
> So you can not even point to any independently verifiable systems or even evidence that supports your claim that products from those vendors can be used to make high assurance systems.
Sure, buddy. I'll get right on that. Right after you provide some evidence for the claims you kicked this thread off with:
> There are exactly zero large scale commercial IT companies that can deploy systems that can protect against commercially-motivated criminal attackers let alone well-funded intelligence agencies.
> Companies such as Cisco, Microsoft, Apple, etc. are just systemically incapable of deploying or even developing secure systems.
Please note: pointing to individual incidents of products from these companies getting compromised isn't evidence that no products from these companies can be used in any way to make a high assurance system. Particularly if the incident in question was a result of a user-misconfiguration of the product in question. Neither is hand-wavy claims that "competent offensive software specialist" claim that "everything is easily hacked".
The default assumption is not “everything is unhackable”. The burden of proof is on you/vendors to prove that a vendor can protect against commercial hackers.
What, are we just supposed to assume that their product works until a exhaustive third-party analysis with no access to the design proves it is mathematically impossible otherwise. Can I also claim I have a faster-than-light drive and you have to believe me until you can prove otherwise even if I have never demonstrated it actually going FTL? That is a totally ridiculous position.
But since you want some evidence. Not a single one of those companies has ever once been able to achieve a EAL5 or higher certification for their software which indicates “resistance to penetration attackers with a moderate attack potential”. They have tried numerous times for their headline products in the past and present and have at most been able to achieve EAL4 for locked down configurations the vendors personally implemented which is only adequate for “assumed non-hostile and well-managed user community requiring protection against threats of inadvertent or casual attempts to breach the system security“.
This is certification standard they are legally required to certify against to sell to the US government and which is advertised, sometimes exclusively, on all of their security pages. All of those companies have tried and failed to achieve higher certification levels for literal decades over tens of software versions and billions of dollars spent.
So yeah, your turn. Find one single counterexample. If you can not even find one measly verified/certified example of what you said in the entire world, I think I have proved my point to anybody still reading this.
If you are going to just keep hemming and hawing or post a unverified or useless certification do not bother to respond, that will just look bad for you.
I see no reason to continue this conversation. All you're doing is deflecting, moving the goal posts and making stuff up (lacking EAL5 certification isn't proof these products are bad and there is ZERO evidence Microsoft, et. al. have spent $1 let alone "billions" of dollars in any failed attempts to get EAL5 certification.)
> If you are going to just keep hemming and hawing or post an unverified or useless certification do not bother to respond, that will just look bad for you.
That's true but reading the article in full (again), nowhere does it say what other special features have been added that other companies might require
Even in the referenced article from The Register there’s no mention. Lots more context of the types of threats being faced, but no additional features beyond time keeping and better reliability at low temperature.
The absolute minimum tin that delivered three people to the moon and return them safely to Earth could have been destroyed by a fault in almost any of its innumerable subsystems and components. It was very much a detail-oriented endeavor and set the standard for unit and integration testing for decades, maybe even up through the present.
In regards to this specific case, because it is expensive and unnecessary.
Cisco’s response involved shipping a large order of modified equipment, specifically designed to maintain accurate time even under radio jamming conditions. This solution employs the Cisco Industrial Ethernet switch with an internal crystal oscillator, enabling new clock recovery algorithms for accurate timekeeping when GPS is unavailable.
These modified versions of the Cisco Industrial Ethernet 5000 series switches, tested and stress-tested in Cisco’s Austin, Texas lab, were sent to Ukrenergo. The project, which cost around $1 million, was supported by the Pentagon, the U.S. Department of Energy, and the Department of Commerce in terms of logistics and coordination. Cisco provided the equipment free of charge.
With how shit cisco's security is and how badly they're having their ass handed to them, you'd think they could roll out any cybersecurity "hardening" in their special ukraniuan firmware to other models.
Read my comment again. I wasn't making any claims outside of the fact that it would be expensive and unnecessary to implement anti-gps blocking features unless you expect to experience GPS blocking.
So you want to pay for a high accuracy local time source in every single Ethernet switch and router and wireless controller…in case the Russians disrupt GPS over one of your locations?
Most Cisco customers:
1) Do not directly use a GPS time clock in their network. They use an NTP-based source that is ultimately timed by someone else’s GPS clock.
2) Do not want to pay for extra hardware they will not use
3) Do not need their network equipment certified to -34C, or want to pay for a device that would work in such conditions.
With how shit cisco's security is and how badly they're having their ass handed to them, you'd think they could roll out any cybersecurity "hardening" in their special ukraniuan firmware to other models.
I read it. But given your response, I think you didn't. There are two, and only two, specific mitigations mentioned in the article: high accuracy local time sources and low temperature certification. If you think you read about something else, feel free to quote it in reply, but we both know that isn't going to happen because it doesn't exist.
If you think, based on the article, that Cisco shipped private security mitigations to Ukraine apart from the timekeeping algorithm, that's something you have hallucinated out of whole cloth. There is no basis for it in the linked article, and it has no precedent anywhere in their 38 year history as a company.
Also, you managed to spell "Ukrainian" incorrectly and link to an article that 404s. Good job, or something.
What cyberattack uses GPS jamming? This is just dumb journos trying to spice up the article. The Register's article says this is a side effect from trying to jam missile guidance systems. Probably by their own EW systems while defending from a missile or a shahed strike.
The trouble is that it has always been a case of "Backdoors for me but not for thee" rather than backdoor free.
And as for the price - I was surprised at the cost stated when even a quick search turned up various RTC modules for as low as 0.50p/$0.50 per module which I'm sure could be obtained (in bulk) for a lot less.
Ah well, I guess I just don't understand big business and geo-politics.
RTCs are only in the tens to hundreds of millions of parts per million accuracy per day, they’re talking crystals so we’re probably talking parts like OCXOs with three or four or more orders of magnitude better accuracy/stability (even down to less than 1ppb per day).
Still these parts are only around one hundred to a couple of hundred dollars per unit, depending on specs, but the bulk of the cost was likely the NRE of hardware design (you actually need to interface these things in, which means board design etc.) and all the software development.
In relatively small quantity electronics, BOM cost is a tiny fraction of the cost of anything.
Apollo and other historic moonshot programs were fundamentally striving to match some set of fixed natural forces with human ingenuity. The conditions to be met and overcome were measurable and knowable. They were natural-world challenges, bristling with natural-world problems. Vacuum, radiation, lunar regolith. The sound barrier, the Polio virus, and more.
Unfortunately, this is in sharp contrast to the challenges of security. Information security is an adversarial, human-centered enterprise. The forces with which we contend, armed with human ingenuity, are themselves armed with the same. It becomes a question of resources and economics. Even the cryptography on which so much relies is ultimately a question of how much breaking it is worth. If we are willing to spend enough, we can probably make it too expensive to successfully attack a system.
Plus, we know that the weakest leak in any system is generally the humans involved.
With all this in mind, we cannot possibly expect to build systems with zero vulnerabilities. It's not a moonshotable goal. What we can do - and what a mature security program will help you do - is plan for detecting, containing, mitigating, and recovering from attacks.
Those are the talking points of the people who got us into this mess who have been systemically incapable of deploying or even developing secure systems. There is no absolutely reason to listen to the liars who have repeatedly promised secure systems while being utterly incapable of doing so for literal decades. The task is not impossible, the commercial vendors like Cisco, Google, Microsoft, Amazon, Apple, etc. are just incompetent at security and are trying to poison the well by claiming that it is impossible just because they can not do it.
If you want to know what actual high security development looks like you can just look to Orange Book Level A1 certified systems and Common Criteria EAL 6/7 certified systems. Systems designed for high security with formal specifications, robust documentation, exhaustive testing, thorough review, spotless penetration testing by well-funded intelligence agencies, formal proofs of correctness, and proven deployment in high criticality settings. The Common Criteria SKPP literally required the NSA to fail a multi-month penetration test while having the full source code, internal documentation, and formal specification.
These commercial vendors believe protecting against state actors is literally impossible even though it has already been demonstrated in front of their faces for decades. Nothing they say about security is useful because they know nothing about what is needed to make systems that are actually secure.
High security development practices will, as you wisely say, go a very long way. That is not quite the same as some kind of zero-vulnerability "secure" system, however.
No system involving humans is, or ever can be, perfectly secure. At best you can make it uneconomic to attack the computerized parts of the system through networks. You've touched on a number of ways to do this, which are known and understood in the commercial world as both feasible and expensive. This is why Common Criteria has a spectrum of evaluation levels... and even EAL7 does not offer any kind of guarantee of zero vulnerabilities.
Meanwhile, adversaries can and will investigate if their goals can be achieved through human attacks or disruption. There's no need to devote extensive resources to breaking into a system if a carefully placed bomb can produce the same goal, after all.
As you say, this task is by no means impossible. How to go about it is well understood. It's merely very expensive.
Quibbling about how "perfectly secure" is impossible is a complete red herring. Yeah, using known techniques cryptography is not perfectly secure, it will just take billions of times longer than the age of universe and billions of times more energy than exists in the entire universe to break it. It is a distinction without meaning.
Making the computerized parts of a system go from 1 M$ to defeat to 10 G$ to defeat while employing human-error resistant design makes attacking the globally-addressable endpoint go from a cost of doing business to grossly uneconomical. It makes the cheapest, easiest, and most accessible way to attack into a nearly impossible wall and they have to look elsewhere to much less scalable vectors. You do not get the necessary 1,000,000% increase in security from 10% or even 100% improvements here or there. And you most certainly do not get it by listening to the people who are not even within a factor of 1,000x of knowing how to do it right.
That is a goal of the government. EO 14028 makes the biden administration the first to take cybersecurity seriously. The problems are two fold:
- An attacker only has to get lucky once. You have to do the right thing everytime.
and
- It's not clear you didn't do the right thing until it's too late.
It's possible to a much better job of cybersec than we're doing today, but everything today is such a hodgepodge of "It works, it barely works, onto the next thing" that it's hard to do so. Developers can barely be expected to write documentation, getting people to routinely do static and dynamic analysis on their code is going to be a high bar to get over.
If there was funding available to build secure systems, do we know how to do it? If we rebuilt everything from scratch today, what would we do differently?
At minimum, we should have open source hardware/software for network infrastructure. None of this closed source Cisco gear where they keep leaving embedded root passwords. I would love to see a proven sel4 microkernel used for more network appliances. C/C++ seem like obvious mistakes at this point given the industry's inability to write secure code. Rust/Ada or any GC language would be appropriate.
You're not wrong about the security issues. But competitive pressures make open-source hardware impossible in the high-end networking market. There are only a few remaining competitors, and since they consider their proprietary hardware designs to be a competitive advantage, they will never release them. Open-source hardware and software can potentially be a viable option in the low-end market, but it won't have the same efficiency or maximum performance.
High end is going to remain state-of-the-art designs, but for 95%+ of consumers, they would be fully satisfied with a one gigabyte router/switch.
I just did a search for 2013 routers (ie 10 year old designs) and gigabyte routers were already available then. While there might be more efficient implementations today, it seems like this level of hardware should be possible to create in an open way today.
Today it could be consumer 1G routers. Tomorrow that could encroach on the 10G market or 50 port switches.
I’ve yet to see “AI-assisted” attack techniques that are outside of the typical social engineering generation, can you give any examples of AI actually developing novel techniques for zero days or being used for that?
>The reliance of Ukraine’s substations on GPS for time synchronization, a standard in industrial control systems for its accuracy and affordability, becomes a vulnerability when faced with such jamming.
>Cisco’s response involved shipping a large order of modified equipment, specifically designed to maintain accurate time even under radio jamming conditions. This solution employs the Cisco Industrial Ethernet switch with an internal crystal oscillator, enabling new clock recovery algorithms for accurate timekeeping when GPS is unavailable.
That's really neat. Didn't know GPS was also used for time synchronization.
The electric grid is already extremely fragile. Its uptime is dependent on constant maintenance and repairs and many people on 24/7 standby to make more repairs in outages.
Its resilience is in the support network, not in the physical structures itself.
That support network building custom hardware in a GPS outage is that system functioning as expected. A GPS outage without a bunch of concerned humans doing everything they can to fix it isn’t going to happen.
Also on making sure that it is balanced all the time. The effort it takes to make sure that there's always input exactly as much energy as is taken out of the system is just staggering. And if only a few of the things needed to adjust it fail .. here comes the brown/blackout.
Most of the effort has been done in design. Rotating mass of generators is a nice self-correcting system, also the frequency is a near-zero latency signaling mechanism across the entire grid. Each participant can measure it and make independent decisions whether the grid needs more power or more consumption.
There is a lot of work that goes into predicting and commanding large participants and it's a complex system but some of the design is just beautiful in it's simplicity.
That's really interesting! I assumed that everybody would aim for a resilient system to reduce support costs, but you're saying the electric grid is built cheap & fragile and the cost of support is just an accepted part of it?
How come the support costs are cheaper for maintaining the electric grid than paying a bit more for hardware that requires less support?
You can't even outsource your support costs to a cheaper COL country.
Is it because the infra is subject to nature & thus always being degraded in all sorts of ways in all different environments?
Yes, once you realize even the ground you walk on is shifting cm's per year like a slow moving ocean, you see that nothing fits nicely in the world indefinitely with then precision needed for electricity infra.
There's a lot of slop in electricity infrastructure especially in those big towers with cable drapes of hundreds of feet.
Australia, the continent, has shifted by 4.9 feet since the last local mapping grid adjustment was made to GPS coordinates back in 1994.
That's about 2.7 inches a year .. that took entire regional electrical service areas along with it as a single unit with no stretching of cables.
In your scenario you need to find electrical works spanning a fault line, rigidly fixed to both sides, and built by Engineers (civil, mech, not software) clueless to drift .. like a steel bridge constructed without expansion joints to account for tempreture changes.
My whole point is that it is indeed incredibly reliable, but only because it is undergoing constant maintenance 24/7 by trained professionals.
Take the humans out of the loop and it falls over by itself very soon.
There is no situation where, say, one of its dependencies goes away and lots of smart people don’t engineer a fix ASAP. This was in response to someone expressing concern that the grid is dependent on GPS. It’s dependent on a lot of things, one of which is probably Home Depot. If any of those dependencies go away, we will re-engineer the dependency tree, because it’s not operating in a vacuum. It’s resilient and reliable because it is staffed and continuously engineered.
There's only two ways you can lose GPS from a fixed location: local jamming, or all of the satellites getting disabled somehow.
A jammer will typically be detected and stopped before any damage is done. Even owning a GPS jammer is illegal in most places.
All of the satellites going down is probably a cataclysmic scenario, like global nuclear war or solar flares. Not worth worrying about because you'll have much bigger problems.
GPS is one of the most reliable systems there is. There are redundant constellations of satellites run by different companies in different countries. It'd be near impossible to disrupt service from all of them at once.
>GPS is one of the most reliable systems there is. There are redundant constellations of satellites run by different companies in different countries.
1. GPS refers specifically to the system run by the united states government. The generic term is GNSS
2. Even though there are multiple GNSS systems out there, and modern phones support multiple, it's unknown whether the same applies to other systems. Supporting multiple systems costs more money, so I suspect hardware vendors might skimp on that, especially when the benefits are so marginal.
Basically, GPS satellites need ridiculous precision in their clocks to make navigation work. They all have atomic reference clocks to accomplish this.
Since GPS satellites are already broadcasting a ridiculously precise signal to the entire planet for free, it makes a lot of sense to use that signal when you need very precise time.
Though in cases like this where critical infrastructure is involved, I'm confused why they don't have an adequate clock source locally. You don't usually rely totally on GPS, you use it as a reference to check your local clock against.
More to the point, all GPS satellites broadcast the time - the receiver takes the multiple (different) times from multiple satellites and the satellite's known location to determine the receiver's location.
It’s related to power generation. Finding line faults, keeping equipment at 60hz or 50hz whatever they use there.
They need precision location information for multiple power plants. They can do without it but you end up with dirty power which most equipment doesn’t like very much and some will shut off and not work at all or break.
You can make a reference oscillator almost as precise as an atomic clock with a temperature compensated crystal oscillator. They cost tens of dollars.
We've also been synchronizing grid generation since long before spaceflight. It seems that the core problems here have been solved a long time ago.
Then again, I don't know much about the grid at this scale. I very much doubt that a few PPM phase difference would actually matter, but I don't really know.
Either way, it still seems insane to rely on GPS for your sole source of truth, even with its remarkable reliability.
Radio clocks [1] often don’t use GPS. Many countries have a radio signal they can pick up. In the continental US, it comes from Fort Collins, Colorado. [2]
Technically you can broadcast a new time to your neighborhood and have most people late to work (the ones having that kind of clock alarm), it's far from secure and accurate.
So the pragmatic approach that everyone is utilizing (including AWS’ latest announcement for microsecond accuracy) relies on hardware GPS. Why am I being attacked here. This is how it’s done.
I'm not the one attacking you, but I think I can explain.
Your original statement was quite absolutist "It’s really the only way to do accurate time." A statement which would have had more agreement would be "It’s really the only practical way to do accurate time."
Precision Time Protocol [1] is an alternative. CERN has been using it as part of a project to achieve sub-nanosecond accuracy over a wired network [2].
thanks, didn't notice that link at the bottom, this article seemed much more coherent. All the hacking FUD aside (sitting here trying to figure out how or why would malware perform GPS jamming?), this seems to be the root cause:
> jamming activity is primarily carried out to interfere with missile guidance systems, but a knock-on effect is disruption to grid operators
