As someone who spent over 20 years active duty, and spent a ton of my career in the IT, security, etc. side of the house:
Unlikely to be a hack, more likely to be a misconfiguration or update sent incorrectly.
That said, the timing of the disclosure and the issue are rather concerning.
Regarding the highest value targets to hit with an attack like this, you would want to target Guam, Hawai'i, and other isolated overseas locations where this would have ripple effects in the local economy. Guam specifically would cause catastrophic supply shortages, since DeCA probably supplies around 50% of the groceries on that island (that's a WAG based on my time there).
Generally agree with your assessment, but in the case of Guam or other more remote installations if there were catastrophic issues we'd just airlift food in. Costly but certainly manageable.
Hawaii I'm not sure why that would be an issue unless the whole island was attacked or shut down. Even if the on-base shops were hacked you could just go shop at Wal-Mart or Costco or any number of other locations on the islands. If there was an extended issue then the commander could authorize meal stipends as they do for some units today and then you would just go buy food off-base. Ideal? No. Manageable? Very much.
That explains why resupply doesn't work as well as in peace time
But at the same time, it's not like that came out of nowhere. If you attack a country, that country shooting back at any military installations in missile range has to be one of the expected outcomes. Given the US had all the initiative in starting the conflict I would have hoped there was a logistics plan that involved not relying on those bases to continue to exist
Maybe airlifting better supplies to the Lincoln wasn't something that needed to be done, and you are simply falling victim to social media campaigns? Maybe trying to compare "grocery stores on base in Guam were hacked" to USS Lincoln, deployed at sea and at war for almost a year runs in to operational issues aren't really things worthwhile to compare?
By the way, having served in the Army we would just be given MREs - many of us would just eat them unheated. Lucky that the US navy has all of these luxuries. I don't recall soldiers taking photos of those and sending them back and complaining about "poor working conditions" when getting shot at.
I didn't mean to imply that this would hit Hawai'i as hard as Guam, just that it would be one of the more effective places to hit with a DeCA supply chain attack.
More to the point for Guam and similar locations, canned and non-perishable food can are probably around and more can be airlifted in to get people through it. It was pretty much the standard back in the day.
The diet in Guam is already heavily dependent on shelf stable foods. I know this from personal experience but theres a surprising (to me) amount of research on it as well (c.v., https://www.guampedia.com/health-consequences-of-modern-diet...). However, it is worth noting that food on a military installation =/= food nearby, for many obvious and non obsvious reasons.
>Unlikely to be a hack, more likely to be a misconfiguration or update sent incorrectly.
Are these materially different? Assuming that there wasn't a literal command to turn off all freezers, from an organizational, non criminal perspective, to the organization the damage will be the same, the root cause will be a bug (whether exploited by chance or malice), and the fix will be the same(fixing the bug).
A couple years ago I worked on a service that had to communicate with a Siemens S7-1500 PLC. Based on my experience with that project, none of what I’ve read recently about unsecured industrial PLCs is surprising.
I opened Siemens TIA Portal and PLCSIM for the first time and thought “wow, I didn’t think the Windows 95 GUI library was still supported.” None of the PLC contractors we had hired knew how to enable TLS on the thing (user/pass eg admin/admin was their usual). Anecdote: I once spent hours reading the docs and clicking around trying to get it to accept an SSL certificate signed by a real CA and it wouldn’t go, but it accepted one I self-signed in openssl.
In all fairness, the people who are experts in the field of Siemens PLC programming are usually mechanical-ish engineers and security is not in their skill set or on their mind.
Most factories I know do not allow their PLC be accessed from the internet. They are usually on a separate Network. However, the "engineering" station (the computer running e.g. TIA Portal) sometimes is.
The PLC engineers I had contact with usually had an electrical engineering background. That's why they like PLCs in the first place with the ladder logic programming languages, grafcet and if they feel fancy a bit of structured text (assembly like) or structured control language (pascal like). They indeed did not know much about software security but a great deal about machine safety.
A real security nightmare are older OPC servers (OPC-DA) which is super reliant on DCOM. OPC is quite important to connect the PLCs to SCADA systems or 3rd party devices.
Then someone plugs in a cable because boss wants something "over there" and there's already a network that runs "over there". Or optimizes to a smart switch with vlans, and then someone else optimizes to a single vlan. It's not hard to not give a shit, or not understand, network security.
That someone can be brought into an office and shown a small diagram of the approved network topology. Then they can be shown a small diagram of the current network topology. Next, they can be asked if they're the same. If they're not, they can finally be asked if they're aware that deviating from the approved network topology without consulting infosec is grounds for termination of their employment.
You're assuming that it'll be noticed at all, and that the person noticing cares enough about it to make a big deal out of it - likely involving several layers of management.
In reality it'll likely first be noticed ten years down the line, by someone who lets out a big sigh, mutters something about "incompetent dipshits not updating documentation", and moves on with their day.
Isn't this the industry expectation in that kind of equipment? If it was signed by a real CA the cert. could expire and render the equipment unable to communicate.
That’s a tough question. If your PLC is on an airgapped LAN, admin/admin is not great security hygiene but you’ve reduced most of the risk by airgapping.
On my project the service I wrote was doing bidirectional communication with the PLC over OPCUA. The server running this pod was connected to the internet, so it was critical to have proper TLS for the OPCUA client/server. Rotating LetsEncrypt certs on the system every 45 days is a lot of toil, but using a self-signed cert that expires in 2040 from some dev laptop doesn’t pass muster in most organizations either.
That’s just the OPCUA path.. In these projects You also typically see WinCC HMIs that can talk to the PLC from anywhere on the network without TLS. And also SIMATIC Web Server pages - minimally secured by default, and the amount of info you can grok about the system in Chrome Dev Tools is troubling.
I'm not sure if you're speaking from personal experience, but most I've interacted with don't have to worry about the self-signed vs. LetsEncrypt debate. They just don't do it. Also there would be no way to do LetsEncrypt as the system is air gapped.
On an airgapped system that is is turned on once and needs to keep running for many, many years? Industrial equipment is a world of its own and internet best practices just don't transfer directly.
Some PLCs run extremely expensive machines. Some machines can't afford to have their control systems stutter or fail because that can lead to physical damage and production outages of enormous proportions. A PLC that stops communicating because a certificate just expired is absolutely not acceptable in some plants.
My mind was blown when I realized that the way tftp works is that as the machine is booting it asks the network if anyone has some software for it to run.
Well, what else can it do, really? It has to boot with pretty much zero knowledge about the external world (maybe except asking the user for the current date and time). Sure, you can hardcode an outdated list of CAs (it's always outdated because the system can be booted 10/20/100 years after it was made) in but that just opens you to unexpected obsolescence, and you usually can't put too much stuff in the bootloader anyway.
Not really dissimilar from the human upbringing: leave a baby with "bad" guardians, and it will grow up corrupted. That's a feature, not a bug: if you knew what behaviour exactly you wanted (other than "whatever Simon says"), you would just bake it in in the first place, right?
> In all fairness, the people who are experts in the field of Siemens PLC programming are usually mechanical-ish engineers and security is not in their skill set or on their mind.
Stuxnet was over a decade ago.
There should be a simple rule that everyone with the ability to understand things like PLCs should be able to grasp: your equipment does not touch the internet or external storage, period.
Those who can't grasp this concept should be shown the door with a recommendation that they find a less mentally-taxing line of work.
The worm was designed to jump across network air gaps via USB flash drives, and spread across isolated networks to other hosts. Finally, it targeted what was likely going to be a service technician laptop, which had to connect to the PLC via a data cable to update and adjust its configuration.
An attack like this can only be avoided if you never transfer any data from the outside to the inside. But that means you won't ever be able to install any form of software update or upload new config files without manually typing them in - or even reinstall the OS on any machine...
The author doesn't really claim it was a hack, just that it is a possibility. But they are charging down the path of the potential hack before asking the more obvious question: How many refrigerators exist in the military at all? And of those, how many are having problems?
Because a half dozen a day sounds plausible as standard maintenance issues, as the author acknowledges. If it were a hack, I'd expect something like 50% of them to have problems. But not knowing how many there are, I don't know how significant these incidents really are.
They are charging down that path because vulnerabilities that effect the refrigerators were disclosed the same day as 14 refrigerators failed in an absurd way. They all turned on the defrost cycle and heated the food.
The problem is the author should have put a few concise bullet points at the top. (14 freezers failed at the same time. They are all internet-controlled, and failed at the same time as a disclosure about a vulnerability. They all failed by turning on the defrost cycle and heating food.)
I really recommend skimming the article to the end.
(Unfortunately, the article really is so verbose it's a borderline rant.)
There are a couple hundred US armed forces bases each with commissaries that would be managed by DeCA.
An attack like the author hypothesized would require a LOTL modus operandi, and doing so on 14 locations wouldn't justify completely blowing up an entire LOTL operation, because it exposes indicators, registers, and tradecraft that is then shared amongst all security vendors.
The way it's framed is clickbait at its worst with the added issue of limited security experience, but the same can be said of HN in general.
Then I would suspect that this is either down to the common control system, or there has been a batch failure of the controllers in the freezers that were presumably ordered and supplied at the same time.
I've seen batch failures in radio equipment where I could predict 100% accurately which devices would fail based on the range of serial numbers.
OTOH how many bases are effected and we didn't hear about it? Those 14 bases are only the ones we know about.
Not just any failure, specifically heating the food (defrost) so it goes bad. Happening overnight, so it wouldn't be caught before it's too late.
All that could still be a coincidence, but the more coincidences start to pile up the more we have to consider other possibilities too. I do agree it would be unusual to 'waste' a vuln like that, but perhaps the implant/CVE was about to be exposed anyway.
The backbone of the US military is the logistics. It's why a US carrier being undersupplied was such a big deal. Making the US military look incompetent can very well be the goal.
Considering Iran is looking for any possible avenue to make the US look bad especially directly before an election with a president who cheerleads the military strongly while not actually putting the time or thought into what makes it strong.
This would be worth far more than the vulnerability itself to Iran right now. No real injuries causing escalation. Making a more capability adversary look foolish.
Not just that, the position to stick a thermometer into the food before serving was axed as DEI, and the position to clean the food prep surface areas of the kitchen is too beneath the warrior ethos. Buying above single ply is too expense and it's too heavy, so have fun with the ED (dual meaning).
And how many shipboard stores have been affected? Hardly something they’re going to talk about, and a far stronger candidate for attack. This could be spillover.
> I learned that commissaries (of which there are ~235 worldwide) aren’t actually independently operated by whatever military installation or base they happen to sit on.
according to the article, the denominator is ~235.
First let's acknowledge that this could very easily be a misconfiguration issue.
But, I'd be a lot more inclined to that idea if it wasn't for how they failed: they started a defrost cycle that turned the freezers into heaters, spoiling the food quicker. And the failure happened overnight, delaying discovery of the problem.
It could be just a compounding of bad luck. But an attacker with access to the specs for the freezers might be aware of how long they would stay cold after being simply shut off.
Probably not, but my closest bet would fall to Hanlon's razor:
I was curious if this was continued evidence of poor appropriations and upkeep or what... I do see "U.S. military commissary refrigeration maintenance, equipment replacement, and physical infrastructure are funded through the 5% commissary surcharge paid by customers at checkout rather than direct congressional appropriations."
So, perhaps the first place would be to follow the money - are these being repaired at the proper rate? Is this repair outsourced to third party vendors? (my guess). Is this gonna end up being the McDonald's Ice Cream machine all over again?
Really though, Hanlon's would be much easier to believe this is yet further ineptitude by those who run things (I am not going to claim malfeasance/malevolence, except a general sense of such across the board by this admin).
Since I'm not on the inside, anything I have to say would be speculative, just like the above, or the author themselves (I have no idea who it is, and perhaps they have a better beat on the ground with regards to this), but it just falls in line with "we're running out of missiles" and "sailors attempting to kill themselves".
We're so insistent on being #1, we can't admit we're in a society that is falling apart (and again, it may be the case that this IS a hack, but if I were to place my bets...)
Ineptitude, lowest cost players, etc "efficiency" indeed. You get what you pay for, and I guess 5% don't pay for a whole hell of a lot these days.
Howdy y’all, author here. Just discovered this thread after wondering why Hacker News was a linked views source to my silly little freezergate braindump.
Wanted to offer a few clarifications:
I’m not a cybersecurity expert; I’m an investigator (in a totally different field), and this was essentially me following a weird thought to see where it went. My background is in natsec so that’s where my mind goes. There have since been at least dozens of additional freezer outages reported in a similar pattern, but I'm refraining from calling/tracking down every individual weirdness based on a Facebook comment at this point since there are much larger outlets with journalists more proficient than I covering this by now. Another interesting thing - There are numerous freezers and fridges within base stores, not under the control of DeCA/DoD, and none of those appear to be impacted.
I completely agree that a bad update/configuration or other shared technical failure may be the much more boring answer. The interesting part to me is that potentially hundreds of varying systems can converge upstream into common monitoring/control infrastructure. Anyone on this forum probably understands that, however IoT was something that was a relatively new concept when I was in school, and my degrees were somewhat relevant. The average person is blissfully unaware how expansive (and how much work behind safeguarding) the IoT is.
Also, since it came up: Yup. Human written. I’m pretty firmly anti-AI as a writer and also just, like, societally. I'll be sure to add some sort of footnote detailing my ai usage at the bottom of future public facing work, because I too detest reading (or questioning if what I'm reading is) slop. Minimal LLM used for understanding technical concepts and what the fuck fridge norms are…The weird formatting, excessive bolding, neurotic parentheticals, and rant energy are, unfortunately, totally my own. Sorry guys.
This was my first ever public post and intended audience was ~ 10 friends forced to read my diatribe, not thousands of strangers very validly raising questions I am not smart enough to answer myself. Appreciate the discussion and will be further educating myself on some of the points a few of you have brought up.
Hi, I'm the poster. Sorry for the unexpected attention! I saw this on Bluesky and thought it was interesting enough to share here. If anything the style of your writing makes it stand out in a good way, we shouldn't always have polished/corporate-speak posts here.
No apologies please, I'm flattered! I figured I should put the substack I made months ago to use and start putting some of my thoughts to paper (/keyboard) when it comes to my hyperfixation-of-the-week, instead of continuing to subject my friends to these rants. It's been an interesting few days.
I came to these comments coz I was curious about the AI usage here, and FWIW I also thought it smelled AI written, but I didn't think it was slop. (IMO not all AI output is slop and not all slop comes from AI).
My main question was "did Claude do the investigation by itself or just write up the article from someone's notes?"
Also though, I'm quite willing to believe this is human written and the human just happens to have a Claudey style. The actual prose isn't that Claudey it's just the structure of how it presents ideas. But, Claude had to get that structure from somewhere. It's not that surprising to see people with that style of communication.
Nope, I'm one of those lucky folks who was privileged enough to be accused of writing in a weird and probably too mature for my high school English paper-way before the advent of gpt, lol. I've had to all but remove the em-dash from my vernacular, thanks to the ~ plague of inauthenticity ~ as I call it.
I did all the digging, researching, writing, etc myself. The reason the screenshots are from mobile is because I wrote a lot of it literally in my notes app, on my phone before I started putting together more dots. I eventually asked gpt if I was correct in my understanding about refrigeration controllers doing XYZ and if it was plausible for them to be hacked in this way, when looking for some docs I asked if it would be standard to have XYZ part of a contract public and XYZ private, and other singular questions like that. I also use a browser extension for spellcheck that (I think) uses AI to suggest phrasing improvements, which I used pretty nominally.
Is there any type of sidebar/footnote that you, as a reader, would like to see to denote AI usage during the writing and/or research process? Would you expect none? I don't plan to do many investigations such as this (literally started because I was personally impacted) but I do have a few potential article seeds about defense procurement tech I've asked a few definition/research questions to AI about and would like to make sure I'm keeping tabs on it since this is something I want to be perhaps hyper aware of moving forward. Keeping in mind this is my personal substack and I would like to maintain integrity, but also, not doing really any groundbreaking work here. Intended to be an outlet for my thoughts, not news.
To summarize for people who TLDR: 14 freezers failed at the same time. They are all internet-controlled, and failed at the same time as a disclosure about a vulnerability. They all failed by turning on the defrost cycle and heating food.
Regardless if this was a hack or a bug, the bigger lesson is that overcomplicated systems fail in catastrophic ways. Why do military commissaries need remote-controlled freezers? It seems like a very fragile, and needless, way to run a freezer.
---
But, there are some options that the author didn't consider:
1: This could be a quickly applied patch that failed.
2: This could be a "script kiddie" hack from someone who isn't a government actor.
I'm less onboard with a state actor. Generally, when a state actor has hacked something, they don't want the victim to know. In this case, if it was a state actor, I would anticipate that they would make a single freezer fail in a way that they could verify using something like a hacked video camera or otherwise by watching public social media feeds. IMO: A state actor would only "make sense" if they knew the hole was closing soon and they don't care if they're discovered, perhaps because their operation is winding down.
Iran is actively looking for ways to attack back against the United States especially against military targets without actually escalating the situation.
I'm sure some script kiddie broke into a government network, hacked an industrial process, and forced a limited supply piece of equipment into a failure mode that takes some thought and is more unique as an attack vector. It's just like buying hacks for CS source right?
But if I was an Iranian operative with instructions to damage the morale of the American war machine, I'd absolutely do a series of things like this.
It doesn't really harm anyone, but enough similar events and the families of servicemembers start to feel unsafe, which is psychologically very different than the servicemembers themselves feeling unsafe.
It's probably more benign than that, most likely firmware or a central controller failure, but still
These aren’t your typical refrigerator or freezer, these facilities have walk-ins or purpose built cold storage with multiple evaporators and condensing units. A building automation system is pretty standard for most buildings above a certain size, and monitoring and controlling the refrigeration is usually a part of it.
Unfortunately, I would wager that all BAS software is full of flaws and holes, allowing access to it for the public internet seems like a bad idea. I need to be on my company VPN to access our locally hosted BAS front end (which I have authorized access for) which seems like the bare minimum security.
Yeah I don't know why "hack" is more obvious than this. Central control pushes an update, it bugs out and cooks a dozen commissaries' frozen foods. Smart hack would be to do this randomly and fly under the radar.
If you are a country currently in a weird war like situation looking for ways to make your opponent look foolish without escalating militarily, this seems like an amazing avenue.
I'm guessing Iran will claim it as an attack even if it doesn't end up being them in the end.
I'm in the firmware bug camp too. Over/under on "the remote management server went down and a bug on all the freezers decided to put them back into some form of local control where its first action was to do a defrost cycle then put it back into offline service"?
It's also possible that, because the US is busy bombing Iran, Iran may be busy attacking the US infrastructure in any way possible? The US is also in a tepid war with russia, last I checked.
Some "cyber" crew getting a shell on an outsourced service provider and running a "defrost" command is also a totally explanation for this situation. It's also totally possible that some crew has mapped out a list of PLC entry points for various orgs and has them in a spreadsheet of "if we find a vulnerability, we should X this Y with this prestaged script that our intern / LMM cooked up last year to defrost these freezers"
This would be a bigger deal for the commissary locations outside the US, though I see none are on the list. Many of the very junior enlisted make very little money (~2400USD/month), and the low pricing at the commissary helps quite a lot. In the US, you would typically have some affordable off-base options. Overseas, it depends. Many of the locations are remote, or in places where the local groceries are significantly more expensive.
The article posits that they may just have centralized monitoring, not control. Since this didn't affect every location, I would assume it was a shared default password or something
Because they’re prefabbed walk-in coolers or freezers (at minimum, they could be purpose built cold storage warehouses) with multiple condensers, evaporators, pumps, temperature sensors, and humidity sensors. The refrigeration equipment needs some sort of control system and direct digital control is the usual way to do that these days.
This is food storage for a commissary, aka a store. They don’t use residential refrigerators.
Freezer's went into defrost melting all the frozen food and ice. If they just go down you have days to weeks to respond before everything unfreezes just based on the thermal mass and size of the freezer.
This is a much bigger deal than the freezer being down.
Weirdly I felt like it got more LLM-y the further I got in. Then I hit the part with:
But the thing I can’t get past is Fort Huachuca’s failure mode.
Not: the freezer compressor died.
Not: the power went out.
Not even: the refrigeration system stopped cooling.
Every freezer went into active defrost.
What's funny is that trump of all people is banning a lot of crap like this - internet connected chinese cars, internet connected solar panels, and other utility stuff.
But I fear the vested/wealthy interests involved in iot data mining, advertising, "relationship management" and plain "we own this"...
It (probably) prevents a comprehensive law supporting common sense.
And at some point in hindsight it will be obvious what fractions of problems were
A. This technology is inherently crap, that's our fault
B. A bored teenager broke it. Bored teenagers are a thing, it literally doesn't matter which country they are in, stop building things bored teenagers will blow up, this is also inherently our fault
C. Foreign Adversaries
It suits both mass media audience figures and a narrative of wily enemies rather than incompetence to pin everything on C and it seems eminently possible that a country with as many enemies as the US would attract this sometimes, but the reality is that both A and B are much more likely despite being embarrassing.
There's a far simpler explanation than some outside actor (either state sponsored or otherwise) deciding that the best thing they can do is to muck around with freezers.
We know there's been a severe rot of operational capabilities in the military thanks for Hegseths purges and general stupidity. It's entirely possible and quite likely that over the course of his various drunken binges he decided to get rid of people who were in charge of operational control for stuff like freezers across military bases.
I would agree, but the freezers going into high heat defrost mode seems like an intentional action from someone, whether that be incompetence or malice on the side of DeCA, or malice from a third party. If they got rid of the people commanding the freezers what to do, I feel like they'd just stay on whatever mode they were already on, rather than suddenly command all the freezers to defrost
Defrosting is very often automatic. That can involve mechanical timers, or software timers, or logic/code of any complexity. All of these things can be badly-implemented and/or become broken or stuck.
Maybe the mode they got stuck on was defrost.
if n>100 then defrost until y=3
If n is found to be greater than 100 and y subsequently never comes 'round to be 3, then defrost starts and runs forever.
Often more like a tax-free cost-plus-5% grocery store that sells some of everything (including cat food), but yeah: Still troops, and some of that stuff might become dinner.
"some of that stuff might become dinner" is splitting hairs, I feel.
Why? Because disruptions along the supply line are disruptions along the supply line. If a deep Russian ammo depot had a sudden smoking accident, you would (rightly) think it absurd for a Russian mil-blogger to quip that "technically we didn't lose ammo, the fuses are put in right before firing, we lost stuff that might become ammo." Seems less convincing now, right?
Maybe tonight's meal isn't disrupted, but the weekly meal planning is certainly disrupted. I hate to say it, but if the intent was an opportunistic hit to troop morale before a CVE got burned anyway, I'd say mission accomplished. :(
If this does turn out to be an attack, it's from decades of higher-ups ignoring cybersecurity coming home to roost.
> Maybe tonight's meal isn't disrupted, but the weekly meal planning is certainly disrupted.
This appears to be happening stateside, and most, if not all, of the listed bases have nearby grocery stores (I'm saying most based on the ones I recognize and know where they are, I didn't look up the rest). This is an annoyance, not a massive disruption.
Honestly, it probably creates more disruption for the retirees in the areas around the bases than it does the local active duty members.
I'm not splitting hairs. I'm just tryin' to talk about the commissary's role. :)
I agree that it is an important role that would be worthy of disruption for a motivated attacker.
But also: These things aren't usually at the center of vast food deserts. There's typically other ways to find some grub, like the Wal-Mart right over there.
That's the exact sort of thing the military would want us to think, because 'we were hacked by another nation' sounds a lot better than 'we fired the people responsible for food logistics'. You're not going to be able to divine the real reason this way.
Oops, you're absolutely right to call me out for that. Starting the defrost cycle without emptying the freezer first COULD lead to spoilage. The load-bearing temperature is 0 degrees Celsius — above that point, and frozen food starts to go bad.
It'll take them two weeks and then they'll find the systems were hacked half a year ago and they had industrial robots write messages on a literal chalkboard in order to share progress.
Unlikely to be a hack, more likely to be a misconfiguration or update sent incorrectly.
That said, the timing of the disclosure and the issue are rather concerning.
Regarding the highest value targets to hit with an attack like this, you would want to target Guam, Hawai'i, and other isolated overseas locations where this would have ripple effects in the local economy. Guam specifically would cause catastrophic supply shortages, since DeCA probably supplies around 50% of the groceries on that island (that's a WAG based on my time there).
Hawaii I'm not sure why that would be an issue unless the whole island was attacked or shut down. Even if the on-base shops were hacked you could just go shop at Wal-Mart or Costco or any number of other locations on the islands. If there was an extended issue then the commander could authorize meal stipends as they do for some units today and then you would just go buy food off-base. Ideal? No. Manageable? Very much.
The Fifth Fleet base in Bahrain got flattened.
But at the same time, it's not like that came out of nowhere. If you attack a country, that country shooting back at any military installations in missile range has to be one of the expected outcomes. Given the US had all the initiative in starting the conflict I would have hoped there was a logistics plan that involved not relying on those bases to continue to exist
By the way, having served in the Army we would just be given MREs - many of us would just eat them unheated. Lucky that the US navy has all of these luxuries. I don't recall soldiers taking photos of those and sending them back and complaining about "poor working conditions" when getting shot at.
One of the reasons for consitency above all else is to use the power of history.
Ahistorical is now the defacto standard. All asdumptions about America power are toilet paper.
Are these materially different? Assuming that there wasn't a literal command to turn off all freezers, from an organizational, non criminal perspective, to the organization the damage will be the same, the root cause will be a bug (whether exploited by chance or malice), and the fix will be the same(fixing the bug).
A hack implies an adversary, and intentional targeting.
I opened Siemens TIA Portal and PLCSIM for the first time and thought “wow, I didn’t think the Windows 95 GUI library was still supported.” None of the PLC contractors we had hired knew how to enable TLS on the thing (user/pass eg admin/admin was their usual). Anecdote: I once spent hours reading the docs and clicking around trying to get it to accept an SSL certificate signed by a real CA and it wouldn’t go, but it accepted one I self-signed in openssl.
In all fairness, the people who are experts in the field of Siemens PLC programming are usually mechanical-ish engineers and security is not in their skill set or on their mind.
The PLC engineers I had contact with usually had an electrical engineering background. That's why they like PLCs in the first place with the ladder logic programming languages, grafcet and if they feel fancy a bit of structured text (assembly like) or structured control language (pascal like). They indeed did not know much about software security but a great deal about machine safety.
A real security nightmare are older OPC servers (OPC-DA) which is super reliant on DCOM. OPC is quite important to connect the PLCs to SCADA systems or 3rd party devices.
Then someone plugs in a cable because boss wants something "over there" and there's already a network that runs "over there". Or optimizes to a smart switch with vlans, and then someone else optimizes to a single vlan. It's not hard to not give a shit, or not understand, network security.
In reality it'll likely first be noticed ten years down the line, by someone who lets out a big sigh, mutters something about "incompetent dipshits not updating documentation", and moves on with their day.
On my project the service I wrote was doing bidirectional communication with the PLC over OPCUA. The server running this pod was connected to the internet, so it was critical to have proper TLS for the OPCUA client/server. Rotating LetsEncrypt certs on the system every 45 days is a lot of toil, but using a self-signed cert that expires in 2040 from some dev laptop doesn’t pass muster in most organizations either.
That’s just the OPCUA path.. In these projects You also typically see WinCC HMIs that can talk to the PLC from anywhere on the network without TLS. And also SIMATIC Web Server pages - minimally secured by default, and the amount of info you can grok about the system in Chrome Dev Tools is troubling.
admin/admin is on a post-it stuck to the monitor because everyone we hire is perfectly capable of rooting a machine they have physical access to.
What is unique about your system using LetsEncrypt that you can't automate certbot to handle this task as it was designed and intended to be done?
Some PLCs run extremely expensive machines. Some machines can't afford to have their control systems stutter or fail because that can lead to physical damage and production outages of enormous proportions. A PLC that stops communicating because a certificate just expired is absolutely not acceptable in some plants.
Not really dissimilar from the human upbringing: leave a baby with "bad" guardians, and it will grow up corrupted. That's a feature, not a bug: if you knew what behaviour exactly you wanted (other than "whatever Simon says"), you would just bake it in in the first place, right?
Stuxnet was over a decade ago.
There should be a simple rule that everyone with the ability to understand things like PLCs should be able to grasp: your equipment does not touch the internet or external storage, period.
Those who can't grasp this concept should be shown the door with a recommendation that they find a less mentally-taxing line of work.
The worm was designed to jump across network air gaps via USB flash drives, and spread across isolated networks to other hosts. Finally, it targeted what was likely going to be a service technician laptop, which had to connect to the PLC via a data cable to update and adjust its configuration.
An attack like this can only be avoided if you never transfer any data from the outside to the inside. But that means you won't ever be able to install any form of software update or upload new config files without manually typing them in - or even reinstall the OS on any machine...
Because a half dozen a day sounds plausible as standard maintenance issues, as the author acknowledges. If it were a hack, I'd expect something like 50% of them to have problems. But not knowing how many there are, I don't know how significant these incidents really are.
The problem is the author should have put a few concise bullet points at the top. (14 freezers failed at the same time. They are all internet-controlled, and failed at the same time as a disclosure about a vulnerability. They all failed by turning on the defrost cycle and heating food.)
I really recommend skimming the article to the end.
(Unfortunately, the article really is so verbose it's a borderline rant.)
An attack like the author hypothesized would require a LOTL modus operandi, and doing so on 14 locations wouldn't justify completely blowing up an entire LOTL operation, because it exposes indicators, registers, and tradecraft that is then shared amongst all security vendors.
The way it's framed is clickbait at its worst with the added issue of limited security experience, but the same can be said of HN in general.
I've seen batch failures in radio equipment where I could predict 100% accurately which devices would fail based on the range of serial numbers.
Not just any failure, specifically heating the food (defrost) so it goes bad. Happening overnight, so it wouldn't be caught before it's too late.
All that could still be a coincidence, but the more coincidences start to pile up the more we have to consider other possibilities too. I do agree it would be unusual to 'waste' a vuln like that, but perhaps the implant/CVE was about to be exposed anyway.
Interesting times...
Considering Iran is looking for any possible avenue to make the US look bad especially directly before an election with a president who cheerleads the military strongly while not actually putting the time or thought into what makes it strong.
This would be worth far more than the vulnerability itself to Iran right now. No real injuries causing escalation. Making a more capability adversary look foolish.
according to the article, the denominator is ~235.
But, I'd be a lot more inclined to that idea if it wasn't for how they failed: they started a defrost cycle that turned the freezers into heaters, spoiling the food quicker. And the failure happened overnight, delaying discovery of the problem.
It could be just a compounding of bad luck. But an attacker with access to the specs for the freezers might be aware of how long they would stay cold after being simply shut off.
I was curious if this was continued evidence of poor appropriations and upkeep or what... I do see "U.S. military commissary refrigeration maintenance, equipment replacement, and physical infrastructure are funded through the 5% commissary surcharge paid by customers at checkout rather than direct congressional appropriations."
So, perhaps the first place would be to follow the money - are these being repaired at the proper rate? Is this repair outsourced to third party vendors? (my guess). Is this gonna end up being the McDonald's Ice Cream machine all over again?
Really though, Hanlon's would be much easier to believe this is yet further ineptitude by those who run things (I am not going to claim malfeasance/malevolence, except a general sense of such across the board by this admin).
Since I'm not on the inside, anything I have to say would be speculative, just like the above, or the author themselves (I have no idea who it is, and perhaps they have a better beat on the ground with regards to this), but it just falls in line with "we're running out of missiles" and "sailors attempting to kill themselves".
We're so insistent on being #1, we can't admit we're in a society that is falling apart (and again, it may be the case that this IS a hack, but if I were to place my bets...)
Ineptitude, lowest cost players, etc "efficiency" indeed. You get what you pay for, and I guess 5% don't pay for a whole hell of a lot these days.
Wanted to offer a few clarifications:
I’m not a cybersecurity expert; I’m an investigator (in a totally different field), and this was essentially me following a weird thought to see where it went. My background is in natsec so that’s where my mind goes. There have since been at least dozens of additional freezer outages reported in a similar pattern, but I'm refraining from calling/tracking down every individual weirdness based on a Facebook comment at this point since there are much larger outlets with journalists more proficient than I covering this by now. Another interesting thing - There are numerous freezers and fridges within base stores, not under the control of DeCA/DoD, and none of those appear to be impacted.
I completely agree that a bad update/configuration or other shared technical failure may be the much more boring answer. The interesting part to me is that potentially hundreds of varying systems can converge upstream into common monitoring/control infrastructure. Anyone on this forum probably understands that, however IoT was something that was a relatively new concept when I was in school, and my degrees were somewhat relevant. The average person is blissfully unaware how expansive (and how much work behind safeguarding) the IoT is.
Also, since it came up: Yup. Human written. I’m pretty firmly anti-AI as a writer and also just, like, societally. I'll be sure to add some sort of footnote detailing my ai usage at the bottom of future public facing work, because I too detest reading (or questioning if what I'm reading is) slop. Minimal LLM used for understanding technical concepts and what the fuck fridge norms are…The weird formatting, excessive bolding, neurotic parentheticals, and rant energy are, unfortunately, totally my own. Sorry guys.
This was my first ever public post and intended audience was ~ 10 friends forced to read my diatribe, not thousands of strangers very validly raising questions I am not smart enough to answer myself. Appreciate the discussion and will be further educating myself on some of the points a few of you have brought up.
My main question was "did Claude do the investigation by itself or just write up the article from someone's notes?"
Also though, I'm quite willing to believe this is human written and the human just happens to have a Claudey style. The actual prose isn't that Claudey it's just the structure of how it presents ideas. But, Claude had to get that structure from somewhere. It's not that surprising to see people with that style of communication.
I did all the digging, researching, writing, etc myself. The reason the screenshots are from mobile is because I wrote a lot of it literally in my notes app, on my phone before I started putting together more dots. I eventually asked gpt if I was correct in my understanding about refrigeration controllers doing XYZ and if it was plausible for them to be hacked in this way, when looking for some docs I asked if it would be standard to have XYZ part of a contract public and XYZ private, and other singular questions like that. I also use a browser extension for spellcheck that (I think) uses AI to suggest phrasing improvements, which I used pretty nominally.
Is there any type of sidebar/footnote that you, as a reader, would like to see to denote AI usage during the writing and/or research process? Would you expect none? I don't plan to do many investigations such as this (literally started because I was personally impacted) but I do have a few potential article seeds about defense procurement tech I've asked a few definition/research questions to AI about and would like to make sure I'm keeping tabs on it since this is something I want to be perhaps hyper aware of moving forward. Keeping in mind this is my personal substack and I would like to maintain integrity, but also, not doing really any groundbreaking work here. Intended to be an outlet for my thoughts, not news.
Regardless if this was a hack or a bug, the bigger lesson is that overcomplicated systems fail in catastrophic ways. Why do military commissaries need remote-controlled freezers? It seems like a very fragile, and needless, way to run a freezer.
---
But, there are some options that the author didn't consider:
1: This could be a quickly applied patch that failed.
2: This could be a "script kiddie" hack from someone who isn't a government actor.
I'm less onboard with a state actor. Generally, when a state actor has hacked something, they don't want the victim to know. In this case, if it was a state actor, I would anticipate that they would make a single freezer fail in a way that they could verify using something like a hacked video camera or otherwise by watching public social media feeds. IMO: A state actor would only "make sense" if they knew the hole was closing soon and they don't care if they're discovered, perhaps because their operation is winding down.
I'm sure some script kiddie broke into a government network, hacked an industrial process, and forced a limited supply piece of equipment into a failure mode that takes some thought and is more unique as an attack vector. It's just like buying hacks for CS source right?
Could be the Iranians, or someone aligned, conducting anti-morale operations. Could be the start of a series of small but annoying failures.
But if I was an Iranian operative with instructions to damage the morale of the American war machine, I'd absolutely do a series of things like this.
It doesn't really harm anyone, but enough similar events and the families of servicemembers start to feel unsafe, which is psychologically very different than the servicemembers themselves feeling unsafe.
It's probably more benign than that, most likely firmware or a central controller failure, but still
Unfortunately, I would wager that all BAS software is full of flaws and holes, allowing access to it for the public internet seems like a bad idea. I need to be on my company VPN to access our locally hosted BAS front end (which I have authorized access for) which seems like the bare minimum security.
Could be a hack or a design flaw. I await the root cause analysis.
I'm guessing Iran will claim it as an attack even if it doesn't end up being them in the end.
It's also possible that, because the US is busy bombing Iran, Iran may be busy attacking the US infrastructure in any way possible? The US is also in a tepid war with russia, last I checked.
Some "cyber" crew getting a shell on an outsourced service provider and running a "defrost" command is also a totally explanation for this situation. It's also totally possible that some crew has mapped out a list of PLC entry points for various orgs and has them in a spreadsheet of "if we find a vulnerability, we should X this Y with this prestaged script that our intern / LMM cooked up last year to defrost these freezers"
This is food storage for a commissary, aka a store. They don’t use residential refrigerators.
This is a much bigger deal than the freezer being down.
Not exactly strong evidence presented here, but it wouldn't be a surprise either
But I fear the vested/wealthy interests involved in iot data mining, advertising, "relationship management" and plain "we own this"...
It (probably) prevents a comprehensive law supporting common sense.
A. This technology is inherently crap, that's our fault
B. A bored teenager broke it. Bored teenagers are a thing, it literally doesn't matter which country they are in, stop building things bored teenagers will blow up, this is also inherently our fault
C. Foreign Adversaries
It suits both mass media audience figures and a narrative of wily enemies rather than incompetence to pin everything on C and it seems eminently possible that a country with as many enemies as the US would attract this sometimes, but the reality is that both A and B are much more likely despite being embarrassing.
> "I'm sorry I'm familiar with that function. Let me research enabling defrost for you."
We know there's been a severe rot of operational capabilities in the military thanks for Hegseths purges and general stupidity. It's entirely possible and quite likely that over the course of his various drunken binges he decided to get rid of people who were in charge of operational control for stuff like freezers across military bases.
Maybe the mode they got stuck on was defrost.
If n is found to be greater than 100 and y subsequently never comes 'round to be 3, then defrost starts and runs forever."some of that stuff might become dinner" is splitting hairs, I feel.
Why? Because disruptions along the supply line are disruptions along the supply line. If a deep Russian ammo depot had a sudden smoking accident, you would (rightly) think it absurd for a Russian mil-blogger to quip that "technically we didn't lose ammo, the fuses are put in right before firing, we lost stuff that might become ammo." Seems less convincing now, right?
Maybe tonight's meal isn't disrupted, but the weekly meal planning is certainly disrupted. I hate to say it, but if the intent was an opportunistic hit to troop morale before a CVE got burned anyway, I'd say mission accomplished. :(
If this does turn out to be an attack, it's from decades of higher-ups ignoring cybersecurity coming home to roost.
This appears to be happening stateside, and most, if not all, of the listed bases have nearby grocery stores (I'm saying most based on the ones I recognize and know where they are, I didn't look up the rest). This is an annoyance, not a massive disruption.
Honestly, it probably creates more disruption for the retirees in the areas around the bases than it does the local active duty members.
I agree that it is an important role that would be worthy of disruption for a motivated attacker.
But also: These things aren't usually at the center of vast food deserts. There's typically other ways to find some grub, like the Wal-Mart right over there.
“If sovereign and subject are in accord, put division between them.” —Sun Tzu, The Art of War
Should be much closer to the top of the article. Otherwise this is just weird and potentially dangerously wrong research.