It’s entirely possible that Iron Mountain will be happy to turnover the data but needs a court judgement to do so without incurring more legal exposure. The article says the system belongs to OSS. If this is something like a colo arrangement then probably that is the only reasonable position to take. They can’t just go crack open a customer’s server and pull data out of it for whoever asserts a claim on it.
Ok, it sucks, losing data always sucks, but what ever happened to the 3-2-1 backup rule? even if your off site backup provider goes tits up, your on site backup is still there.
If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)
> but what ever happened to the 3-2-1 backup rule?
I wasn't 100% sure, but the data backs this up[0]; the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.
I can pretty clearly remember hearing about the "3-2-1 backup rule" by name 10 or 15 years ago on the Security Now podcast with Steve Gibson. So it is _at least_ that old, but probably even older. The concept of having multiple backups is not at all new, however. Since I've been involved with computers (~30 years), I have heard folks say things like "two is one, one is none" in reference to backups and data replication in general.
Also, this TV station has not been sending their show archives to Iron Mountain for the past 70 years. Probably what happened was: In 2019, they had an on-site or nearby archive. Then some enterprising IT director came along and convinced management that they would save a bunch of money and time by outsourcing everything to a third party who said they could do it for cheap. Very confident in their genius, they then scrapped their own archive entirely.
Part of me pities them for ending up in this situation, the other part of me believes they got everything they asked for.
The concept of multiple copies in multiple locations already existed when backups on CD-R were common, so 20+ years ago. The “2” in the 3-2-1 rule refers to using different physical storage media and predates cloud storage.
This has been a rule for like 30+ years (I have only been in the industry that long). Whether it was called "3-2-1" before that or not, the practice is the same. Have an onsite backup, have an offsite backup. This isn't a new concept at all.
Your description leaves the reader wanting. You make it sound like having a backup on an HDD in one location and then having another copy on the exact same make/model HDD in another location is a-okay. What if the vendor had a defect that strike the make/model of HDD used?
> the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.
They signed up with them in 2019. Not long ago.
Even if it is a "new" concept, common sense isn't. Pretty much anyone who cares about their data will do the due diligence, and this is not something that requires technical chops. This has been a concern for well over 20 years!
It really sounds like some senior management trying to be cheap.
> I also remember things like "have one backup in one place, another in another place" since before this mnemonic
I've heard variations of this all the way back to some 1990s vocational IT courses in the country I live, and it wasn't even technologically advanced at the time. It's hard to have any sympathy for someone who misses this concept in today's age.
The 3-2-1 rule is ~20years old at this point. But just because an archive is "old" doesn't mean we can't apply modern standards to that archive as time marches on, otherwise the BM_ARC would have never been build and opened a couple of years ago.
I mean, that 70+ year old archive must have been digitised at some point, unless Iron Mountain was storing the reel to reel tapes and/or film from the original recordings. So some "modernation" has been applied to it since its creation.
Wouldn't be surprised if it wasn't digitised. TV was using BetacamXR up until very recently (and still might be). Tape is pretty reliable and lasts awhile.
Now add in tons of other legacy formats and digitising becomes a pretty big cost for a public station.
The backups (and servers) seem to still exist, but KETC's vendor is estranged with its own hosting / siting provider, Iron Mountain, which is a going concern.
IM seem to be holding KETC's data hostage, which is Not Cool.
It's not that simple. If I'm a host and someone shows up without the keys or passwords and says "that's my data, I was a client of a client of yours", I have no idea how to tell if that's true and no idea how to deliver the data.
Maybe they mixed data from many companies on the same volume like S3 does. Maybe there were ACLs on that. Maybe it includes PII. Maybe it's a bunch of shards and needs an index on OSS servers to reassemble. Maybe it's encrypted by keys OSS had.
From the article: It's not clear if the channel has backups of its data, but based on its determination, we suspect not. This is strange, though, especially for a major organization, as one can easily purchase a six-bay NAS on Amazon and populate it with the best HDDs. This also goes against the 3-2-1 backup rule, which says you should keep three copies of your files stored in two different mediums, with one of them running offsite. Hopefully Nine PBS will implement a better backup system once it regains access to its data.
It's worth noting that PBS affiliates are non-profits that have fairly constrained budgets, especially now that the Corporation for Public Broadcasting has been dissolved due to DOGE.
Non-profit IT has always seemed, to me at least, to be a land of compromises. There's rarely enough money, knowledge, or time to do things "the right way", so you have to do the best you can. If I could only choose one backup to have, it'd be remote on a provider like Iron Mountain. It would seem as if they are storing things on Iron Mountain through an intermediary company, who may or may not have actually paid the bill.
The guillotine is too nice for the people that dissolved the CPB. That just makes me so mad. How many children learned via the CPB like me? America isn’t even the same without CPB and I guess that’s why these traitors did it.
It's nice to know that even in the end the CPB lived up to its mission and promise and swallowed its poison pill rather than be corrupted by partisan politics, but it's also gross to have seen that forewarned poison pill needed in my lifetime. There's some hope that we can still find the next Mr. Rogers to put a heart on a sleeve in front of some future Congress and build a sequel CPB with similar strong core principles.
I thought this would be like the backups I've heard broadcasters have of every broadcast they've ever had (I've heard of people being able to request 20 year old news clips before).
But just 50TB of data? Duplicating that would have been cheap and trivial.
> But just 50TB of data? Duplicating that would have been cheap and trivial.
Seriously. I'm just a random nerd and I have a home server with 96TB of raw disk space divided in to two RAIDZ1 pools which cost me around $2500 to build five years ago and has required basically zero maintenance since then.
Hard drive prices are absolutely bonkers right now but even with that in mind if I were to build a modern version of this same machine I could do it for under $10k using brand new hardware and it'd have double the storage
Yeah, we are talking about 3 20TB drives in a closet here.
I'm an independent photographer and I have my 20TB drives behind my monitor, backing up to backblaze with an offsite synology. Keeping this archive up does take time and money, but PBS should have the budget and IT deparment for this.
I think for most of broadcasting history video was fairly low quality (according to today’s standards). So I am not totally surprised that this number is low.
Most broadcasters would consider their ultimate back up as the tape library/vault of their broadcast video tapes. Older stations will have a large collection of analog tapes. Most will only be digitized if someone requests it as capturing their entire library just to have digital files few people will ever want is an expense a station just isn't going to do for the lulz.
I wonder how they got the contract? Seems to be 4 people 'employed' at this company. Of which, one is retired, one is "Linkedin Member", one is a controller and the last one is a social media person: https://www.linkedin.com/in/isabelle-anderson-7aa250b5/
So, just the person to deposit checks and a social media person to get paid. Who exactly is left to work on storage infrastructure?
To be somewhat fair, Iron Mountain is just a storage facility. OSS were likely the consultants who worked out how to get the archival data from the station's tapes into Iron Mountain and then get the data back out when needed. So they probably offered a worthwhile service, they just eventually stopped doing it and left the station in a lurch.
And they sent OSS a legal demand letter to keep paying the Iron Mountain bill for their data. If OSS actually was using IM and not just running a scam, then the data should be pretty safe, if just not accessible for a few weeks/months.
It's funny to me that entire companies exist that have a fraction of storage that I personally have in a datacenter on flash let alone in total. 50 TiB is so little. It's like $7k annually on R2 and under a thousand bucks on S3 Glacier Deep Archive. I suppose it's just whoever knew the cousin of whoever at whatever position of authority but it's funny how just using deep glacier means he would have such insane margins over paying Iron Mountain.
Your assuming it's all digital and can be stored on drives. It might be a large amount of physical film reel, magnetic tape, and tons of mixed legacy formats
Don't trust Internet Archive as your only backup method though. There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
> Don't trust Internet Archive as your only backup method though.
Agreed, no doubt about it.
> There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
What sort of content? All the content I've judged to be of public interest and uploaded to Internet Archive, is still online there today. But I'd understand that depending on the type of content, that might not be true for everythibng.
Inability to access items does not mean deleted. In the case of PBS content, it is unlikely access would be lost. I cannot speak to your use case, but the Patron Help Desk can confirm if an item is relevant to their preservation efforts if you'd like to check before uploading. Email info at archive dot org to inquire.
Someone dumping their Twitch stream or random Youtube content into the storage system will certainly have that content deleted and be banned if prior arrangements have not been made, for example. Don’t do that.
It's incredibly frustrating. The Internet Archive offers a Vault product, where you pay $2-3/GB one time for perpetual storage into the future of your collections, items, media, etc. This pricing was previously predicated on the cost decline curve of storage. It's gotten a bit more expensive due to the AI bubble inflating the cost of storage, but to your point, the Internet Archive is a superior custodian than some rando tier 2 cloud provider who won a contract from a non profit and will lose all of the data when they go bust.
> This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Okay hang on now.. that strategy comes with its own risks. Risks that I imagine PBS was trying to avoid by using a third party that specializes in archival data storage. This is a silly takeaway.
Edit: I was too quick to comment. I think they are suggesting that the NAS should have been used in addition to the third party storage, which is actually a great takeaway.
I read it as a statement of just how little data this is by modern standards more than anything else. This company is going out of business and PBS is losing this valuable data, and there's legal drama and multiple corporate entities involved, and if you read all that you might think this is some amazing amount of data that is actually difficult to deal with. Instead it's something I could stroll down to Best Buy right now, plonk down my personal credit card, and walk out with enough stuff to do a credible backup job myself, and we're not even challenging my credit limit or anything.
10 years ago, this was a lot of data. It isn't anymore and it makes it that much more bizarre that they're having that much trouble getting it back. It would be less to just give them access and let them download it themselves and perhaps stick it somewhere else temporarily than to pay lawyers to deal with legal filings.
To put it in vaguely physical terms, what once was a university library is now a good solid bookshelf full of stuff. Legally wrangling over a huge library is one thing, but if it's just a bookshelf, metaphorically, let them come in for an hour and clean it out themselves rather than bringing in the law. Obviously not a perfect metaphor on data size, this is more about how lawsuits are more of a hassle then just giving it to them.
The reason nobody's just giving it to them isn't because the amount of data's too much of a hassle, it's because nobody can figure out who has the keys to the now-defunct storage company, and Iron Mountain quite reasonably won't hand out their customer's data to a third party without a court order.
More like 20 years ago (sorry). 10 years ago this would have been slightly stretching, but not by much - 11x8TB raidz3. Shuckable drives would have cost $3300 (plus the chassis), and already out of the husk probably 1.5x more. So let's say $5500 total even back then. (I built my current array around the same time)
Funny, these days you could do it with 4 spindles (4x24TB raidz2) but with the rampant price inflation in storage, you're still looking at $4500.
I recently pulled up my resume from 25 years ago and came across this amusing (by today's standards) tidbit: "Re-located a web-server farm between two data centers with zero downtime. This included migrating over 150GB of data between three Network Appliance Filers and moving more than 12 racks of equipment."
> It's not clear if the channel has backups of its data, but based on its determination, we suspect not. This is strange, though, especially for a major organization, as one can easily purchase a six-bay NAS on [Amazon] and populate it with [the best HDDs].
I think they just put that stuff in so they could shill their affiliate marketing articles rather than out of any actual concern.
Having that be there only backup solution would have been silly and dangerous. Having it be part of their overall backup solution including with something like the provider would have been smarter and better and prevented this entire problem
Yeah I bought a five bay Synology NAS two or three years ago I think it and three 18TB WD Red drives total cost $3,000CAD? I was going to save up for two more drives. Now each drive is $1,000 CAD a 150% jump. edit: I checked $1,350 each so about 250% jump ugh!
Technically you're still correct even when you would advocate for both. Every additional copy increases the risk that the data is accessed by someone else.
A NAS is at risk from malware that can erase all of it. 100GB Bluray discs exist and work more easily for archival storage, as does multi-terabyte tape drive storage, both of which can stay disconnected At least two offline copies, combined with error correction codes, are advised. A cloud backup is also advised.
It's really too bad the disc based storage options made after Blu-ray were discontinued. They made a lot of progress and seem a lot more user friendly than tape.
When I worked for Syncplicity, we offered an on-site version for "cloud" storage specifically for this reason. (Ok, not specifically for this reason, it was because customers demanded it and an on-site version of our product was a great differentiator in our niche.)
> This unidentified vendor provided “hardware, software and cloud-storage services” for storing the public broadcaster’s archival materials and other data.
By my calculation using what I can find of 2019 pricing, a more reasonable solution would have been:
Synology RackStation RS1219+ - $1,200
8x Ironwolf Pro 12TB drives - 8x $450 = $3600
so for a usable RAID-6 capacity of 65TB about $4800 + OSS, or double that if they have two locations they could and replicate between them.
I know there would be cheaper hardware options available (I'm partial to older surplus enterprise hardware myself) but it isn't particularly reasonable to expect a small business to choose a more DIY (at least in terms of software configuration) option. You could step down to 10TB drives, but it seems unlikely they would have wanted to risk that not being quite enough.
On top of that, since 2019 is 7 years ago, it is reasonable to expect they would have had a hardware refresh at some point, I would think 5 years would be more reasonable than 3 year for something like this. At that point, the nas would be more ($1300) but the drives would be less (8 * $300), so add either another $3700 (double if both are replaced, but maybe they figured the offsite backup could be stretched even longer than 5 years)
So, rather than $2500, it would probably have been more like $8500 + OSS or $13,300 without OSS. Still that would have meant a proper are archive and a proper backup and they wouldn't be in this mess, so I completely agree with you about what they should have done, but think you were underselling the price a good bit.
Not PBS but an affiliate broadcaster and the data is not lost as in destroyed, but legally encumbered.
Full headline: PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct — public TV channel sues Iron Mountain data center, which hosts archival materials, to ensure preservation
The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.
> Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.
This does make sense. I'm guessing the first request some rep just went sure its you're data why not, but in reality, it's OSS's data, and you paid OSS to store it. It's also likely encrypted in someway.
TLDR still, is especially when it's a reasonable amount of data, just have more than 1 backup.
I store my belongings with a storage company. They rent facilities from a property owner. The storage company goes under while my stuff is still in the storage unit. I go to the property owner and say my stuff was in there. They say, no, that's owned by the storage company. Incorrect - the belongings were owned by me the entire time, the storage company was just holding it for me. I need to prove it was mine, but I don't lose property rights just because it's in someone else's possession.
Likewise if I store data with a cloud provider. Sure, Iron Mountain shouldn't just offer it up without proof, but if I give them notice of a legal case they should be sure to hold on to it, and if I win the case they need to hand it over to me. All I need to do is provide proof that I am the rightful owner of those goods.
In the edge case that it wasn't Nine PBS requesting the Nine PBS data, you've now opened your company to a huge can of worms. Only OSS knows who Nine PBS is in their database, hence why, from a liability standpoint, it makes sense to say it's OSS's problem.
Also your storage company case is actually it's own can of worms, see the Lego drama from a couple months ago about that. It seems obvious, but in the eyes of the law there's more paperwork than you'd think.
This local PBS wasn't playing it's own footage 24/7. The data in question could be just their local segments, an hour or two of local coverage and maybe their fundraisers.
Isn't 50TB just 6 years of videos, in even the best case (1GB/h)?
They weren't recording in 4K last century.
One hour of 1970's-grade NTSC video is around 300MB.
More to the point, it's not like they archived 70 continuous years of video. Many local PBS stations produce an hour or less of video each week. This being a St. Louis station, however, it's probably somewhat more.
Calling out Nine PBS for not accounting for the fact that their vendor could effectively just decide to commit crimes to get out of their contract and their vendor’s vendor could just decide to compound the problem in the face of a court order is wild. Yes, we’ve all learned to anticipate everyone else on the planet being an absolute shitheel; no, Nine PBS did not do anything wrong by assuming the professionals they were paying to render a professional service to them would do so. We’re all paranoid engineers, but this is absolutely no goddamn way to run a society.
I work in several labs where I have to give this advice, which some labs are smart enough to make an ersatz backup that we store somewhere (PI's house) as a cheap offsite. Even a partial recovery is great in the face of fire/flood/explosion, or the most common error: IT fucks it up and then says "oh well".
Hope someone has some copies and they can rebuild, what a nightmare.
>This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Look, I know that a certain percentage of Tom's Hardware articles are just sales pitches for a NAS, but this is asinine. Iron Mountain deals largely in tape storage. 50TB of tapes fits in one of their small archive boxes. If they're using the latest tech (LTO-9), it fits on 3 tapes.
This is exactly how you are supposed to store archival media for decades, not on a NAS. A NAS is for storing the files you need to actively access.
(I've submitted this to HN's mods via email. The original submission was to a very poor TomsHardware link.)
OT: The tendency of local broadcasters to fail to identify either on websites OR QUITE OFTEN ON BROADCASTS THEMSELVES even vaguely where they operate is ... a long-standing annoyance of mine. Sure, local audiences may know, but those passing through, or catching the signal at a distance, likely won't.
I've spent more than a few nights at higher elevations (campgrounds, highways) listening to clear channel stations reporting on weather and traffic "in the local area", sometimes for hours, without ever hearing what "the local area" is. That for signals travelling hundreds of km, and not infrequently multiple thousands of km.
(Don't get me started on entire countries where stations are identified simply by number, e.g., "Channel 4" in the UK.)
It's actually funny you mentioned the lack of indicating call sign or operating location. I saw something similar done on purpose.
I worked at WCCO in Minneapolis briefly a few years ago. Right around that time, CBS corporate was attempting to standardize all their stations and associated websites. In their mind, that meant making everything "CBS News [Location]" and stripping just about everything else from the public marketing.
WCCO, KPIX in Los Angeles, and a few other stations battled back about this because the local name recognition of the call sign was stronger than CBS would be by itself.
So now, when you bop around the "Local News" section of the CBS Website, some cities have logos with the call letters, and some do not.
In short, when you see a station that that's difficult to identify, try finding out who owns it. Very often the ownership is to blame for one reason or another.
The United States' Federal Communications Commission (FCC) enforces specific requirements for identification that must be followed by all terrestrial radio and television stations. Stations must, when they sign on, sign off, and as close to the top of each hour as feasibly possible (such as within a "natural break" in programming, like a commercial break), present a visual (television) or aural (radio) station identification that contains, at minimum, the station's callsign, followed by its designated city of license. As a courtesy, top-of-hour identifications may also contain additional information, such as frequencies and a declaration of the station's ownership.[6] Only the name of the licensee, the station's frequency or channel number as stated on its license, and/or network affiliations, may be inserted between the call letters and station location.
Technical information is provided. This can include station identification (call sign and city of license), transmitter power, frequency or channel number, translators used, transmitter locations, list of broadcast engineers, and/or studio/transmitter links (STL).
In my experience, the 1980s were a time when some TV stations would still sign off for a few hours, perhaps only on Sunday night. It is unusual today in the United States, for commercial radio or TV stations to sign off and sign on. When stations did sign off/on, I did hear some of them broadcasting an extended Station ID. A man's voice would call out those details listed. They would mention what mountain peak their main transmitter was on, and the wattage of "peak effective radiated power". Since I lived near the Mexican border, there were stations with Mexican licenses who broadcast in English programming, except for the Mexican National Hour. "Border Blasters", as they were called.
Pretty clear that this is AI copied slop that no one at Toms Hardware bothered editing as it has no clarity that Nine PBS is the PBS affiliate in St Louis, Missouri.
Just email…
If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)
I wasn't 100% sure, but the data backs this up[0]; the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.
- [0] Interestingly, Ngram (https://books.google.com/ngrams/graph?content=%5B3-2-1+backu...) says ~2010 while Trends (https://trends.google.com/explore?q=3-2-1%2520backup%2520rul...) says slow build up from ~2020, guess "everyday" language would sit somewhere around those two different truths.
Also, this TV station has not been sending their show archives to Iron Mountain for the past 70 years. Probably what happened was: In 2019, they had an on-site or nearby archive. Then some enterprising IT director came along and convinced management that they would save a bunch of money and time by outsourcing everything to a third party who said they could do it for cheap. Very confident in their genius, they then scrapped their own archive entirely.
Part of me pities them for ending up in this situation, the other part of me believes they got everything they asked for.
Which is not as thorough, but still makes the point that you should do more than one backup
They signed up with them in 2019. Not long ago.
Even if it is a "new" concept, common sense isn't. Pretty much anyone who cares about their data will do the due diligence, and this is not something that requires technical chops. This has been a concern for well over 20 years!
It really sounds like some senior management trying to be cheap.
I've heard variations of this all the way back to some 1990s vocational IT courses in the country I live, and it wasn't even technologically advanced at the time. It's hard to have any sympathy for someone who misses this concept in today's age.
I mean, that 70+ year old archive must have been digitised at some point, unless Iron Mountain was storing the reel to reel tapes and/or film from the original recordings. So some "modernation" has been applied to it since its creation.
Now add in tons of other legacy formats and digitising becomes a pretty big cost for a public station.
IM seem to be holding KETC's data hostage, which is Not Cool.
Maybe they mixed data from many companies on the same volume like S3 does. Maybe there were ACLs on that. Maybe it includes PII. Maybe it's a bunch of shards and needs an index on OSS servers to reassemble. Maybe it's encrypted by keys OSS had.
Non-profit IT has always seemed, to me at least, to be a land of compromises. There's rarely enough money, knowledge, or time to do things "the right way", so you have to do the best you can. If I could only choose one backup to have, it'd be remote on a provider like Iron Mountain. It would seem as if they are storing things on Iron Mountain through an intermediary company, who may or may not have actually paid the bill.
So, y'know, fraud's a possibility here.
https://www.newson6.com/story/5e3687772f69d76f6209d656/tape-...
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
— Andrew S. Tanenbaum
Cram a few 12 TB drives into that sucker and you’ve got all the storage you need for their whole 50 TB archive.
I thought this would be like the backups I've heard broadcasters have of every broadcast they've ever had (I've heard of people being able to request 20 year old news clips before).
But just 50TB of data? Duplicating that would have been cheap and trivial.
It depends - especially if you go back further than 20 years. The Wikipedia article for "Lost television broadcast" goes into detail:
https://en.wikipedia.org/wiki/Lost_television_broadcast
Seriously. I'm just a random nerd and I have a home server with 96TB of raw disk space divided in to two RAIDZ1 pools which cost me around $2500 to build five years ago and has required basically zero maintenance since then.
Hard drive prices are absolutely bonkers right now but even with that in mind if I were to build a modern version of this same machine I could do it for under $10k using brand new hardware and it'd have double the storage
I'm an independent photographer and I have my 20TB drives behind my monitor, backing up to backblaze with an offsite synology. Keeping this archive up does take time and money, but PBS should have the budget and IT deparment for this.
I wonder how they got the contract? Seems to be 4 people 'employed' at this company. Of which, one is retired, one is "Linkedin Member", one is a controller and the last one is a social media person: https://www.linkedin.com/in/isabelle-anderson-7aa250b5/
So, just the person to deposit checks and a social media person to get paid. Who exactly is left to work on storage infrastructure?
https://www.linkedin.com/company/osstorage/people/
Agreed, no doubt about it.
> There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
What sort of content? All the content I've judged to be of public interest and uploaded to Internet Archive, is still online there today. But I'd understand that depending on the type of content, that might not be true for everythibng.
Someone dumping their Twitch stream or random Youtube content into the storage system will certainly have that content deleted and be banned if prior arrangements have not been made, for example. Don’t do that.
https://old.reddit.com/r/DataHoarder/comments/sq6wbq/please_...
https://archive-it.org/vault/
(huge fan of Stokes as a fellow archivist, recommend the documentary on her: https://en.wikipedia.org/wiki/Recorder:_The_Marion_Stokes_Pr...)
Okay hang on now.. that strategy comes with its own risks. Risks that I imagine PBS was trying to avoid by using a third party that specializes in archival data storage. This is a silly takeaway.
Edit: I was too quick to comment. I think they are suggesting that the NAS should have been used in addition to the third party storage, which is actually a great takeaway.
10 years ago, this was a lot of data. It isn't anymore and it makes it that much more bizarre that they're having that much trouble getting it back. It would be less to just give them access and let them download it themselves and perhaps stick it somewhere else temporarily than to pay lawyers to deal with legal filings.
To put it in vaguely physical terms, what once was a university library is now a good solid bookshelf full of stuff. Legally wrangling over a huge library is one thing, but if it's just a bookshelf, metaphorically, let them come in for an hour and clean it out themselves rather than bringing in the law. Obviously not a perfect metaphor on data size, this is more about how lawsuits are more of a hassle then just giving it to them.
More like 20 years ago (sorry). 10 years ago this would have been slightly stretching, but not by much - 11x8TB raidz3. Shuckable drives would have cost $3300 (plus the chassis), and already out of the husk probably 1.5x more. So let's say $5500 total even back then. (I built my current array around the same time)
Funny, these days you could do it with 4 spindles (4x24TB raidz2) but with the rampant price inflation in storage, you're still looking at $4500.
A whopping 150GB!
I think they just put that stuff in so they could shill their affiliate marketing articles rather than out of any actual concern.
At that point, why wouldn't you pick a major cloud vendor? This feels like more of a governance failure.
> This unidentified vendor provided “hardware, software and cloud-storage services” for storing the public broadcaster’s archival materials and other data.
I know there would be cheaper hardware options available (I'm partial to older surplus enterprise hardware myself) but it isn't particularly reasonable to expect a small business to choose a more DIY (at least in terms of software configuration) option. You could step down to 10TB drives, but it seems unlikely they would have wanted to risk that not being quite enough.
On top of that, since 2019 is 7 years ago, it is reasonable to expect they would have had a hardware refresh at some point, I would think 5 years would be more reasonable than 3 year for something like this. At that point, the nas would be more ($1300) but the drives would be less (8 * $300), so add either another $3700 (double if both are replaced, but maybe they figured the offsite backup could be stretched even longer than 5 years)
So, rather than $2500, it would probably have been more like $8500 + OSS or $13,300 without OSS. Still that would have meant a proper are archive and a proper backup and they wouldn't be in this mess, so I completely agree with you about what they should have done, but think you were underselling the price a good bit.
Full headline: PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct — public TV channel sues Iron Mountain data center, which hosts archival materials, to ensure preservation
The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.
This does make sense. I'm guessing the first request some rep just went sure its you're data why not, but in reality, it's OSS's data, and you paid OSS to store it. It's also likely encrypted in someway.
TLDR still, is especially when it's a reasonable amount of data, just have more than 1 backup.
Likewise if I store data with a cloud provider. Sure, Iron Mountain shouldn't just offer it up without proof, but if I give them notice of a legal case they should be sure to hold on to it, and if I win the case they need to hand it over to me. All I need to do is provide proof that I am the rightful owner of those goods.
Also your storage company case is actually it's own can of worms, see the Lego drama from a couple months ago about that. It seems obvious, but in the eyes of the law there's more paperwork than you'd think.
They weren't recording in 4K last century.
One hour of 1970's-grade NTSC video is around 300MB.
More to the point, it's not like they archived 70 continuous years of video. Many local PBS stations produce an hour or less of video each week. This being a St. Louis station, however, it's probably somewhat more.
I work in several labs where I have to give this advice, which some labs are smart enough to make an ersatz backup that we store somewhere (PI's house) as a cheap offsite. Even a partial recovery is great in the face of fire/flood/explosion, or the most common error: IT fucks it up and then says "oh well".
Hope someone has some copies and they can rebuild, what a nightmare.
>This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Look, I know that a certain percentage of Tom's Hardware articles are just sales pitches for a NAS, but this is asinine. Iron Mountain deals largely in tape storage. 50TB of tapes fits in one of their small archive boxes. If they're using the latest tech (LTO-9), it fits on 3 tapes.
This is exactly how you are supposed to store archival media for decades, not on a NAS. A NAS is for storing the files you need to actively access.
"Nine PBS" seems to be <https://www.ninepbs.org/>, A/K/A KETC, St. Louis, MO.
<https://en.wikipedia.org/wiki/KETC>
The upstream cited article which should be subbed in for this submission does clearly identify the station in the first line of its report: <https://current.org/2026/08/nine-pbs-sues-iron-mountain-over...>.
(I've submitted this to HN's mods via email. The original submission was to a very poor TomsHardware link.)
OT: The tendency of local broadcasters to fail to identify either on websites OR QUITE OFTEN ON BROADCASTS THEMSELVES even vaguely where they operate is ... a long-standing annoyance of mine. Sure, local audiences may know, but those passing through, or catching the signal at a distance, likely won't.
I've spent more than a few nights at higher elevations (campgrounds, highways) listening to clear channel stations reporting on weather and traffic "in the local area", sometimes for hours, without ever hearing what "the local area" is. That for signals travelling hundreds of km, and not infrequently multiple thousands of km.
(Don't get me started on entire countries where stations are identified simply by number, e.g., "Channel 4" in the UK.)
I worked at WCCO in Minneapolis briefly a few years ago. Right around that time, CBS corporate was attempting to standardize all their stations and associated websites. In their mind, that meant making everything "CBS News [Location]" and stripping just about everything else from the public marketing.
WCCO, KPIX in Los Angeles, and a few other stations battled back about this because the local name recognition of the call sign was stronger than CBS would be by itself.
So now, when you bop around the "Local News" section of the CBS Website, some cities have logos with the call letters, and some do not.
In short, when you see a station that that's difficult to identify, try finding out who owns it. Very often the ownership is to blame for one reason or another.
Here is the source article...
https://current.org/2026/08/nine-pbs-sues-iron-mountain-over...