My first job was building out the first mega-datacenters. 2005-2007, I was a datacenter assistant monkey working from Google working somewhere in the Chicago suburbs, swapping out hard drives and ram and writing shell scripts, as myself and my friends unknowingly laid down the prototype for the kinds of datacenters we all see today.

And so it is with some significant expertise that I say:

Fuck datacenters. Datacenters are an anti-pattern.

@cwebber that's an interesting point of view. I mean, of course the current datacenter craze is complete madness, but it seems you consider an anti-pattern the concept of datacenter itself. Why is it so? What do you suggest as an alternate solution to the problems data centers try to solve?

Working in that environment, seeing as Google rolled out the idea of "cloud computing" meaning "you have no involvement or agency in your computing because we do it for you" radicalized me for much of the work of my career.

It was one thing to run a datacenter to index the world's public web information. I understood that, it made sense.

But watching as Google and Apple co-developed the idea that computers, which I cared about, got abstracted into toys and jewelry that had all your key computing done in a way you had no agency over... where I saw firsthand the kinds of churn of resources necessary to keep these things going, it made me want to fight for a different computing future.

@farfalk Datacenters are concentrations of power. Anytime a datacenter is involved, it's a sign of power centralization. The rise of datacenters corresponds with the death of p2p and other visions of a more decentralized internet.

@cwebber do you have a writeup expanding on “datacenters are an anti-pattern” because this is 100% how I feel. I’m not fighting over AI. I’m still fighting over the cloud. Society has badly fumbled the fact that everyone has incredible computing power in their pocket. We’ve already been wasting electricity on idle servers and inefficient high level code. I’ve been - and am - part of the problem, moving functionality to the cloud for business reasons I hate.

@thomasjwebb I've been meaning to write a blogpost for a long time. Sounds like it's time to write it!

@cwebber
The core problem is not AI, cloud, or interfaces.

The real problem is ambiguous and centrally controlled information.

Without deterministic foundations, civilization will keep scaling dependency, manipulation, and chaos.

CTMinfo is the first real attempt to solve this at the root.

https://zenodo.org/records/17593591

@cwebber @thomasjwebb I remember "datacenter" starting as "colocation hoster" - you rentet rackspace or several racks with redundant power supply, internet link, packed in some pizzaboxes and a router, and there you go. Physical safety was better than the rack with dev servers in the basement, so what else could you ask for?

@cwebber At the moment we ended up with a phone that could do most of what we wanted, but had no agency over, at the backend there was the datacenter explosion, which we had no agency over. Now you can't even buy a laptop anymore because the datacenter explosion caused scarcity of its components. Check mate.

@cwebber I agree with your premise. However, realistically, convincing businesses to return to on-prem is challenging. It's not just about management – maintaining, securing, and procuring on-prem hardware is a significant expense and effort compared to provisioning cloud resources.

MSPs could help, but even with their assistance, the ongoing costs and complexity often favor cloud-native solutions.

How do you think we could convince small business owners to move from Shopify for example?

@cwebber @farfalk I think it more corresponds to the death of personal computing as it was? People don't have desktops anymore and barely have laptops other than for work? Which is a problem for p2p? Seems like most people's decentralized/federated nodes for things are hosted in data centers? All question marks because just speculating.

@cwebber @farfalk Well you see Datacankers that way and I entirely agree. When will those presently apathetic about Datacancers realize their browsing and posting options have shrunken in quantity and quality? Some may be built for bitcoin mining but most have an objective I would call brain mining.

@cwebber @farfalk like, I can appreciate some of the advantages of having them. Like you could get more computer per watt, maybe. I think valuable research is done with supercomputers and modern, more modular approaches to big data. But we could do way more with way fewer datacenters if these weren't used as a way to paywall functionality at the server side. The move to the cloud almost makes me miss when my problem was Cubase requiring a USB dongle.

@johns @cwebber @farfalk it's telling that hardware for user-controlled computing is disappearing. Memory and storage are disappearing from the market and it feels *intentional*. https://investors.micron.com/news-releases/news-release-details/micron-announces-exit-crucial-consumer-business

@johns @farfalk Yes, all things I worried about, as they were happening, and all of which have enormously clear and worrying impacts on user agency

@cwebber The main motivations for using data centers are security, reliability, flexibility, and efficiency. A data center can manage power consumption much better than your typical on premises installation. Infrastructure can be shared by many users, which allows dynamic scaling without having to over provision on premises. It also allows users to share expert reliability and cybersecurity engineers.

@cwebber @farfalk what are your thoughts on smaller businesses running their own datacenters? Having done rack and stack in a few myself, those at least seem less evil to me.

@cwebber
huh

interesting.

about that same time;

had 3 interviews with the goog

as a 'linux beige box wrangler'

at a big campus dc in northern va

my linux clue-kit was deep & shallow at the same time

they passed me over

I was bummed, bigly

then it all changed

@cwebber I did this in Oregon. The town looks nothing like its old self.

@cwebber yes, this!

I said that when I heard of that cloud thing for the first time. They laughed at me. People still tell me that it’s the new and now only way of doing things and required for work 🤮

@Reinald @thomasjwebb @cwebber colocation of your own stuff is ok

@thomasjwebb @cwebber @farfalk there is always a "but sometimes" so maybe we can take it as a given that for any strident statement in short form chat there isn't all the nuance about exceptions.

The overall direction seems right to me though. We've got a not insignificant ipv6 deployment for residential use, where is the Cobalt Qube of personal computing? There is no good *technical* reason I shouldn't be able to host my personal email on my own domain, at home, on my own computer, along with a website or whatever. Big monopolistic platforms, which require huge datacenters and complex tech stacks, are an antipattern.

@raven667 @cwebber @farfalk sure I just don’t want to be seen as someone who hasn’t considered the obvious counterpoints. I have the “always include depth-first nuance” kind of autism and ocd. But yeah I think if we design protocols right, maybe people won’t even have to self-host in many cases. It could be some truly p2p stuff than can run on the client.

@thomasjwebb @raven667 @cwebber @farfalk exactly! where we're going we don't need websites, we must abandon the concepts that require single-point-of-failure "hosting" because that is not viable in a local-first world.

@cwebber datacenter bad, colocation good

@cwebber @farfalk Taking the long view, it’s the return of the data center coupled with public access and it is part of a swing between centralization and distribution of computers that dates to the 1960s.

The essential tension driving this is economy of scale versus localization of control. Before the PC it was departmental computing versus corporate control of resources.

I think data centers can serve a useful purpose but the pendulum has swung too far.

@raven667 @valpackett @farfalk @cwebber yeah I don’t want to force normal people to be sysadmins. Maybe some sysadmining can be like a public service so everyone gets free static web hosting. But no server should be needed to send messages, comment, etc.

@cwebber @farfalk
And I'd add the obvious. Not exactly the death itself, but a systematic effort directed against, personal computing.

@mirabilos @thomasjwebb @cwebber I love it. Put some powerful blade system like a cisco UCS, some storage system and a small tape library, and you are ready for allmost everything. For some more complex network setups a dedicated load balancer might be useful. But you can build so crazy powerful stuff in two full racks...

@Reinald @thomasjwebb @cwebber eh, my needs are much lower. At this point in time, I have:

• a computer from the last millennium as main server at home, not public-facing, but fixed IP
• a dedicated server from the early 2000s which still works, at hoster A currently in France
• two rather tiny VMs with annoyingly slow storage at hoster B in Germany
• a slightly beefier VM with SSD-backed storage at nominally hoster A but probably really the american company who bought them but still in a data centre in France
• a VM on my gf’s dedicated server at mass hoster C in Germany, which is my backup server (currently restic), deliberately IPv6-only
• domains at hoster D in Germany, but the nameservers are among those systems listed above
• an old computer from my former employer, with two new HDDs, on the consumer DSL at home, as copy of the backup storage and also occasional build server, etc.
• so far one “pihole” (really AdGuard Home atm), will need to set up a second
• laptops
• SPARCstations

This gives me geographic split into at least two operating and two backup sites. I don’t have the need for that much oomph at any given single site but like the redundancy (I also (had to… forgot a LUKS password…) tested restore, half an hour and the VM was back up). Recovery for a lost site would take longer due to changing IPs, but still possible with low enough data loss, but definitely still annoying of course.

I also need to move some things around and set up OpenVPN (likely classical server-clients star layout, even if mesh would be nicer here) so they have fixed IPv6 “among each other”, currently backing up those behind consumer or roadwarrior connections is done with SSH forwarding, which is annoying, and LUKS unlock at boot also needs IPv6 (wrote the initramfs glue for that myself, as the existing v4 support fails for the v6-only box and one of the others due to networking irregularities). Ideally, this can also give me v6 on the laptop at sites lacking it. For this, I need to set up a CA first… *sigh…* (existing ones were just made unusable) I need more spoons.

@cwebber get off the internet ,you're on a data center somewhere! get off get off get off get off the rat poison is about to strike it's a data center there's data center rats no no no nononononononononononononononononononononononono you need to hide right now now now now now now now now now now now

@cwebber

I worked in a data center from 2002-2013, moving from an infosec role to sysadmin halfway through (so more and more down among the racks as time went on).

This particular datacenter was built in the late 70s, under the rule of "old I.T./Data Processing," where guys would come in at 8, grab a cup of coffee, grab a stack of greenbar printouts, and spend a couple hours going over reports before really starting the day. The laid-back atmosphere lasted until the late-mid 2000s, when the company got sold, we all got outsourced to a TLA, and it turned into (relatively speaking) a living hell. (Probably paradise compared to today's I.T., though).

The actual computer rooms were all in the basement, so all you saw above-ground was offices/cubicles.

I had dreams about deep, dark foreboding computer basements for years after leaving I.T. 😂

I cannot imagine what modern datacenters must be like.

If things progress the way they are, I can see datacenter terrorism (e.g. Molotov Cocktails and such) being a real thing. Just imagine people seeing their children not able to access potable water while Giga Mechanical Turk* down the streets sucks down the municipal water supply while belching pollutants and screeching noise.

I wanted the future to be Star Trek, not Cyberpunk, dammit.

I'm incensed that the modern computing needs of the average person could be handily served by a $100 junk PC, but now we have to build planet-destroying industrial data centers to serve up stochastic smalltalk. Freaking heck.

* Referring to a historical fake-automaton, not an epithet against Turkic people, of whom I (partially) am one.

cc: @mirabilos

@rl_dane @cwebber

I cannot imagine what modern datacenters must be like.

Cold, and loud enough you’d need ear protection by law, at least in the actual server rooms. Artificial lighting only, of course, even if it’s above-ground. And security to fear, even if [sometimes duck-tried (Thread)].

That’s modern as in before 2019.

@cwebber Data centers will be around till someone writes software making AI computation in a distributed fashion.

Please do!
@cwebber @thomasjwebb

@cwebber

there is this character/mask of turin, part of the commedy of art, name is gianduia (which is also as gianduiotto a typical candy) ...

he is famous for hiding money in somebody elses pockets ...

i think of it every time i hear the word cloud ...
🤣

https://en.wikipedia.org/wiki/Gianduja_(commedia_dell'arte)

@thomasjwebb Not to pre-empt Christine (I don’t know what she has in mind anyway), but you may find the essay The eternal mainframe by Rudolf Winestock interesting if you haven’t read it already: https://www.winestockwebdesign.com/Essays/Eternal_Mainframe.html (archive: https://archive.today/mz7Zk). (It’s from 2013, so admittedly a bit old.)

What do you suggest as an alternate solution to the problems data centers try to solve?

@mirabilos @Reinald @thomasjwebb @cwebber apparently not. Most of the people in these threads seem never to have heard of a data centre before last year. I’ve been hosting stuff in them since 1994 at least. At that point they were mostly for telecoms gear.

Though I’ve almost never done the cloud compute thing.

@Colman @mirabilos @Reinald @thomasjwebb Yeah, we are far from those days for the most part. I'm very familiar with those. But most datacenter deployments aren't like that anymore.

@cwebber @Colman @mirabilos @thomasjwebb nobody needed Datacenters. Internet was located at universities, including mail, web, usenet, ... Personal homepages started with ~/johndoe...
Colocation providers started, AFAIR, with private hosters offering shared webservers and domains for everybody. With the first "internet bubble" the demand grew, and facilities were build either near network hubs or near energy hubs. Many grew organic. With a wild mixture of infrastructure.

@cwebber @Colman @thomasjwebb @Reinald if you’re situated at a university, then you’re lucky, I guess. But from my PoV, UserDir at a university server is data centre hosting except your URLs break when you change departments or leave, i.e. even worse.

@Reinald @cwebber @Colman @thomasjwebb yeah. My initial response was not detailled enough, I was thinking of deployments where you put the public-facing stuff into one, or even more than one to spread the risk, but do all the development, archiving, etc. in an in-house LAN.

@cwebber Indeed! I have several issues with datacenters.

First, using large amounts of water for cooling is NOT required. Evaporative cooling is NOT required. The fact that every so-called journalist writes about datacenter projects without calling out the megacorps who simply don’t want to pay for non-evaporative cooling just makes things worse.

I have servers colocated in three different datacenters in three different US states. None use evaporative cooling. For years I had a quarter rack in a datacenter in Phoenix, Arizona, also without evaporative cooling. In Phoenix!

Sure, people who like to make excuses for megacorps will say that high density compute like what AI companies want to build can’t be cooled without evaporation, but that’s simply not true. It can be done, but it costs much more, and AI companies don’t care about 10+ year ROI.

The articles that don’t point out that all of these projects are bribing - er, negotiating cheaper power and water from municipal sources that are driving residential prices higher are doing everyone a disservice.

All of this is one of many failings of capitalism. When a single corporate entity can demand so much of a service’s output that it can adversely shape the costs of basic, required services, there’s something wrong. The fact that there’s no governmental protections from this kind of abuse is highly problematic.

It sure would be nice if journalists pointed out that the results of megadatacenters aren’t inevitable, but chosen based on how much hurt can be pushed on to everyone else.

@AnachronistJohn @cwebber the data center water use scandal is just fake outrage. There are other industries that are using orders of magnitude more water

@slyecho @cwebber Do you have some examples and numbers to share?

Also, it’s not “outrage” to not want your local municipal water prices to go up, for your water quality to go down, for your aquifiers to run dry and/or for your area to run out of water, or anyone else’s. Saying that someone else is doing something worse so that doing something bad isn’t as bad isn’t really a persuasive argument.

@AnachronistJohn @cwebber here is one article that has a lot of examples and references: https://blog.andymasley.com/p/the-ai-water-issue-is-fake

@slyecho @AnachronistJohn This guy has done a lot of really poor writing on AI and energy resource usage, making himself sound like an expert when he's really just asking an AI chatbot to confirm his own suspicions. He does it about every topic, and he enthusiastically admits in this article that his primary method of research is just asking an AI chatbot for him https://andymasley.com/writing/data-centers-heat-exhaust-is-not/

@cwebber The appliance-ization of computing really depresses and infuriates me.

Growing up with early personal computers in the 80s, the magic and the promise was you could use these machines to do *anything*, including things nobody else thought or cared to do!

A big part of what I fought to build in Android was a mobile platform that'd also deliver on the "you can run anything you want" promise of the machines I grew up with... sadly over the past decade that has been seriously eroded.

@wouter @cwebber @farfalk

Though, why could I no run my family server just in my attic? Think about this (the technical reasoning) and you see what's wrong with your picture ...

@wouter @glitzersachen @cwebber @farfalk also, dust and climate control are much easier in bulk.

The machine I have in colocation still runs fine after ten years, but everything I have at home has needed several new fans at inopportune times, and runs way too hot in summer.

@cwebber that is... a bit too broad. AI Data centers? Fuck no, you are right 1000%.

"Classic data centers", as in "big managed server rooms with IaaS services where people can host their websites without being sysadmins"? That's simply a good thing. Easier to maintain than each computer take individually, gives people jobs, more "eco-friendly" (as much a computer can be eco-friendly!), more secure if the managers are not idiots.

Your opinion is strong, and as such it's interesting. But nuance!

@GyrosGeier @wouter @glitzersachen @cwebber yeah, my thoughts exactly. There is a threshold were too local and too global are both quite silly. I think small data centers with IaaS approach are a bit like public transport (and i would LOVE if governments were to provide small data centers as a public service!)