The Black Liszt

Category: Internet

Summary: Computer Data Centers and Networking

This is a summary with links to my posts on computer data centers and networks. The two subjects are intimately related because the whole point of networks is to connect computers with each other.

Like everything else in computing, fashions have a strong impact. A bit over a decade ago, the world of computing started yammering about “the cloud” and “virtualization.” These things were the hot subjects all the cool kids talked about. If you weren’t driving towards moving to the cloud, you were obsolete. The reality of course was much different.

https://blackliszt.com/2011/12/the-name-game-of-moving-to-the-cloud.html

The Cloud is just another virtue-signaling fashion word like Big Data.

https://blackliszt.com/2012/04/im-tired-of-hearing-about-big-data.html

The fact that the underlying reality hardly changes at all is way beyond the technical knowledge of the vast majority of the people who talk about it.

https://blackliszt.com/2012/05/the-cloud-and-virtualization.html

When something like “the cloud” heats up, all the vendors related to it rush to promote their products as ideally suited to the new thing.

https://blackliszt.com/2013/04/storage-vendors-in-the-cloud.html

The managers of large in-house data centers rarely know much about what they’re buying and what the alternatives could be. As a result, they often spend way more than they need to on equipment, something which competing cloud vendors are less likely to do.

https://blackliszt.com/2014/08/data-center-managers-spend-too-much-on-equipment.html

Like anything where people are involved, there are politics and fights over who should perform a given function. One of the classics is making replications of the data in DBMS’s for safety purposes. The data center people want to use their mechanism and the DBMS people want to use theirs. There is a clear right answer, but it only wins sometimes.

https://blackliszt.com/2014/03/replication-good-idea-storage-replication-nah.html

One of the key things people who write applications want is for them to “scale,” i.e., be able to handle any load without slowing down. The way some systems applications are built, based on decades-old designs, can get in the way. But there are solutions.

https://blackliszt.com/2014/02/obstacles-to-scaling-centralization.html

People also want their applications to continue to be available in the event of data center failure. For people working in data centers, there are perverse incentives at work.

https://blackliszt.com/2010/04/is-your-site-working-do-you-really-care.html

To be fair to data center managers, applications originally written decades ago expect to be able to run in modern data centers. It’s tough!

https://blackliszt.com/2010/01/paleolithic-mainframes-discovered-alive-in-data-center.html

Way back in 2015, it was clear that hardware had evolved to make dramatic improvements possible in software.

https://blackliszt.com/2015/08/the-data-center-of-the-future.html

With software people ignorantly focused on network-connected services as the way to build scalable applications, they assure that the incredible hardware power will be minimally utilized.

https://blackliszt.com/microservices/

Network neutrality was a hot topic and still comes up. The idea is that everyone should be charged the same for internet access and services. When you dig into the subject with actual knowledge of the technology, you see that the whole furor by virtue-signaling ignorance.

https://blackliszt.com/2014/11/net-neutrality-it-aint-broke-dont-fix-it.html

One of the arguments made about net neutrality and legal privacy provisions was unusually divorced from reality.

https://blackliszt.com/2017/04/on-the-internet-youre-naked-and-for-sale.html

Years later, a weak form of net neutrality was repealed. There were demonstrations with US Senators making predictions about the disaster that would ensure. Here’s an analysis of the non-disaster a year later.

https://blackliszt.com/2019/06/the-aftermath-of-the-net-neutrality-disaster.html

August 31, 2023
The Aftermath of the Net Neutrality Disaster

After many years of fighting against entrenched corporate interests, the FCC finally extended its regulatory authority to the internet, instituting the so-called "net neutrality" rule in 2015. Finally, the internet would remain free and open.

Then a new administration came in, with new leadership at the FCC. There was a move to repeal the common-sense principle of "net neutrality!" Why would anyone think that letting giant corporations pick and choose what we can see on the internet was a good idea, making things they didn't like too expensive or simply blocked — crazy! It's kind of like repealing the laws against murder, hoping that people would just not do it!

Some leading Democrats made it clear what would happen if net neutrality were repealed:

The senate was being careful here — what about ISP's blocking access to sites they don't want you to see? Senate leaders were outspoken about the danger:

People were upset. There were demonstrations at more than 700 Verizon stores nationwide:

Protestors even spread out through the new FCC chairman's neighborhood, demonstrated in front of his house, took pictures of the inside of his house through his windows, and made it clear that this man, Ajit Pai, was leading the movement to destroy the internet as we know it.

After a vote near the end of 2017 to repeal this important protection, the repeal finally took effect in June 2018, about a year ago. That's when the disaster of corporate control of the internet, access limitations, slower speeds and no more innovation started. Articles were written in the NY Times and elsewhere spelling out what would happen: "repealing these onerous rules “would be the final pillow in (the internet’s) face” (The New York Times), would cause “erosion of the biggest free-speech platform the world has ever known” (ACLU), and would be “end of the internet as we know it” (CNN)."

It's now about a year into the disaster. I started looking into just how bad it's been and was struck by how hard it is to find the news stories about all the sites that have been blocked, all the students relegated to slow lanes and prevented from watching their favorite video streams, and all the price hikes. Have the giant corporations really been so scarily effective at suppressing the consequences of their self-serving actions?

I dug in and found this:

Last year, average internet download speeds shot up almost 36%, and upload speeds climbed 22%, according to internet speed-test company Ookla in its latest U.S. broadband report.

There are more users than ever. More videos to watch. More content to consume. More commerce being conducted. No sites are being blocked. No one is complaining that their service is being throttled. And more people are gaining access to broadband.

During President Trump’s first year in office, in fact, the number of people without access to a broadband connection dropped by 18%.

Meanwhile, the 5G era approaches, which will increase speeds — for mobile and fixed broadband — exponentially while injecting more competition among providers. Elon Musk’s SpaceX has started launching mini-satellites for his Starlink initiative, which will provide broadband internet to subscribers wherever they are on the planet.

Oops. Maybe the disaster isn't a disaster at all. Maybe it's even a good thing, hard as that may be to understand. Here's what I wrote about it when repealing net neutrality was being discussed, and here's the long post I wrote on the subject when the campaign to institute net neutrality was underway.

p.s. That earlier post on net neutrality touched a few nerves. A few programmers at companies I worked with got all hot and bothered, and refused to interact with someone so completely unethical that they could harbor such awful thoughts. Well, now we know how it all turned out, and what the repeal-net-neutrality disaster amounted to…

June 20, 2019
On the Internet, you’re Naked and For Sale
Wild claims are being made by powerful people about how certain regulatory changes being discussed in the US Congress will remove your privacy on the internet, enable evil corporations to sell your personal information to the highest bidder, and control where on the internet you can go. See here and here.These claims grossly misrepresent the facts.

The situation on the internet is like this: Imagine that when you walk around New York City, all the business owners know everything about you. Imagine that when a business owner learns something new about you, he immediately puts the information up for sale. You are naked! No privacy! Now you get on a train to Boston, and when you get off, you find exactly the same happens there — AND, everyone in both cities knows what you did in all the other cities you've visited.Yes, there are things you can do about that. But most people don't do those things. They've decided that it's OK.

Until October 27, 2016, you were just as naked and for sale when on the train as you were while walking the streets. That's when the FCC issued new rules, which haven't gone into effect. Because the new rules have not gone into effect, rescinding them means that nothing has changed: you were naked and for sale from the start of the internet, whether in a city or on a train, and that won't change.

What's under discussion in Congress and at the FCC is this:
- You should continue to be as naked and for sale while you're on the train as when you're in a city — no more and no less. That's the "privacy" they're talking about, which is unchanged from the start of the internet.
- Business should be free to offer different ways at different prices for consumers to travel from NYC to Boston, for example, cheap but slow bus, faster but more expensive train and fastest but most expensive airplane. So-called "net neutrality" rules amount to making everyone take the bus and ending consumer choice.
Ignorance and lies surround us. Business as usual.

You have no Privacy on the Internet

On the internet, you are naked. You always were. What has evolved are the methods for buying and selling pictures of your naked self, and histories of where you've been and what you've done.

I've visited a wonderful but obscure website for a credit card anti-fraud company I'm involved with, Feedzai. Today, I clicked on an item in my Facebook newsfeed about introverts and extroverts. When I got to the page, here's what I saw:

It's a great ad, and I never mind seeing it. How the heck did Feedzai manage to place an ad on this from-another-universe site? Simple: when I got to the site, it put my information up for auction, Feedzai looked at the information about me and placed a high enough bid to win the auction and place the ad. It's called real-time bidding. It takes place when you visit most websites, and happens in a fraction of a second.

I purposely picked an itty-bitty advertiser placing an ad on an obscure site to show how ubiquitous the information marketplace is. But the big boys do it too, and it's even creepier. I have a cat. I recently bought food for the cat. Here's what I just saw on the blog of Scott Adams:

Yes, that's exactly the product I bought on Amazon

There are private rooms on the internet, sites where the communication is encrypted. Google is one of them. The information about you learned by the encrypted site is entrusted to the site — which may or may not be able to keep it secure. Google, for example, immediately puts the information up for sale, which is why you see ads related to the search you just did. The retail company from which you just bought something may not manage to keep your information secure, even if they try.

You are naked on the internet, followed everywhere, and you're always for sale. Period.

What would change if the privacy rules went into effect?

If you listen to the people screaming about it, lots would change. Sorry, it wouldn't.
- The ISP's don't have much data to sell — they're free to sell it now and have been free for years, but mostly they don't. ISP's are a tiny fraction of the internet ad market.
- The ISP's get less of your data as time goes on. Far from having your complete browsing history and knowing your SSN and the size and color of the most recent clothing you just bought, as famous people have declared, they have little access to it, and the fraction gets smaller over time.
  - They never had access to your SSN. The proposed rule would have had no practical impact.
  - Most commercially important data passes through the ISP encrypted. It's literally not visible to them. Look up at the URL of this blog. It's not encrypted. On the other hand, the fact that you're viewing it is commercially useless.
  - Go to Google and look at the URL. You'll see https:// at the start, which means it's encrypted. The ISP literally can't see what you're searching for! And when you go to Amazon or another commercial site, it's the same. The ISP doesn't know what you've bought. But Amazon does, and stalks you with ads about it, as illustrated above.
This whole thing is a tempest in a teapot.

What about Net Neutrality, which the FCC proposes ending?

This is a complex subject, with heated rhetoric all around. I go into the issues in depth here. Briefly, ending net neutrality is a good thing for consumers.

Net neutrality says, for example, that all the seats at Yankee Stadium have to be sold at the same price and offer the same views. We all know that the views from the grandstand and the upper deck are quite a bit different from the boxes behind the dugouts. Most people understand and accept that the prices vary accordingly. Under net neutrality, everything would have to be the same and charged the same price. Switching metaphors, why would any company offer air service if they could only charge the same rates as the bus? Why would any company offer better service without any incentive to do so?

Naturally, there are arguments against the summary I've presented here. To go into depth, read this.

Conclusion

You're naked and for sale on the internet. It's the way it is. It stops almost no one from using the internet, and many people find it convenient, seeing things that are more relevant to them than not. The frantic noise about the awfulness of rescinding a change that never took effect is nothing but posturing to the ignorant.
April 4, 2017
Internet Driver’s Licenses Needed for Users

We give kids sex education. We give them driver education, and require a driver test and license before driving. But we let any fool onto the internet to wreak whatever havoc they can on themselves and others without a second thought. It's time for a change!

Education for Meaningful Use

Education on the basics of how the internet and associated technologies work and how to control, respond to and interpret what you see is totally neglected. There are no significant efforts that I know of to make people educated consumers of this important, ubiquitous service that is so widely used. But there is a more important issue…

Education for Safety

By far the most important subject for internet education is safety. Maintaining internet safety has some similarities to general safety, but is different in important ways.

Internet "driving" safety

The most important aspect of safety while driving is avoiding driving while impaired in any way, and paying sharp attention to the road and other vehicles at all times. Driving while impaired by drugs or alcohol or while engaged in texting or talking
are recognized factors.

So imagine how hazardous internet driving must be when people don't even know how to read the road signs (the URL's) and can't tell that they've wandered onto a road constructed by criminals specifically for the purpose of enabling them to steal your car, drive it to your bank and take out a big withdrawal! But that's exactly what it is! Here's an example of a more brazen attack (image from a good guy, Yoo Security), demanding that you send the money yourself:
Unfortunately, there are criminals out there who have grown far beyond simple smash-and-grab operations. These sophisticated criminals with a long-term view trick you to "drive" onto their criminally-constructed "road" for the sole purpose of making your car an instrument for stealing from other people or organizations. They can make your computer into a zombie to participate in botnets. It can serve that purpose for minutes or years without your awareness. Is the problem big? You betcha. There are more computers that have been hi-jacked into botnets (maybe yours!) than most people are aware of:

Sometimes, of course, the criminals are stupid, greedy or malicious — I guess those are the drop-outs from the "criminals should be good citizens" certification program. So your hi-jacked device could slow to a crawl, do weird things, look over your shoulder as you type until they get the information needed to drain your bank account or max out your credit card, or even (just because it's fun!) wipe out your machine while leaving some cute "it was me! Have a nice life!" Message on your screen.

Internet E-mail fraud

How often do you get a letter purporting to be from your bank asking you to send them a letter containing your account number just so they can verify that everything's OK? If you got one, do you think you'd respond as requested? Apparently you're not alone — criminals are the supreme capitalists, and abandon efforts that are unprofitable before long.

But how about letters on the internet, i.e., e-mail? Along with everyone I know, I get an amazing number of criminal solicitations, ranging from the laughable (at least to me) to the amazingly credible every day. Data-driven capitalists that they are, the only explanation for the persistence of these efforts is that more than enough of them work to cover the costs and trouble of running the schemes, certainly more than getting a legal job. I've seen fewer solicitations from Nigeria lately, but the slack has been taken up by Libya.

Here's one of the new breed from Libya:

Here is a somewhat more plausible one from a place that really could be your bank:

Conclusion

Uneducated internet users cause billions of dollars of harm to themselves and others every year. You think this would result in outcry by those users and people who know them for education. You might think this might merit a bit of attention from the institutions who so assiduously and expensively educate, authorize, license and otherwise keep us on the straight and narrow. When I'm in Central Park in New York, there are rangers watching my every move; they set me straight when I ride my bike where I'm not supposed to, or walk in one of the ever-changing restricted areas. The conclusion is obvious: every move I make in the Park is more worthy of watchful restriction by people in uniforms than the millions of actions on the internet that seem, at least to me, far more destructive. I must be missing something.

April 7, 2015
Net Neutrality: It Ain’t Broke, Don’t Fix it
There is lots of talk about "net neutrality" now, after years of passionate advocacy by partisans. I have a simple response to the issue, driven by my simple-minded engineer's mentality. There's no problem here, so don't you dare try to "fix" it!

Net Neutrality

The way "net neutrality" is normally described, it's shocking that it's not already the rule of the land. Opposing net neutrality is described as being like a racist, something which is obviously unacceptable in a civilized society. (Just to be clear: discriminating on the basis of race, sex or any other human variation is totally unacceptable to me.) It amounts to evil internet service providers slowing down or discarding network packets from sources of which they don't approve, and speeding up access to approved sources. This could be done for commercial gain, to push some brand of politics, or any number of nefarious motives.

The argument in favor of net neutrality is normally made in terms of simple fairness: preventing giant ISP's from preventing or impeding access to internet resources customers want. The feared consequences will range from high prices and/or poor service for companies whose services threaten the ISP's such as Netflix, to barring consumers from accessing politically or commercially threatening web sites. Anyone who opposes this view of enforcing simple fairness is accused of being paid off by corporate interests or morally corrupt. Or simply stupid, for not understanding how the internet works.

I claim that I am none of the above: not bought off, not morally corrupt, not stupid, and furthermore relatively knowledgeable of internet internals.

It would take a long paper or short book to lay out all the facts and arguments. I don't have the time or the patience. But here are some headlines.

"Net Neutrality" is all about Innovation-Killing Regulation

Net Neutrality may be a moral crusade about fairness and equality for many of those who promote it, but the proposed solution is that the same inept crew that raises costs, protects the powerful and stifles innovation in so much of our lives will now be able to wield their magic-killing wands on the internet. It's not about "fairness" — it's about control by a bunch of ignorant, remote bureaucrats.

Here's a good summary, see the article for more:

The Internet boomed precisely because it wasn’t regulated. In 1999 the FCC published a paper titled “The FCC and the Unregulation of the Internet.” The study contrasted the dramatic growth of the open Internet with that of the sluggish industries subject to Title II’s more than 1,000 regulations. Sen. Ted Cruz got it right last week when he tweeted that Title II would be ObamaCare for the Internet.

Amazing as it seems, under these regulations federal bureaucrats in the 1970s decided whether AT&T could move beyond standard black telephones to offer Princess phones in pink, blue and white. A Title II Internet would give regulators similar authority to approve, prioritize and set “just and reasonable” prices for broadband, the lifeblood of the Internet.

These guys don't know how to build technology. They are incapable of keeping it secure. Their regulations are certain to be obsolete before they're written, and counter-productive.

You're Afraid Greedy ISP's Might Limit Internet Access?

Really? Well, just wait until the government gets involved. Once a bunch of bureaucrats operating essentially in secret gets going, it's hard to stop them.

It's well-known that South Korea has the world's fastest internet connections. But the internet there is anything but free and open. Government-driven censorship is severe. Here are some of the basics:

Internet censorship in South Korea has been categorized as "pervasive" in the conflict/security area, and also present in the social area. Categories of censorship include "subversive communication", "materials harmful to minors", and "pornography and nudity". Internet censorship has been expressed by the shutting down of anti-conscription and gay and lesbian websites,^[1]^[2] the arrest of activists from North Korea-sympathetic parties, and the deletion of blog posts by writers who criticize the South Korean president.^{[citation needed]} Censors particularly target anonymous forums; South Koreans who publish content on the Internet are required by law to verify their identity with their citizen identity number. The most common form of censorship at present involves ordering internet service providers to block the IP address of disfavored websites. A government agency announced the planning of new systems of pre-censorship of controversial material in the future.

ISP problems are Caused by Regulation. The Cure is More Regulation??

The ISP's, like Comcast, Cablevision, Time Warner, Verizon and the rest, provide the "last mile" of access to the internet. They're the guys who bill you for use. All the rest of the internet just magically happens, supported by a variety of means, mostly advertising.

The last mile is where the problem is. These guys are mostly descendents of the phone and cable companies. They exist and operate at the pleasure of various federal, state and local regulators. Just like the power companies, they have centers from which their wires weave out to sub-stations, down major streets, branching to local streets and eventually to houses and buildings. More agencies than you can shake a stick at stand in their way at every step, demanding this and that. In exchange, they get a monopoly or close to one.

Are these nimble, creative, innovative guys? Duhhhh. How can they be? They go to all the trouble to put wiring in, and they try to keep it in service as long as they can, milking every advantage out of it they can. Given all that, I'm surprised things work as well as they do.

Bottom line: the ISP's are already regulated. That's their problem. Let's not make it worse by adding in federal regulation and spreading it to more of the system. Since when has federal regulation made technology better?

There are Fast Lanes and Slow Lanes on the Internet. And the Problem Is???

Advocates of net neutrality are big on talking about how grubby issues of crass money will cause unfavored sites and consumers to be relegated to the slow lanes of the internet, while all the fat cats will cruise on the fast lanes.

Exactly how is this different from, like, everything else in life?

There is nothing like "NY Yankees neutrality," for example. Here's the price and the view from the expensive seats:

And here's the price and the view from the bleachers:

How unfair! How unequal! Someone should do something about Yankees neutrality!

By comparison, all the "seats" on the internet offered by ISP's are just fabulous. Access rates are thousands of times faster than in the past, and at good prices. You can get even faster speeds if you're willing to pay — and that's OK.

The Greatest Current Threat to the Internet is Apps and Mobile

"Net Neutrality" is mostly a "what-if" threat, based on the minimal things ISP's have done, and the horrible things someone imagines they could do. Apps, driven by mobile, are a huge, here-and-now threat, growing by the day. As users shift their attention to mobile, they are shifting away from the open, highly competitive web to the walled gardens of the mobile world, which is exactly what monopolistic giants like Apple, Google and Facebook want.

Here's a good summary, see the article for much more:

It isn’t that today’s kings of the app world want to quash innovation, per se. It is that in the transition to a world in which services are delivered through apps, rather than the Web, we are graduating to a system that makes innovation, serendipity and experimentation that much harder for those who build things that rely on the Internet. And today, that is pretty much everyone.

The Internet is Wildly Complex and Rapidly Evolving

People who complain about net neutrality typically have no idea how the internet works and how it's evolved over time. There's a lot going on; it's not just a set of pipes that get bigger and faster over time.

This is the part that's tough for me to limit what I say. While there are people who have spent more time inside the internet and its predecessors than I have, I was involved early, in the ARPA-net in 1970 and 1971 at a time when it had fewer than ten nodes, and periodically since then to the present. Here's the ARPA net in 1977:

A good chunk of the fun stuff, both the power and the problems of the internet, comes from the fact that the "internet" is a network of networks, an "inter-network" that connects many networks together, sort of like the interstate highway system connects the states, though much less uniformly than that. Here's an early version of the network of networks:

If this were the interstate highway system, some things to note would be:
- ISP's control the local roads and entrance ramps to the big roads.
- There are different ways to drive cross-country.
- If you care a lot about drive time, you get to know the best routes.
- If speed is really important to you, you take the toll roads to avoid the choke points. This is the origin of Internap, for example; its big early customer was Amazon.
- If you've got lots of stuff to deliver to many customers in many cities, you pre-deliver it to warehouses near the customers, so that when they order, delivery is fast. We call it a CDN, content delivery network.
- Special sub-networks are constantly being developed to solve problems, and the people who use them pay for their use. Business as usual.
The fact that the internet is an evolving web of variously connected networks is key to its vitality and astounding growth. Let's stand back and enjoy its continued unimpeded, unregulated growth.

Worried about Comcast and Netflix? You Shouldn't Be

Net Neutrality adocates like to create fear with all the things big scary ISP's could do that would be just awful — therefore we have to regulate them before they do those things, as in the Philip Dick story The Minority Report. They also love to recount the charges Netflix has made against Comcast as evidence of actual wrong-doing. In other words, they like to take the side of the monopolist of content (Netflix) against the local monopoly of access (Comcast). Once you dig all the way to the bottom, you realize that Netflix wanted to be able to dump content onto Comcast's network amounting to more than a quarter of its total traffic and demand that Comcast deliver it with uninterrupted regularity — for free, leaving Netflix to keep all the money it charged its customers. In the end, they cut a deal similar to typical CDN deals (see above).

When you buy HBO, you expect that the cable company and HBO somehow split what you give them — why should you care what they work out? But then when you buy Netflix, net neutrality advocates demand that the cable company deliver it for free. Only on Planet Stupid is this anything like "fair."

Summary

A cardinal rule in engineering is "if it ain't broke, don't fix it." Enthusiastic young engineers break this rule all the time. Hard experience usually educates them. Applying that rule to the internet, we get: the internet is a big collection of moving parts and blobs, constantly evolving. It works remarkably well. Parts of it are crappier and slower than they could be, anywhere from 2X to well over 1,000X. Most people who operate various parts of the internet have no reason to care about the ultimate consumer experience and act accordingly. The slowest and crappiest parts of the internet stay in use way past their natural expiration dates, but eventually die off. The biggest entities and/or the most regulated and/or the most monopolistic tend to be the slowest and crappiest of all. They try to implement and/or enforce practices and technologies from many years ago, and do so poorly, at great expense to themselves and everyone involved. Sometimes they act in a nakedly self-interested or "principled" way and make things even worse. But all in all, the consumer experience on the internet has improved with remarkable speed and few glitches compared to almost anything else, and way better than if it had been regulated. So let's leave it alone, and worry about the true threats.
November 18, 2014