The Black Liszt

Category: Twitter

Summary: Software in Government, Big Business and Big Tech

This is a summary with links to my posts on the many ways that large organizations including government, big business, big tech and the rest diligently apply modern software procedures as taught in academia and required by professional management; they consistently produce disastrous results in software quality, cost, security and everything else that matters.

There are of course issues that are common to all these large organizations, for example in cybersecurity.

https://blackliszt.com/2015/06/systemic-issues-behind-the-cyber-security-disasters-at-opm-citi-anthem-etc.html

Government

Government software disasters are government-as-usual, so much so that disasters that wreck lives barely make the news. For example, over 10 million people world-wide enter a government-run lottery for immigration slots that can lead to US citizenship. How hard can picking a bunch of random numbers be? Apparently too hard for the government software people, with the result of horrible consequences for the declared lottery winners whose immigration slots were invalidated.

https://blackliszt.com/2011/07/software-quality-horror-tales-electronic-diversity-visas.html

Consider the sets "Excellence" and "Government IT." There is a great deal of evidence that these are non-overlapping sets. I learned there are organizations promoting and celebrating digital government. They hold awards ceremonies. I tried to find out what the winner had done to deserve winning. Surprise, surprise, the link at the organization’s website explaining it all was broken. Pathetic.

https://blackliszt.com/2015/05/excellence-in-government-it.html

Even simple things like making Social Security statements available on-line appears to be beyond them — including of course lying about it.

https://blackliszt.com/2024/03/excellenece-in-government-it-the-social-security-administration-.html

The NSA (National Security Agency) has a budget of over $50 Billion and is touted as being the world’s best at cybersecurity. It turns out the only reason we know their super-top-secret budget is because their security was blatantly breached with massive internal data made public.

https://blackliszt.com/2014/05/bureaucracy-regulation-and-computer-security.html

Given that this army of highly-paid cyber geniuses can’t protect itself, it’s not surprising that its analysis of a high-visibility security breach may have sounded good to the public, but was in fact entirely fraudulent.

https://blackliszt.com/2017/01/russia-hacks-dnc-podesta-email-fake-news.html

What do you do with such a huge budget when you’re unable to do what you’re supposed to do even with your own secrets? You set up a massive program to teach students your excellent methods and hope to train over a million certified experts. I tracked the program from a local community college to the NSA’s own description of its program – which was both broken and insecure!

https://blackliszt.com/2017/06/government-cyber-security-tops-the-oxymoron-list.html

Unfortunately, this isn’t just about keeping information safe. Government ineptitude kills people. Instead of taking a quick, simple approach to preventing train crashes:

https://blackliszt.com/2015/05/an-app-to-prevent-train-crashes-like-amtrak-philadelphia.html

The government presses on with its super-expensive solution using obsolete technology, which leads to yet more preventable crashes and deaths.

https://blackliszt.com/2016/10/scandal-hoboken-train-crash.html

It’s not just big governments. The little government of several islands in the Caribbean managed to create a multi-front disaster using best practices to foist a digital currency system on its innocent citizens.

https://blackliszt.com/2022/03/dcash-government-cryptocurrency-shows-why-fedcoin-would-be-a-disaster.html

https://blackliszt.com/2022/03/what-is-behind-the-dcash-central-bank-digital-currency-disaster.html

The US government continues to pursue a national digital currency of the kind that has already proved to be a disaster in the Caribbean. They do so ignoring the fact that the US Dollar is already largely digital, with extensive software support structures that are in place and working well..

https://blackliszt.com/2020/12/we-dont-need-fedcoin-we-already-have-a-national-digital-currency.html

Important things like voting systems are some combination of broken and insecure. I took the trouble to define a simple combination of tech and non-tech to build a modern, secure voting system that was auditable, with operations visible to every voter while keeping what they voted for secret. Will any government institution pay attention, much less implement it? We all know the answer.

https://blackliszt.com/2025/03/voter-id-and-paper-ballots-dont-prevent-cheating.html

Big Business

Executives in big business want to succeed and advance, but this can only happen by avoiding risk. The best way to avoid risk is to do what “everyone else” is doing, what the experts say is best. That’s where industry advisory groups come in.

https://blackliszt.com/2017/05/the-value-of-computer-industry-advisory-groups.html

Giant advisory firms counsel their customers on how to make the best decisions. Getting your customers to like you is high on the list. Carefully crafted words are of supreme importance to such large organizations. Actions that match? Not so much.

https://blackliszt.com/2016/07/gartner-group-big-company-customer-service.html

A giant health insurance company “lost” the personal information of "tens of millions" of its members sometime in 2014; they're not sure how many, whose records were "lost," or when it happened. The details are an astounding illustration of big-corporate IT incompetence.

https://blackliszt.com/2015/02/the-anthem-of-cyber-insecurity.html

I soon found out that my information had indeed been stolen. The company’s response to the theft was right in line with their letting it happen.

https://blackliszt.com/2015/02/my-anthem-account-was-hacked.html

What company doesn't want to be part of the digital revolution and have an app? If you're a major health insurance company, why wouldn't you replace old-fashioned insurance cards with something always up-to-date that comes on an app? Here’s what ensued when one of the industry giants tried.

https://blackliszt.com/2021/02/why-cant-big-companies-build-or-even-buy-sofware-that-works.html

I've covered many big organization face-plants. The awfulness encompasses a broad range of consumer-dissing inconvenience, Here’s a case of some software that "works" but puts customer inconvenience front and center.

https://blackliszt.com/2021/03/why-cant-big-companies-build-software-that-works.html

Here’s a case of a giant company software issue that is low on the “it matters” scale, and high on the “a smart high school student could have done it better” scale. It’s the kind of issue that leads one to wonder whether we’d all be better off if they refused to hire any more people with college degrees for any job, and in particular, management.

https://blackliszt.com/2021/05/anthem-needs-my-feedback-reveals-deep-problems.html

Big Tech

Whether the software is a cool social app, an academic website or a real business, there is a common theme: the software is poorly designed and, even worse, it just breaks. You might think the cool internet apps like Facebook and Twitter are an exception, but they’re not.

https://blackliszt.com/2012/01/internet-software-quality-horror-shows.html

How can you innovate? Did the leaders of the current big tech companies benefit from training in innovation? Once they became large, have the big guys like Google demonstrated excellence in innovation? Uhh, sorry, the facts indicate otherwise.

https://blackliszt.com/2016/05/organizing-for-successful-innovation-recent-history.html

The widely-accepted logic is: Facebook is wildly successful; FB is built on software; therefore, FB software must be excellent. I should hire people from FB to help me build excellent software! The history and facts support neither the logic nor the conclusion.

https://blackliszt.com/2014/12/fb.html

I looked at FB’s mobile app when it had over 700 million people using it. Over 20 million people had written reviews, more than 6 million of which were 3 stars or less. A random sample of those reviews yielded juicy results.

https://blackliszt.com/2014/11/facebooks-software-quality.html

The difference between image and reality at FB is astounding. Here is an interview and a recent book that should lead any ambitious young company to avoid hiring people from there.

https://blackliszt.com/2017/03/software-giants-image-and-reality-facebook.html

Large organizations have trouble building software. This has been true since the dawn of software history, and shows no signs of changing. The decades-long, rolling disaster of Microsoft Windows is a great example of this.

https://blackliszt.com/2015/08/large-organization-software-fails-the-case-of-microsoft-windows.html

Microsoft illustrated multiple issues relating to digital ownership in a case I dug into. Among other things they attempted to require use of their own pathetic browser.

https://blackliszt.com/2014/05/giant-software-bureaucracies.html

There are big problems with software quality. The social apps in particular have decided it's embarrassing. But instead of actually, you know, fixing the problems, they seem to have decided to mask the problems! Twitter is a great example of this disease.

https://blackliszt.com/2013/05/twitter-software-quality-stinks.html

I did detailed studies on Twitter and found that they do indeed produce provably bad search results.

https://blackliszt.com/2013/05/twitter-software-quality-an-oxymoron.html

People write and talk about what's "trending on Twitter" as though the trend meant something. It doesn't. It's based on deeply flawed Twitter search software that gives random, widely varying results.

https://blackliszt.com/2013/05/the-bogus-basis-of-trending-on-twitter.html

Twitter fired boatloads of software engineers in 2022 leading some to predict that software disaster will ensue. But then, most people don’t know much about software and don’t realize what a disaster Twitter software has been for years.

https://blackliszt.com/2022/11/twitter-can-improve-software-quality-by-losing-most-of-its-engineers.html

Then there is Apple, the high-prestige computer company making expensive devices. In 2016, terrorists killed a bunch of people in California. Law enforcement and the FBI worked hard to find out what happened and who else might have been involved. This required looking in the government-issued iPhones used by the killers. What happened? Apple did its best to protect the criminals. Here are the highlights.

https://blackliszt.com/2016/03/the-apple-fbi-fiasco.html

And here are the details:

https://blackliszt.com/2016/03/apple-can-help-fight-crime-while-maintaining-privacy.html

https://blackliszt.com/2016/02/apples-cancer-prevention-strategy.html

https://blackliszt.com/2016/02/apples-approach-to-privacy-terrorists-and-criminals.html

I reviewed a book about government security on Amazon. The author was impressive and had loads of experience. Many of the reviews were positive, with a few pointing to obvious bias. I wrote a review that pointed to the positive aspects, but also mentioned some of the bias. The review disappeared. I interacted with Amazon, and was told that suppressing the review was a mistake. It appeared again. Then it disappeared. I tried to write a review and was told I've been banned!

https://blackliszt.com/2023/03/early-evidence-of-criticism-suppression-by-intelligence-agencies-.html

Yelp isn’t as big as the industry giants, but it’s pretty big. A random plunge into their system demonstrates the same kind of slick surface with rotten underpinnings as their larger brethren.

https://blackliszt.com/2021/05/yelp-big-tech-incompetent-corrupt.html

Conclusion

There is a better way! The winning methods aren’t even new – they’re proven in practice by small groups that need to win. See:

https://blackliszt.com/2023/07/summary-software-innovation.html

https://blackliszt.com/2023/07/summary-wartime-software-to-win-the-war.html

November 17, 2023
Twitter can improve software quality by losing most of its engineers

Twitter has fired boatloads of software engineers and more are jumping ship. Most people predict that losing all those engineers will lead to software disaster. But then, most people don’t know much about software and don’t realize what a disaster Twitter software has been for years. With some intelligent leadership, Twitter's software could dramatically improve while dispensing with 90% of their engineers.

Twitter Customer Losses

First, let's note that the situation has users really worried. Customers are bailing the ship:

Here's some up-to-date analysis:

While some say the massive customer losses are due to the new CEO, others are sure that the loss of so many excellent Twitter software engineers will seal the deal.

Twitter Software Quality

Those cool internet software companies – their top people wear hoodies or whatever they feel like. They must be fabulous programmers, right?

Nope. Facebook, for example, produces amazingly bad software. See this and the included links for details on how bad it is – and how they hide as much as possible from their users.

Facebook, Twitter and the rest aren’t better than “normal” software companies, except in how rich they made their founders. Internet software is a horror show, as I document here.

I dove into Twitter in particular years ago to see just how bad it was. I found and documented inexcusable failures, which they went to great lengths to disguise.

I tested doing searches for tweets with Twitter’s own search engine, and documented random patterns of tweets being dropped out or included from searches just days apart. See this and this.

What this means, among other things, that the basis of “trending on Twitter” is based on bad Twitter data – the kind produced by those wonderful, oh-so-cool Twitter engineers.

Twitter hides the awfulness of its terrible software from you

Note that the errors I documented were NOT associated with an error message. They were cases where the right answer (for example search results) could be determined and compared to what Twitter provided. That’s what I did, and found that Twitter blithely would state “here’s your answer,” and then provide an answer that was demonstrably false. It would have been more honest had they given an error message instead. But no — that would have been honest; Twitter engineers, while terrible at building software that actually works, have become masters of the simpler, nefarious job of masking bad software.

Even with such errors, Twitter would still admit failure from time to time, showing this image:

Since then Twitter has gotten better at hiding their errors. Following the pattern of bad search results, they give you your feed of tweets and you merrily scroll down, reading, retweeting, liking, and whatever. They have learned to almost never admit failure. The fail whale shown above is history. How do you know if you’re getting all the tweets from the people you follow, and in the right order? You don’t! In fact, there is loads of anecdotal evidence that part of how their engineers spend their time is figuring out how to manipulate your feed for a variety of reasons — including masking the results of their inability to do it the right way — instead of just giving you a common-sense, complete ordered feed.

Fail whale? I haven’t seen it in years. They can have dozens of servers crash and burn and no one will know the difference. What Twitter does is just pull together your search results or feed based on whatever servers are still limping along with whatever the pathetic Twitter software will give them, and give you whatever crap it has. How will you ever know it’s incomplete, missing important things, etc.? Unless you’re an obsessive crazy person like me and run tests, you’ll never know.

So what do all those thousands of software engineers do all day long except attend meetings while relaxing on comfy bean bags and digesting all the free food the company provides? Obviously not much in the way of useful programming.

The alternative

Small teams of smart, motivated programmers typically out-perform teams of hundreds and thousands of people with titles of “software” employed in big bureaucracies – including in “cool” companies. The concept is simple: most programming organizations are like people who build bridges in times of peace. They take thousands of people years to build. When you build bridges in war, like the bridge over the Rhine in World War 2, you have to build it in a DAY, while under enemy fire. And it has to work. Small, under-resourced groups of programmers operate in wartime mode, getting things DONE while groups of thousands continue plodding away at requirements planning. See this. For more, see this.

I see this all the time in my work of evaluating small, innovative start-ups. The small companies don’t have enough time or money to do things the “right” way. They have to get things done. Fast! The ones that do succeed.

A great example from another field is the author of the first important dictionary of the English language, Samuel Johnson. He produced excellent work, arguably 200 times more productively than the big committee trying to do the same for the French language. See this for the story.

What Twitter does isn’t hard! And what it does hasn’t fundamentally changed for at least a decade! Those thousands of engineers couldn’t make it work well nine years ago when I ran tests, and all they’ve managed since then is do a more polished job of hiding the errors and bad answers.

Whether Elon Musk or someone like him is in charge, the best thing for Twitter and its customers would be for most of the programmers to be shown the door, for the remainder to get with the program of making things actually work and work well, or join their former colleagues as ex-employees. If they really get it going, some programmers who are actually good and, get ready, WANT to write code, GOOD code, may seriously consider joining a re-invented Twitter engineering group.

It might actually happen, since Mr. Musk is violating the most sacrosanct rule of management: getting involved with the people on the front lines of actually doing the work! There's even evidence that he seems proud of his disgusting behavior.

Who does he think he is??!!

November 19, 2022
The Bogus Basis of “Trending on Twitter”

People write and talk about what's "trending on Twitter" as though the trend meant something. It doesn't. It's based on deeply flawed Twitter search software that gives random, widely varying results. I know the weatherman is often wrong, but what if he said it was going to be sunny in the 70's tommorow and as often as not there was a blizzard — would you keep listening? It's the same with Twitter, only worse.

Trending on Twitter is everywhere

It's amazing how widespread this useless stuff is. New York Times editors are in on the game.

It's even now got a prominent place on Wall Street!

You can not only follow what's trending in general, but you can narrow it down to different locations.

When a Twitter account is hacked, bad things happen.

And sure enough, the markets react.

We seem to care not only about what the Boston bomber says on Twitter:

But we also pay attention to the useless Twitter trends about it:

We've really got to stop this. It's not as though we've got reliable data here. It's just not. Twitter has been a technical joke for years, and there are no signs of improvement.

Trending on Twitter is meaningless garbage

I don't have the access to perform a universal test. But I did perform a test, and anyone else can reproduce my results. I did searches over a couple week period for the same term and saved the results. Sometimes the results were correct, but most of the time, items that were there before disappeared, only to pop up again on a subsequent search. Sometimes just a couple things were missing, and sometimes the gap was massive. Here is the evidence.

Then I took the search that appeared to have the most gaps, and performed the identical search about a week later. As I documented, one search had just 5 items and the other had 32, when they should have been identical. About 85% of the search results had been dropped by Twitter!

"Trending on Twitter" is based on comparing results of a search performed on one day to the same search performed on other days. If the number of results goes up or down, you've got a trend. Or so you think. But what if the results are really as bad as I have documented? I found that "blackliszt" went up or down by a factor of 6, like 600%! Wow!

Conclusion

Twitter software has always been bad. Management has learned to disguise the awfulness by suppressing the appearance of the "fail whale," but they clearly haven't actually, you know, made the software better. Anyone who takes its results as actually meaning something is depending on bogus data.

May 16, 2013
Twitter Software Quality: An Oxymoron

Twitter software quality Stinks. As I've demonstrated. On revisting and updating the facts, I've decided that "Twitter Software Quality" should be promoted to the status of oxymoron, joining the august company of terms such as "southern efficiency," "northern hospitality," and "government worker."

A Brief History of Random Awfulness

I took samples of searches for "blackliszt" on these dates: Apr 18, 19, 20, 22, 24, 25, May 1, 8. A total of 8 samples.

All searches were done as "All" to tell Twitter I wanted, you know, all the results, not just the ones Twitter felt like disclosing at the moment.

I only grabbed the first page from each search. I've shown the results in another post. Of the 8 searches, the one on May 1 is the most extreme. Here's a copy of the May 1 search for "blackliszt:"

You can see there are 5 tweets in the list of results, from Apr 11 to Oct 13. I decided to try to find out how many tweets there actually were between Oct 13 2012 and May 1, 2013, the date of the search pictured above.

I did this research on May 8. At least on May 8, Twitter was willing to admit that there were a total of 32 tweets in the same date range, although one of them (Feb 27) appears twice. Here they are:

A Twitter search for "blackliszt" performed on May 1 resulted in a list of 5 tweets going back to Oct 13. The same search for "blackliszt" performed on May 8 (above) resulted in a list of 32 tweets that should have been returned by the May 1 search. Maybe there are more! Given that one is double-counted (Feb 27), who the &*() knows?? What I do know is that on May 1, Twitter decided to discard 27 out of 32 potential results of a search. Roughly 85% of the tweets were gone!

Summary

I already knew that Twitter software quality was bad. It turns out that it's worse than I ever imagined. It's "Twitter-quality"-is-an-oxymoron bad.

You know all those "trending on Twitter" items you're seeing now that seem so modern and cool? They all assume that getting more or fewer results from a search means something. We now know that the results can easily go up by a factor of six, or drop by the same factor, just because of Twitter "quality." It's obvious that "trending on twitter" deserves to be the punchline of a joke, not something that anyone pays attention to.

May 15, 2013
Twitter Software Quality Stinks

There are big problems with software quality. The problems range from social apps to corporate to academia, include "mission critical" software, and everywhere in between. The social apps in particular have decided it's embarassing. But instead of actually, you know, fixing the problems, they seem to have decided to mask the problems! Twitter is a great example of this disease.

Two ways of Responding when you don't know the Answer

Suppose you're a kid and someone is demanding answers from you. Either you know the answer or you don't. If you know the answer, it's simple: just give the answer!

Q: When did Columbus sail the ocean blue?

A: 1492

If you don't know the answer, there are two ways to respond: the right way and the wrong way. The right way to respond is simple: Just say you don't know!

Q: When did Columbus sail the ocean blue?

A: I don't know.

The wrong way to respond is a little more complicated. You have to guess at an answer, state it as though you knew the answer, and hope no one cares or that the person asking doesn't know either so you can get away with it.

Q: When did Columbus sail the ocean blue?

A: 1542.

When the question you're asked has several answers, you can be wrong in a different way. For example:

Q: Name the ships in Columbus' voyage to the New World.

A: The Nina and the Santa Maria.

Q: Is that all of them?

A: Yes.

Twitter's Response when it doesn't know the answer

I never thought it would happen, but now I have fond feelings for Twitter's Fail Whale, which I haven't seen recently. You would think that the fail whale not showing up as often would be a good sign. It's not. It's a sign that Twitter has decided that it's better to lie than to admit it doesn't know the answer to the question you're asking. Instead of forthrightly saying "I don't know," Twitter now brazenly gives the wrong answer. Even worse, it gives a different wrong answer from one day to the next!

Twitter's Bogus Search results

Here are some screen shots of the results of the identical query, for "blackliszt," over a couple of weeks. I always selected "All results" to remove any excuse that Twitter was selecting the "top" results to help me out.

Let's go through time. Here's the result from the first day, Apr 18:

I tried again the following day, Apr 19, and was quite surprised with the result: the Rebelmouse tweet simply disappeared, pulling an older one into the results!

On Apr 20 I added a tweet and did the search again. My new tweet was there, and RebelMouse came back!

On Apr 22 I tried yet again and got another brand-new variation: this time Cadencia's tweet disappeared!

The results were unchanged on Apr 24 and 25. I gave Twitter a couple days to lose some data, and had my patience rewarded when I searched again on May 1. The first result was Rebelmouse; the most recent posts, my post on ballet, Cadencia and Rob Majteles, were all gone! Here's May 1:

Finally, look at this simple list of my tweets taken Apr 23, not a search:

Note that I had tweets on Apr 10 and Mar 25, both of which included "blackliszt," neither of which appeared in any of the search results!!

Sadly, I can't even claim that the folks at Twitter have it out for me. It's just the way things work there … uhhh, I mean, the way things don't work there…

Conclusion

Social Media software quality stinks. It's worth every cent you paid for it. Oh, you didn't pay anything for it, you say? Well, that's my point. When a program like Twitter gives you an interface, lets you do a search, gives you a result that's even worse than my "Nina and Santa Maria" answer, brazenly implies that it's the right answer and everyone just ignores the issue, something is wrong.

Q to Twitter exec: Why does your software randomly leave out results from searches? Why should anyone look at "trending tweets" or anything else when the data is randomly bogus?

A: I've never been asked that question before. The answer is simple: I do it because I can, because I don't care, because no one else seems to and because I'm worth a great deal of money and you're not. Next question please.

Thanks to MaryAnn Bekkedahl for inspiring me to write this up.

May 2, 2013
Internet Software Quality Horror Shows

Whether the software is a cool social app, an academic website or a real business, there is a common theme: the software is poorly designed and, even worse, it just breaks. As in falls flat on the floor, waves its arms in surrender, and just gives up. And not just once — it keeps breaking! As I've said before, we really need a revolution in software quality.

Cool Social Apps

Hey, social is where it's at — how can billions of Facebook users be wrong? Before long, there will be as many FB users as MacDonald's has sold hamburgers (billions and billions)!

Those guys must be great programmers, huh? I mean, just look at their office:

Here's one of them giving a talk at a conference:

See how cool he is? He's just wearing a t-shirt, not even "business casual."

The other social media are just as cool. Here's a "chill" Twitter office:

And Jack Dorsey, the Twitter CEO — quite the opposite of a buttoned-down financial guy, huh?

It's perfectly obvious that these guys must write just the coolest, most awesome code ever. There's no way people this cool could make elementary programming mistakes, particularly when their application is so very dead-simple, and hardly ever changes — they could spend practically all their time being cool and polish up some already-faultless code a couple times a day, and still be OK.

Except this little detail, which I scraped from my own screen, and which I personally have seen countless times:

Yes, the famous Twitter fail whale. I think Twitter got tired of all the publicity their "cute" failure message was getting them, so they reverted to something more discrete; here's an example:

FB is just as bad, of course, and they've always tried to minimize the message when they screw up:

Apparently, FB is incapable of keeping even the most recent day's worth of updates on-line — you should try going back in history and seeing how far you get. Oh, you thought the stuff you wrote was your data, did you?

Naturally, it makes sense to consider that you get what you pay for; all these cool social apps are, after all, free. You can hardly complain when something you didn't pay for is flakey — return it and demand a full refund!

So let's turn to a more promising field. Everybody's supposed to go to college and learn stuff, so…

Academia

Let's see if the universities do any better. I was just on a local college's website, and it was even worse than Twitter — Twitter's code knew it was screwing up and put up the fail whale. In this case, any number of links I hit encountered badly broken code:

Oh, alright. The colleges are perpetually underfunded, and putting up a website that works isn't a high priority compared to … all the other things they spend money on. I guess.

Probably a real business does it better, right?

Profit-making Big Company

Even more so, an essential public service, like the cable company! Those guys have the money, the funding, the experience and the mandate to do it right. Let's pick the case where their motivation is the highest: collecting money.

Oops.

Just a few days ago, I was on my local cable provider's site trying to access my account. Here's what I got:

Not just once, but repeatedly, for hours!

But maybe it's just TW that's got problems — surely all the other big companies do things great, with their huge staffs and policies and procedures and all, right?

Sadly, no. Here's just one personal example from Verizon:

Summary

There's no getting around it. Software is just bad. Everywhere. We can speculate about why this is the case, but let's agree on the facts: it's bad, and not getting better.

January 22, 2012