The Black Liszt

Category: Big Tech Companies

Summary: Software in Government, Big Business and Big Tech

This is a summary with links to my posts on the many ways that large organizations including government, big business, big tech and the rest diligently apply modern software procedures as taught in academia and required by professional management; they consistently produce disastrous results in software quality, cost, security and everything else that matters.

There are of course issues that are common to all these large organizations, for example in cybersecurity.

https://blackliszt.com/2015/06/systemic-issues-behind-the-cyber-security-disasters-at-opm-citi-anthem-etc.html

Government

Government software disasters are government-as-usual, so much so that disasters that wreck lives barely make the news. For example, over 10 million people world-wide enter a government-run lottery for immigration slots that can lead to US citizenship. How hard can picking a bunch of random numbers be? Apparently too hard for the government software people, with the result of horrible consequences for the declared lottery winners whose immigration slots were invalidated.

https://blackliszt.com/2011/07/software-quality-horror-tales-electronic-diversity-visas.html

Consider the sets "Excellence" and "Government IT." There is a great deal of evidence that these are non-overlapping sets. I learned there are organizations promoting and celebrating digital government. They hold awards ceremonies. I tried to find out what the winner had done to deserve winning. Surprise, surprise, the link at the organization’s website explaining it all was broken. Pathetic.

https://blackliszt.com/2015/05/excellence-in-government-it.html

Even simple things like making Social Security statements available on-line appears to be beyond them — including of course lying about it.

https://blackliszt.com/2024/03/excellenece-in-government-it-the-social-security-administration-.html

The NSA (National Security Agency) has a budget of over $50 Billion and is touted as being the world’s best at cybersecurity. It turns out the only reason we know their super-top-secret budget is because their security was blatantly breached with massive internal data made public.

https://blackliszt.com/2014/05/bureaucracy-regulation-and-computer-security.html

Given that this army of highly-paid cyber geniuses can’t protect itself, it’s not surprising that its analysis of a high-visibility security breach may have sounded good to the public, but was in fact entirely fraudulent.

https://blackliszt.com/2017/01/russia-hacks-dnc-podesta-email-fake-news.html

What do you do with such a huge budget when you’re unable to do what you’re supposed to do even with your own secrets? You set up a massive program to teach students your excellent methods and hope to train over a million certified experts. I tracked the program from a local community college to the NSA’s own description of its program – which was both broken and insecure!

https://blackliszt.com/2017/06/government-cyber-security-tops-the-oxymoron-list.html

Unfortunately, this isn’t just about keeping information safe. Government ineptitude kills people. Instead of taking a quick, simple approach to preventing train crashes:

https://blackliszt.com/2015/05/an-app-to-prevent-train-crashes-like-amtrak-philadelphia.html

The government presses on with its super-expensive solution using obsolete technology, which leads to yet more preventable crashes and deaths.

https://blackliszt.com/2016/10/scandal-hoboken-train-crash.html

It’s not just big governments. The little government of several islands in the Caribbean managed to create a multi-front disaster using best practices to foist a digital currency system on its innocent citizens.

https://blackliszt.com/2022/03/dcash-government-cryptocurrency-shows-why-fedcoin-would-be-a-disaster.html

https://blackliszt.com/2022/03/what-is-behind-the-dcash-central-bank-digital-currency-disaster.html

The US government continues to pursue a national digital currency of the kind that has already proved to be a disaster in the Caribbean. They do so ignoring the fact that the US Dollar is already largely digital, with extensive software support structures that are in place and working well..

https://blackliszt.com/2020/12/we-dont-need-fedcoin-we-already-have-a-national-digital-currency.html

Important things like voting systems are some combination of broken and insecure. I took the trouble to define a simple combination of tech and non-tech to build a modern, secure voting system that was auditable, with operations visible to every voter while keeping what they voted for secret. Will any government institution pay attention, much less implement it? We all know the answer.

https://blackliszt.com/2025/03/voter-id-and-paper-ballots-dont-prevent-cheating.html

Big Business

Executives in big business want to succeed and advance, but this can only happen by avoiding risk. The best way to avoid risk is to do what “everyone else” is doing, what the experts say is best. That’s where industry advisory groups come in.

https://blackliszt.com/2017/05/the-value-of-computer-industry-advisory-groups.html

Giant advisory firms counsel their customers on how to make the best decisions. Getting your customers to like you is high on the list. Carefully crafted words are of supreme importance to such large organizations. Actions that match? Not so much.

https://blackliszt.com/2016/07/gartner-group-big-company-customer-service.html

A giant health insurance company “lost” the personal information of "tens of millions" of its members sometime in 2014; they're not sure how many, whose records were "lost," or when it happened. The details are an astounding illustration of big-corporate IT incompetence.

https://blackliszt.com/2015/02/the-anthem-of-cyber-insecurity.html

I soon found out that my information had indeed been stolen. The company’s response to the theft was right in line with their letting it happen.

https://blackliszt.com/2015/02/my-anthem-account-was-hacked.html

What company doesn't want to be part of the digital revolution and have an app? If you're a major health insurance company, why wouldn't you replace old-fashioned insurance cards with something always up-to-date that comes on an app? Here’s what ensued when one of the industry giants tried.

https://blackliszt.com/2021/02/why-cant-big-companies-build-or-even-buy-sofware-that-works.html

I've covered many big organization face-plants. The awfulness encompasses a broad range of consumer-dissing inconvenience, Here’s a case of some software that "works" but puts customer inconvenience front and center.

https://blackliszt.com/2021/03/why-cant-big-companies-build-software-that-works.html

Here’s a case of a giant company software issue that is low on the “it matters” scale, and high on the “a smart high school student could have done it better” scale. It’s the kind of issue that leads one to wonder whether we’d all be better off if they refused to hire any more people with college degrees for any job, and in particular, management.

https://blackliszt.com/2021/05/anthem-needs-my-feedback-reveals-deep-problems.html

Big Tech

Whether the software is a cool social app, an academic website or a real business, there is a common theme: the software is poorly designed and, even worse, it just breaks. You might think the cool internet apps like Facebook and Twitter are an exception, but they’re not.

https://blackliszt.com/2012/01/internet-software-quality-horror-shows.html

How can you innovate? Did the leaders of the current big tech companies benefit from training in innovation? Once they became large, have the big guys like Google demonstrated excellence in innovation? Uhh, sorry, the facts indicate otherwise.

https://blackliszt.com/2016/05/organizing-for-successful-innovation-recent-history.html

The widely-accepted logic is: Facebook is wildly successful; FB is built on software; therefore, FB software must be excellent. I should hire people from FB to help me build excellent software! The history and facts support neither the logic nor the conclusion.

https://blackliszt.com/2014/12/fb.html

I looked at FB’s mobile app when it had over 700 million people using it. Over 20 million people had written reviews, more than 6 million of which were 3 stars or less. A random sample of those reviews yielded juicy results.

https://blackliszt.com/2014/11/facebooks-software-quality.html

The difference between image and reality at FB is astounding. Here is an interview and a recent book that should lead any ambitious young company to avoid hiring people from there.

https://blackliszt.com/2017/03/software-giants-image-and-reality-facebook.html

Large organizations have trouble building software. This has been true since the dawn of software history, and shows no signs of changing. The decades-long, rolling disaster of Microsoft Windows is a great example of this.

https://blackliszt.com/2015/08/large-organization-software-fails-the-case-of-microsoft-windows.html

Microsoft illustrated multiple issues relating to digital ownership in a case I dug into. Among other things they attempted to require use of their own pathetic browser.

https://blackliszt.com/2014/05/giant-software-bureaucracies.html

There are big problems with software quality. The social apps in particular have decided it's embarrassing. But instead of actually, you know, fixing the problems, they seem to have decided to mask the problems! Twitter is a great example of this disease.

https://blackliszt.com/2013/05/twitter-software-quality-stinks.html

I did detailed studies on Twitter and found that they do indeed produce provably bad search results.

https://blackliszt.com/2013/05/twitter-software-quality-an-oxymoron.html

People write and talk about what's "trending on Twitter" as though the trend meant something. It doesn't. It's based on deeply flawed Twitter search software that gives random, widely varying results.

https://blackliszt.com/2013/05/the-bogus-basis-of-trending-on-twitter.html

Twitter fired boatloads of software engineers in 2022 leading some to predict that software disaster will ensue. But then, most people don’t know much about software and don’t realize what a disaster Twitter software has been for years.

https://blackliszt.com/2022/11/twitter-can-improve-software-quality-by-losing-most-of-its-engineers.html

Then there is Apple, the high-prestige computer company making expensive devices. In 2016, terrorists killed a bunch of people in California. Law enforcement and the FBI worked hard to find out what happened and who else might have been involved. This required looking in the government-issued iPhones used by the killers. What happened? Apple did its best to protect the criminals. Here are the highlights.

https://blackliszt.com/2016/03/the-apple-fbi-fiasco.html

And here are the details:

https://blackliszt.com/2016/03/apple-can-help-fight-crime-while-maintaining-privacy.html

https://blackliszt.com/2016/02/apples-cancer-prevention-strategy.html

https://blackliszt.com/2016/02/apples-approach-to-privacy-terrorists-and-criminals.html

I reviewed a book about government security on Amazon. The author was impressive and had loads of experience. Many of the reviews were positive, with a few pointing to obvious bias. I wrote a review that pointed to the positive aspects, but also mentioned some of the bias. The review disappeared. I interacted with Amazon, and was told that suppressing the review was a mistake. It appeared again. Then it disappeared. I tried to write a review and was told I've been banned!

https://blackliszt.com/2023/03/early-evidence-of-criticism-suppression-by-intelligence-agencies-.html

Yelp isn’t as big as the industry giants, but it’s pretty big. A random plunge into their system demonstrates the same kind of slick surface with rotten underpinnings as their larger brethren.

https://blackliszt.com/2021/05/yelp-big-tech-incompetent-corrupt.html

Conclusion

There is a better way! The winning methods aren’t even new – they’re proven in practice by small groups that need to win. See:

https://blackliszt.com/2023/07/summary-software-innovation.html

https://blackliszt.com/2023/07/summary-wartime-software-to-win-the-war.html

November 17, 2023
Who is Censoring US Intelligence Agency Criticism on Amazon?

A critical review I wrote of a book on Amazon was censored/suppressed. Was it profane or otherwise deserving deletion? No. It simply gave one star and pointed out errors and omissions, while acknowledging that fans of US Intelligence Agencies would admire the book. The review appeared briefly, then the link to it went bad. I interacted with customer service, who said they would submit a trouble report. I wrote another review, which appeared briefly then vanished. I tried once more. The last review was up for months. I got a warning from Amazon warning me that an unspecified review I had written failed to meet community standards, and if I did it again I would be banned. The review stayed up for many more months. And then it disappeared!

Spies, Lies and Algorithms was published by the Princeton University Press. It was written by Amy Zegart, a prominent, well-connected Professor at Stanford. I'm writing up this little experience because I suspect it's not an isolated incident; It's a step towards a society run like Orwell's 1984. Here's the story.

US Intelligence Agencies and Cyber-security

The grossly misnamed US "Intelligence" Agencies continue on the path of proclaiming their wonderfulness while US cybersecurity continues its free fall into unmitigated disaster. I describe the details of the sad state of cybersecurity in both government and private sectors in these posts. For government cybersecurity, see this. Here is a detailed analysis of the DNC server hack declared by Intelligence agencies to be a Russian hack. They failed utterly to prove it, and ignored the clear evidence that the most prominent emails never resided on the DNC server!

It should come as no surprise that they are doing what all bureaucracies do, i.e., protect themselves and their reputations by hiding the facts and suppressing criticism. The author of the book clearly and explicitly takes the side of the Agencies on this issue and similar ones. While she has given courses to congressional staffers on cyber issues, she clearly does little but echo the propaganda of Agency Experts.

The Book

Amazon, always serving up books they think I might buy, showed me Spies, Lies and Algorithms: The History and Future of American Intelligence by Amy Zegart.

What a cleverly designed cover! I checked it out. The author seemed to be well connected and educated about the Agencies and recent events. Take a gander at her background:

Amy Zegart is the Morris Arnold and Nona Jean Cox Senior Fellow at the Hoover Institution and Professor of Political Science (by courtesy) at Stanford University. She is also a Senior Fellow at Stanford’s Freeman Spogli Institute for International Studies, Chair of Stanford’s Artificial Intelligence and International Security Steering Committee, and a contributing writer at The Atlantic. She specializes in U.S. intelligence, emerging technologies and national security, grand strategy, and global political risk management.

Zegart has been featured by the National Journal as one of the ten most influential experts in intelligence reform. She served as a commissioner on the 2020 CSIS Technology and Intelligence Task Force and has advised the National Security Commission on Artificial Intelligence. She served on the Clinton administration’s National Security Council staff and as a foreign policy adviser to the Bush 2000 presidential campaign. She has also testified before the Senate Select Committee on Intelligence and advised senior officials on intelligence, homeland security, and cybersecurity matters.

Skipping my usual practice of carefully scanning the reviews, I bought the book.

I then read the book. The author was nearly everything you could want: highly educated, knowledgeable, articulate, connected. There was just one little problem: she has completely bought in to the deep state, to the extent of denying that there is such a thing. She faithfully ignores what they ignore, reflects their lies and omissions and shows near-total ignorance of computers and the "algorithms" of her title. That's all! In fact, her book scared me, because it was such a compelling read to anyone who was ignorant of cybersecurity (i.e., practically everyone) or hadn't dug into the big lies.

The Reviews

I've written a number of reviews of all sorts of products since joining Amazon in early 1998. Most have been positive, all of them constructive. I thought this book needed a review from me; while my immersion in the Intelligence community was trivial compared to the author's, my knowledge of cybersecurity and related subjects clearly eclipsed hers.

I went to the review section and found a handful of authoritative-person reviews, all highly complementary, and about a dozen reviews by readers, all but one very positive.

The sole negative review, 1 star, was by a self-described retired Intelligence officer who titled his review "bias detracts too much." 57 people found it helpful. He didn't go into detail but said:

She tries to deny it but the Deep State is real and is our biggest challenge. … She tipped her hand with selective alarm and omission of key events rampant in the first chapter.

The next most popular review, granting 5 stars and titled "There is some bias: here's why it isn't important" mostly demonstrates the power of being in the bubble:

I am a politically conservative individual studying intelligence and the IC for a number of years. Amy’s summary is accurate, thorough, and well-written. …
I couldn’t find any factual inaccuracies that would render the book a decorative piece or kindling.

These reviews demonstrate the problems of the Intelligence community and its image. It isn't about bias so much as it is about ignorance and denial.

Writing a Review

I dove into writing a review. My goal was to praise the author for her evident virtues, but to give a few facts to illustrate the basis for my giving it 1 star. I typed it into the box Amazon provided without keeping a copy of what I gave — silly me. I posted it and waited. Three days later I got a notice that my review was posted.

Hooray! I clicked to see the full review. This is what I got:

Cute dog, but what's going on here?

I went back once a day for many days and clicked again. The review never showed up.

Next I got online with Amazon customer service in a chat. After a bit of effort, the person agreed that the error about the Zegart review was a problem. I asked him to submit the issue for resolution, and he said he would. I waited until he said it was submitted.

Writing a Review Again

After a number of days with no change, I decided to simply post another review, this time saving what I had written. My submission of the second try was confirmed:

I never received the confirming email from Amazon saying that my second try at a review had been posted. But I kept checking, and to my surprise, my second review was posted. The bad link to the original review (Feb 12) now led to the new review (Feb 19). Hooray, it's posted! Here's a screen shot of the first part of it:

There's nothing bad going on here — I'm just being paranoid! And the review has already gotten several votes for being Helpful.

The Second Review Disappears

I went to the page again, and noticed that the total number of reviews had gone down. I look for my review. I was thorough. It's not there. Not there.

I had a thought. I had saved the original confirming email I got from Amazon pointing to my original review (which never showed up), which mysteriously DID point to my second try at a review. I click, and here's the result:

At least they're different dogs…

The Second Review Magically Reappears

I went to the page again and found, to my great surprise, that the elusive review was back!!

Just as a test, I wrote a positive 5-star review of socks I had bought on Amazon.

The Warning

A day or two later after the reappearance of the review, this showed up in my inbox:

Notice the careful wording. Notice that the exact review was not referenced, and that no specific violation of the guidelines was identified. I carefully read the community guidelines. Nothing that I wrote could be construed as violating them. The only offense of the review was to point out the deep issues of the Intelligence Community and to give a couple specific examples that illustrate it — one of them the supposed Russian hacking of the DNC server.

The second review stays up for many months, and then…

I checked the review occasionally over the following months. It stayed up and received dozens of "helpful" votes. It was second or third on the list of "critical" reviews.

Then, more than 6 months later, I got this.

Nothing about which content I posted that violates the Guidelines. No links. Nothing about removing the awful content. I'm just supposed to know, I guess. Naturally, I suspected that the Zedaris book review was the issue, so I went to the book's page. The book is doing well — it's now an editor's pick in nonfiction. I went down to the reviews. The "Bias detracts too much" 1-star review is still up and went from 57 up votes to 102. Sure enough, mine is gone.

Boy, those misinformation police at Amazon are really on the job, giving proper notice and telling you what law you violated. And prompt! And careful to only take down evidence-based reviews that might change some minds. Or something.

The last chapter

I wrote all of the above. Before posting it, I checked once more and found to my surprise that the review was up again! It wasn't on the top reviews list, even though it had over 50 "helpful" votes, but it was there. Oh well, much ado about nothing. The review is visible, my post is irrelevant.

Many months later, I had an impulse to check again. The review is gone again!! I carefully checked, looking at all the 1-star reviews, which takes some clicking just to get to them. The "bias detracts too much" review is still there, still with exactly 102 up votes, unchanged from at least 6 months ago. I bet that somehow the review isn't shown to many people. The handful of other critical reviews each had just a couple votes at most.

My review — gone again!

Update on Dec 2023:

I tried to write a review of a book. I got this:

All my reviews have been deleted!

I've reached out to them again. Something's going on!

Conclusion

Somewhere operating deep within Amazon there is an operation to suppress criticism of the CIA/FBI/NSA/whatever. I guess it could be Amazon employees who feel the same way, without official blessing. Or maybe, as we now know was the case with Twitter, there was an explicit back-channel connection to one of the 3LA's (3-letter agencies). After being allowed to be viewed for many months, my review garnered dozens of "helpful" votes. During that time I thought that the "image protecting" operation was walking a fine line between total suppression and letting low-attention items like my review alone to avoid attracting notice. But then it was visible again, just hard to find. And now, roughly a year after it was originally posted, it's gone.

FWIW and to be explicit about it, my review was NOT politically motivated. It was driven by facts and competence, consistent with these posts. Specifically see this. The job of the agencies is to enforce the law and protect us all from criminal and foreign attacks, a job for which, sadly, they have a track record of failure. And the job of prestigious professors who teach people about cybersecurity is to objectively assess the quality of the work, not ignore failures and incompetence.

March 7, 2023
Yelp is Big Tech Incompetent and Corrupt

Yelp is a Big Tech company that isn't listed when people talk about "big tech," but it's nonetheless all about tech and it's unarguably big, with well over 100 million unique visitors per month to their site and a valuation in the $ billions.

How good is their software? Are they a company from which non-tech corporate America should poach talent to upgrade their in-house tech? I recently had occasion to spend a little time with Yelp, and found that they are indeed an exemplar of the Big Tech tradition of tech bumbling as I have documented.

Harvard Business School famously teaches according to the case study method. Students are presented the case and challenged with figuring out what to do. Yelp would make an excellent case study. Any of the bright, aggressive b-school students who declared that the "solution" to the case was to fire all the executives and replace the entire product and tech team with talented high school grads, would get at least a B. Anyone who further suggested avoiding hiring anyone with a Computer Science degree would get an A.

Yelp and Local Business

Yelp is an advertising company that features listings of local businesses and user-generated reviews of those businesses.I recently visited my local dentist and received excellent care on the way to getting a crown put on one of my teeth. She knows I'm a computer guy and I've helped her out in the past, so she brought up her concern about a couple of malicious false reviews that were posted to her Yelp page. I'd said I'd check it out. I did. The results were illuminating. I sometimes complain about bumbling big corporate companies and their painful but fixable software problems. Yelp provides a great example of how second-tier Big Tech companies are no better at building software than the super-famous first tier ones are.

Yelp and a Local Dentist Business

I checked out Yelp's page for my dentist, Doctor Vu. It had a couple long negative reviews. So I posted a review of my experience, along with a selfie of me at her office.

I went back after a couple days to check out the page in more detail.

At the top of the page:

I see that she has 5 reviews with an average score of what looks like 3.5.

I scrolled down and found the photo section with the photo I had posted:

I took the selfie in her office holding the coaster I had given her last year that I couldn't resist buying at my favorite store in Cape May. She's had it in her waiting room ever since; I hear it often elicits amusement. It just seemed right to take the picture at the end of a visit to get a crown. 🙂

After scrolling through ads and other information I got to the reviews. Mine was first:

I scrolled through the rest, including the two 1-star slams. Something didn't seem right. I went back and carefully counted the reviews. I double-checked. There were 6 reviews! The header at the top said 5! Then I made note of the star ratings. The two super-baddies were 1 star and the other 4 were 5 stars each. That's 22 points divided by 6 reviews = 3.67. Looking carefully at the points at the top, the dividing line is clearly at the top of the star, so Yelp thinks it's 3.5.

But then Yelp said there were only 5 reviews in spite of displaying 6. Is it 22/5? Nope, that's 4.4. A complete mystery. Maybe I'll figure it out later.

At the end I found this:

What's this about? In faint type 16 other reviews? I check it out. At the top of the page that shows up there's a video I can watch and this long, serious-sounding thing about how Yelp cares and they're smart and they're doing the best for everyone:

Impressive! Finally I scroll down to check out these malicious reviews and here's the first thing I see:

Wow. Billions of data points! That software must be good, right? So I scroll down to see these scurrilous reviews. Imagine my surprise when I see this:

Wow. My review. Again!

At least the mystery of the rating average is solved. Even though my review was shown among the legit reviews — first, no less — it appears that their ludicrously bad software had marked it as not recommended and so some piece of software decided there were just 5 reviews. So 3 5-star plus 2 1-star is 17 stars, divided by 5 is 3.4. Close enough to the 3.5 average rating they displayed.

Even better — I received a congratulatory email from Yelp after posting the review with the picture, saying how great it was and I should post more pictures along with my excellent reviews. My picture is indeed featured. And my review is both first on the list of legit reviews AND first on the list of those not recommended. Score! This is what you get by employing thousands of Silicon Valley's best super-programmers…

Those billions of data points must have somehow confused the Yelpers. And their software.

Finally I read through the rest of the reviews that weren't recommended. There was nothing fake about them.

I visited Dr. Vu again to get actually crowned — the first visit was just preparation. She told me she had contacted Yelp about the situation. First she learned that the Yelp algorithms were state-of-the-art great, and in any case there was nothing a Yelp employee could do to alter their results. Second she learned that she really should pay Yelp lots of money so she could fix up her page, post pictures, etc. Maybe if she paid something could be done about those bad reviews…

My conclusion from this wasn't hard to figure. Yelp is corrupt (pay to play!). Yelp is incompetent (bad algorithms, double-posting, bad arithmetic). Sensible people should ignore Yelp.

Conclusion

I'm tempted to say that Yelp fits right in with its first tier Big Tech buddies, and of course it does. If you can stand it, check out the summary of their issues on Wikipedia. I did and also went to some sources to make sure it wasn't a Wiki hit-job. I also checked out the disastrous business and personal judgment and behavior of the leaders. Pathetic. Yelp is now on my blocked list of websites.

May 12, 2021
Software Giants: Image and Reality at Facebook

I am perpetually amazed at the flood of reverent articles about the wonderful big software companies that are inflicted on us. How great are their leaders! How wonderful it is to work there! Everyone should emulate their business practices! Their products are awesome!

The reason why the sycophantic flood continues is based in simple economics, but why most people appear to buy the b.s. is beyond me. You don't have to be a hardened cynic to see past the image.

This subject is worth a book or two. I've contributed just a couple blog posts. This post is another one on the wonderful Facebook, which (supposedly) does so much to demonstrate software excellence and contribute to our betterment.

Why Facebook is Wonderful

An article has just appeared about the wonderfulness of Facebook. The article is an interview by John Battelle with Lori Goler, who is "VP of People" at Facebook, leading the company's growth from 500 employees to 15,000. Here she is:

She sounds like a really nice person. I've worked with the interviewer, John Battelle, at one of his prior ventures, and he's a great guy.

The whole article is worth scanning. But the subhead gives you the idea: Facebook is "the world's most admired employer." Here are a couple quotes from Ms. Goler:

We are really looking for builders…What goes along with that is a learning mindset.

Being a strengths-based organization is a place where you are really looking to put people in roles where they are doing work they enjoy that plays to their strengths. … It’s where you get the best teamwork. It’s where the people are able to do the best work of their lives.

For us, the mission is, “To make the world more open and connected,” so it makes sense that our culture is open and connected. Then internally, we reflect that culture.

What we find is that what it really means is that people have all the context they need to be able to work with great autonomy in the organization, which of course leads to greater innovation and greater impact. It’s been a virtuous cycle for us.

According to the article, Facebook is a great place with a socially uplifting mission, populated by great people who are always learning and in roles where their strengths are tapped and their work has impact is fulfilling, in a completely open and supportive environment. Wow. Who wouldn't want to work there?

Another view of Facebook

For a contrasting view of Facebook, I recommend reading this book:

Warning: I had to force myself to get through the book; the author's self-described behavior was distasteful, to put it mildly. But it made the rest of his descriptions the more credible, and he said nothing that contradicted my inside knowledge.

The book has gotten lots of attention. It's been reviewed by major media, for example the New York Times:

And by tech journals, for example Tech Crunch, which declared it was the "year's best non-business book about business"

The book is #1 in several categories at Amazon. The top-rated review is telling:

….

Perhaps you can see that there is a contrast, shall we say, between the wonderfulness of Facebook as presented by its leaders and the reality. But this makes sense. What was the job of the VP People before getting that job? Marketing! In other words, telling stories to get you to buy stuff. She is continuing to do her job well, i.e., selling Facebook as a great place to work.

Facebook's Product

Well, maybe it isn't such a great place to work in spite of all the propaganda, but at least those 15,000 people turn out a great, high-quality product, right?

Here's a post about software quality issues with a section on Facebook, and here are details about the inability of those 15,000 engineers to turn out a product that has reasonable quality, even after many attempts — as judged by their own users.

It's not just Facebook. It's Google and the rest. Think about this: with such wonderful employees and huge cash reserves, why can't they make their own products work, much less innovate? If they're so innovative, why does so much of what they "innovate" come from acquisitions? See this for details.

You might ask, if their software quality is so awful, how did they become so big and valuable? Good question. Zuckerberg made some world-class smart business strategy moves to get it going. See here for details.

Why this matters

These observations about image and reality at the big famous software companies have huge practical implications for small companies, managers and programmers.

I have often observed that when board members want to hire a top executive, or when managers want to fill an important software position, they often value highly a candidate's having done a stint at one of these famous giants. They'll think something like "Facebook has a product that nearly everyone uses; I want to build a product that nearly everyone uses; therefore, I'll hire people from Facebook, and I'll get a product that everyone uses."

Of course rarely will someone come out and say something like that, but the Facebook (or Google or whatever) aura is so strong, people often act as though they believe it. On the other hand, if you really get the perspective about the inept software giants described here and confirmed widely, you will tend to avoid hiring people from Facebook (or wherever) because you know you're likely to drag down your company to its abysmal level!

Lots more detail on this and related subjects is in my book on Software People. Or for an illustration from a whole different direction, consider the incompetent doctor and nurse in the PBS civil war hospital drama Mercy Street.

Nurse Hastings frequently brings up the fact that she served with Florence Nightingale as proof that her opinions are the best.

Conclusion

Nothing is going to change. Major corporations of all kinds, even more so the big tech ones, will beat the drums of self-promotion, selling themselves to customers and potential employees. It's in the interest of groups that hunger for money and attention from the big companies to make nice, and trumpet the self-congratulations. The big companies will continue to be unable to innovate, and will instead buy innovative companies. Sometimes the contrast between the image that is widely promoted and the reality gets to me. At minimum, my hope is that you're working at a place that's far better than Facebook, and that you avoid error of attribution I have described here.

March 21, 2017
Organizing for Successful Innovation: Recent History
One of today's hottest trends is fostering innovation. It's real important! There are books, conferences, certified experts and all sorts of things. Let's do two things: (1) look at the origins of today's acknowledged tech leaders; and (2) see how those tech leaders innovate today.

The origins of today's tech leaders

What would we find if we looked at the origins of some important organizations that took the market by storm, grew rapidly and became part of the modern landscape? Did they come from people following the popular methods for fostering innovation? Let's look at some big, successful tech companies, and find out how they got started. There are two possibilities:
1. It came out of some large organization that followed modern innovation methods. Or its founders were avid readers of books on innovation, certified innovation trainers, attendees of innovation conferences, or otherwise showed that they were nurtured by innovation thinking.
2. It was started by one or two people who set out on a mission without any of #1 and kept marching forward until they got it done, perhaps with the help of VC's (venture capitalists).
I've already discussed the cases of Microsoft, Facebook and Oracle here. Their founders not only lacked training in "innovation," they were all college drop-outs! How could they have possibly founded three of the largest, most successful and valuable tech organizations? Must have been luck, I guess.

Maybe they're the exceptions! Maybe most of the rest fit comfortably into the mold of #1! Let's look at a few:
- Apple. Jobs and Wozniak. College drop-outs.
- Amazon. Bezos. Princeton grad, worked in finance, hedge funds. Jumped on a vision.
- Dell. Michael Dell. Started in his dorm room, dropped out.
- Google. Stanford grad students, dropped out. VC backing.
Why all the college drop-outs? In each case, a founder who was already obsessively good at something saw a related opportunity (sometimes with a buddy), dove in to make sure he didn't miss the opportunity, and made it more important than anything else. Including "education."

A pattern seems to be emerging here. It's not looking good for theory #1.

How today's tech leaders foster innovation

Now that these companies are big and successful, surely they are great at innovation, right? They must have certified innovation experts just crawling the halls, and a good fraction of the staff out attending innovation conferences, right? They must be just cranking out innovations left and right!

They try, in various ways. But it turns out that these great companies aren't any better than any other large organization at innovating — and you can see it by all the acquisitions they do! They have HUGE resources — how could some scrappy bunch of nobodies possibly come up with something they couldn't invent themselves?

Well it happens. It's happening right now, in AI. Look at who's doing the acquiring:

Facebook alone has made more than 50 acquisitions, most of them since 2010.

How about Google, who are supposed to be the smartest and most innovative of all? They've acquired more than 180 companies. Someone figured out the ten most expensive buys:

And the grand total is …

Now, that's innovation for you. Just ask your boss for a multi-billion dollar budget, and you'll be able to innovate like Google!

Until then, remember, there really is a better way to approach making your company better, even if it is unlikely to win any awards for "innovation."
May 3, 2016
The Apple-FBI Fiasco

The brouhaha with Apple and the FBI's investigation of an act of terrorism is tragic, comedic, scary and ridiculous. The only good "side" to take here is a side that few people, and none of the major actors, appear to be on.

Here are some of the major points.

Why the FBI needed help. The FBI should have submitted the phone to Apple for cracking immediately, using Apple's standard procedure for this. Instead, they bungled it. They changed the password and locked themselves out of the phone and its iCloud backup.

Whose phone? Apple got on it's higher-than-high horse refusing to help crack the phone because it protects the privacy of individuals. But the terrorists had already destroyed their personal phones. This was a terrorist's government-issued work phone. No privacy was involved.

The FBI's "unprecedented" request to Apple. Apple has a department that cracks phones. They crack thousands a year, and hundreds a year just for national security cases. Apple has a formalized process for it, which as of today remains on their website. The FBI's request should have been run-of-the-mill. Details here.

The slippery slope. Apple made claims about how responding to the FBI request would create a master key that would soon render all Apple phones insecure. This was bogus, as I detail here.

Privacy uber alles. Apple stood up as the firm defender of personal privacy — including that of murderers and other criminals.

Lost opportunity. Apple could have come out of this a hero — a strong protector of personal privacy and a strong ally of law enforcement against terrorists and criminals. Here is how.

Apple's insecure software. Apple's wants us to think their software is wonderful and their security flawless. No one mentions the scores of bugs that riddle their software. With each release, they introduce at least as many new bugs as they fix. Some of the bugs are security holes! White-hat hackers find some of them and tell Apple; Apple responds by eventually fixing the bugs and eventually releasing the fixes.

Finally cracking the phone. After all the sturm-und-drang, a "private company" approached the FBI and offered to crack the phone — and cracked it, leading the FBI to withdraw their suit against Apple. The company is Cellebrite, which has a commercial service that cracks iPhones in a forensically sound way. Do you think someone at the FBI could have used Google to find this group before suing Apple? Do you think Apple could have referred the FBI to them quietly instead of making a stink?

No one comes out of this mess looking good, including the media, which did little research and simply took sides. For example, I have found no media outlet mention Apple's standard phone-cracking service, which I published here. After this and the recent events in Europe, who can feel good about either the FBI or Apple?

March 31, 2016
Apple can help fight crime while maintaining privacy
Apple can and should maintain the privacy of the information their customers have on Apple devices. But what if the owner is a criminal or terrorist, and the relevant law enforcement agency has a court-ordered warrant? Apple should bend over backwards to help the agency fight crime and terrorism. It can do this without "back doors" or any of the awful things that some people talk about.

The government

The government scares me. I don’t want them anywhere near my private information. They have way too much power. If any little thing goes wrong, someone in government can trample all over me. My fear is equal opportunity. If Republicans are in charge, some of them will be corrupt and will decide to use my private information to trample on my rights. If Democrats are in charge, same thing. And bureaucrats of whatever stripe … I shudder. I want to be able to have my private information encrypted and secure, so that no one – including the institutions who are supposed to be keeping us safe – has access to it. PERIOD.

Sadly, the government already has whole huge piles of my private information all over the place in their files and computers. Moreover, the government appears to be incompetent at keeping private information private. The IRS has been hacked. The White House itself has been hacked. Even that biggest and baddest of security agencies, the NSA, had a massive insider breach. This is not the sort of thing that’s going to be fixed, because they don’t even have the theory of information security right, much less the practice. Details here.

On the other hand…

There are bad guys out there!

Bad guys are bad. They want to steal things. Some of them want to hurt me. They have all sorts of reasons. Some are crazy, some are sociopaths, some are evil, some are driven by a religious and/or political ideology that leads them to commit acts of violence; sometimes we call them terrorists. People in various institutions have the job of keeping law-biding people safe from the depredations of criminals, crazies and terrorists, and/or tracking them down after they’ve done one of the heinous things they are wont to do. These protectors including various branches of the military and other branches of the government, including the CIA, FBI, NSA and others. Like any normal, sane person, I want to be safe. I want someone to keep me safe from the bad guys, and when bad things happen, I want someone to track down the bad guys to prevent them from doing more bad, and to send a message to other bad guys that they probably won’t get away with whatever bad thing they have in mind.

This means…

The government needs to keep out of the private business of the citizens. We are part of a country ruled by a Constitution. There is a Bill of Rights, the fourth amendment in particular. HOWEVER: The government's job includes keeping us citizens safe while protecting our rights. Part of the job.

The people who keep us safe and dig into crimes when prevention hasn’t prevented need to be able to do their jobs. If the courts agree to issue a subpoena, they need to be able to search for evidence. Under the fourth amendment and codified in long-standing procedure, there is a process for ensuring that the privacy of law-abiding citizens is maintained, while at the same time ensuring that, with proper judicial approval, searches and seizures can be performed to maintain the safety of citizens.

Under the right circumstances and controls, sane people want government law enforcement agents to do their jobs, protect us and catch wrong-doers.

What about Apple?

Prior to iOS 8 and the current brouhaha, Apple responded as it should have to requests of this kind, thousands per year of normal requests and hundreds per year involving national security. See here for details. Suddenly they changed. Here is the choice they made.

Currently Apple has a well-deserved reputation as a criminal’s friend and supporter of terrorists. Do you think the bad guys don't pay attention? They do.

What Apple should do

Apple should become:
- the best friend of law-biding citizens who want to maintain the privacy that is their right under the Fourth Amendment, while at the same time becoming
- the scourge of criminals and terrorists.
Specifically, Apple should strengthen and grow the facility they already operate on their Cupertino campus to receive and crack the devices of criminals and others, under strict subpoena and court order control. As they do today. They can and should extend this valuable, safety-maintaining service to iOS 8 and all future hardware and software.

Would this be expensive? What if it cost, say, $20 million a year? That amounts to less than 0.01% of the CASH that Apple has on hand. It would be a rounding error at ten times the cost.

Apple could brand the center as the scourge of criminals and terrorists, and make their phones something that bad guys actively avoid using. That way, anyone who uses an iPhone is proclaiming that they’re a good guy – and they’re also proclaiming that Apple keeps their private information safe and secure, unlike (I’m sad to say) most government agencies.

Is this possible? Yes. Apple has wisely avoided denying that they are incapable of cracking a phone that is in their physical possession. Which are the only phones they should be cracking anyway. Should they give their tools to anyone else? NO WAY!

What about phones that are in the field? Could Apple remotely hack them? Of course they could! Strictly under court order, strictly from the Cupertino Bat-cave, and solely the identified phone under Warrant.

Apple's ability to crack phones under these strictly limited circumstances has NOTHING to do with creating dangerous "back doors" or somehow defeating amazing encryption. It's about hardware and the software that runs on it, both of which are entirely of Apple's design and under their control.

Apple has the opportunity to protect the privacy of its customers much more effectively than the government does, while at the same time helping law enforcement protect us against criminals and terrorists. I hope they'll step up and do the right thing.
March 16, 2016
Apple’s Cancer Prevention Strategy

The CEO of Apple declared that he has joined the ranks of the nation's oncologists, and is working to prevent the government from forcing Apple to create a new form of cancer and "expose hundreds of millions of people to issues."

The CEO of Apple is anxious to prevent future "issues."

Let's look at the case of Brittney Mills,

This is an example of an "issue" that took place in April of 2015 in Baton Rouge, LA, long before the Apple CEO got worried about cancer. Here's the "issue" that Ms. Mills experienced:

Investigators still haven't been able to find who killed her and her unborn child. They've tried hard.

They went to Apple for help. Apple refused to help the police get the evidence that might lead them to the person who killed Brittney Mills and her unborn child. The local district attorney wrote to the US Senate Judiciary committee about the case:

His pleas and those of Brittney Mills' family were ignored. The case of Brittney Mills isn't the only one:

Law enforcement getting information from a dead person's cell phone is similar to getting information from their wallet: not something anyone would normally do — but when the person is dead, the only way to proceed.

Apple's refusal to help Baton Rouge law enforcement catch the person who murdered Brittany Mills is taking place in thousands of cases all over the US:

Apple's response? An escalating war of words. A half hour's worth in ABC's "exclusive" interview with the CEO.

While declaring how important safety is, "doing this," i.e., helping get information from the cell phones of murdered pregnant women, "could expose people to incredible vulnerabilities." Does this mean the Apple CEO is concerned about future "incredible vulnerabilities" that are worse than being murdered?

And then we have the old slippery slope argument:

OOOhhhh: law enforcement might turn on the camera!! I guess the Apple CEO thinks that's worse than being a pregnant woman living alone, opening your door at night for someone you know, getting shot and dying. And not being able to find out who did it.

Now we get to what Apple is being asked by the courts to do, which is the equivalent of creating cancer:

I demonstrated in my prior post that Apple has cooperated with law enforcement in the past, and given out private information on literally tens of thousands of cases, including at least a thousand cases a year involving national security. Apple was able to provide this information because they had written for earlier releases of iOS a much stronger version of what is needed for iOS 8. Apple has written it. It wasn't cancerous before. How would it be cancerous now?

Similarly, when he claims that helping the court would "expose hundreds of millions of people to issues," he assumes this software would somehow escape from Apple's control, when the prior versions did not.

Apple does know a way to avoid the problem. And it's had years of experience over tens of thousands of cases that the method is safe and effective.

The issue is simple. Apple refused to provide the help needed to identify the murderer of Brittany Mills and her unborn child. Apple says providing that help is like unleashing a plague of cancer. I say to Apple: please unleash that cancer.

February 25, 2016
Apple’s Approach to Privacy, Terrorists and Criminals
Apple is locked in a public battle with the prosecutors of the San Bernardino terrorist case about helping the FBI. Tim Cook has been in full public-relations mode asserting how this "unprecedented" request is like distributing a "master key" that will make everything on iPhones public.

The government's request (as opposed to how it's described in the media) is reasonable; it is a simple extension to iOS 8 of part of a service that Apple already provides to government agencies for tens of thousands of Apple devices. By refusing to continue providing the service, Apple prevents local police from returning stolen iPhones to their rightful owners. Apple prevents law enforcement from solving crimes of murder, sex abuse of children, sex trafficking, robbery and other crimes. And Apple prevents the FBI from keeping us safe from terrorists.

The awful things Cook claims will happen if he complies are already enabled by horribly buggy and security-hole-ridden Apple software. Nothing the government has requested will make things worse.

Apple’s official privacy policy

What was Apple’s privacy policy before the recent war of words on the subject? The policy is clearly stated on the Apple website. There are lots of words about how Apple loves and respects it customers, and Apple is wonderful. The words lead to this conclusion:

That sounds pretty stark! No back door and no server access. Ever! That sure sounds like my information is secure, no matter what!

Apple’s actions on privacy

As it turns out, those are weasel words. Which you can find out by a little digging. All you have to do is go to their “government information requests” page. There they admit that they respond to subpoenas and search warrants. But they “limit our response to only the data law enforcement is legally entitled to for the specific investigation.” Well, maybe it’s not so bad…

Scanning down the page, in HUGE type, is this assurance that practically no one is affected by all this:

An amazingly tiny fraction of “customers” have been affected by this grudging acceptance of government coercion.

How much does that tiny, tiny fraction amount to? Being super-conservative about doing the calculation, I took the quarterly sales just of iPhones only for the last 3 years (2013 to 2015) as reported publicly by Apple. Truncating each reported result to the lower million, the total is 546 million iPhones. The real number, including iPads and going back further in time, is probably more than twice that. But the arithmetic even for that number is interesting. Using Apple’s own 0.00673% number, the total is 36,745 customers.

That number does not include “national security” requests, which according to the same page, is more than 750 requests for the first half of 2015:

To summarize rhetoric and reality about Apple and privacy:

Rhetoric: We don’t create backdoors and “have never allowed any government to access our servers. And we never will.”

Reality: We dish out customer data as required, and do so by the tens of thousands. But we pout while we’re doing it.

What Apple really, really does

Dig a bit further, and you can download the details of what and how customer information is handled at Apple, in this document:

Here’s a bit of the table of contents:

You can see that the range and scope of information available goes way beyond anything you might imagine from scanning Apple's website pages.

The document also declares that Apple can provide an incredible amount of information from any iOS device prior to 8.0, but “will not” perform data extractions from 8.0 or later. The extraction “…can only be performed at Apple’s Cupertino, California headquarters…”

What the government wants

The government’s request is short and to the point.

They want help defeating iOS 8’s PIN brute-force avoidance mechanisms:

Here’s what they suggest an acceptable means of providing the help would be, a piece of loadable software:

They specifically request software that works for only that phone:

They don’t demand possessing the software; it’s OK if Apple physically has the device and keeps the developed software on site, without even requiring that government agents be present:

And if Apple can think of a different way to accomplish the same results, it’s OK with the court:

In summary, the court will provide Apple with the terrorist’s government-issued iPhone, and wants Apple to create software that will enable the government to do the hard work of figuring out the iPhone’s PIN code so that the government can access the data on the phone. The government is willing to let Apple do this work with the phone at Apple’s offices, with no government agents present, wants the software to work only for the iPhone in question, and does not request a copy of the software.

Tim Cook’s response

Apple hacks and gives the government the private data of tens of thousands of customers. Probably a thousand times a year for national security issues. It does this in its facilities, using software it developed for the purpose.

The feds are investigating a terrorist attack on US soil in which 14 innocent people were murdered. The phone in question wasn’t personally owned by Syed Farook; it was owned by the government agency for which he worked, and whose employees he murdered. Breaking years of Apple practice, Tim Cook refuses to help. He explains himself on the Apple website:

He declares the request “unprecedented.” Sure, if you ignore the tens of thousands of other requests Apple had no trouble satisfying.

He says the order “threatens the security of our customers.” And the possibility of future terrorist attacks doesn’t?

He says the order “has implications far beyond the legal case at hand.” Yes it does. But not the way he means it.

A little further down, he gets to the crux of the matter:

He claims he doesn’t have what the government wants. Everyone knows that, and it’s implied in the court order. But he had the equivalent for earlier versions of iOS.

He claims it’s “too dangerous to create.” While he blathers about encryption and about how Apple can’t get at your data, here he makes no claim that the software is impossible to write – and it’s not! He’s just saying he won’t create it, because he’s too moral or something, and the software would be too "dangerous." Although more powerful versions of the requested software were built by Apple for prior versions of iOS, and they somehow weren't dangerous.

He claims the request is for a “backdoor to the iPhone.” Wow. You can review the actual request above. It’s no such thing. It’s a piece of software that circumvents the iOS 8 defense against brute-force PIN-breaking. Apple gets to create the software and use it at their offices on the provided phone.

Cook goes on:

“The FBI wants us to make a new version of the iPhone operating system.” Maybe that sounds technical and accurate to someone who didn’t read the documents, but it simply isn’t true.

“In the wrong hands, this software…” How exactly is it going to get in the wrong hands, Mr. Cook? Apple employees have full and unfettered access to the source code of Apple software, including iOS. Any time one of them felt like it, they could make an unauthorized version and spirit it to some off-site server, and do all sorts of evil with it. That was true yesterday, is true today, and will remain true regardless of what happens here. The current situation doesn’t change the chances of malicious software being used for bad purposes one iota.

“…would have the potential to unlock any iPhone in someone’s physical possession.” BZZZTTT! What this software would do would be exactly and only what the government is asking for: make it possible to brute-force hack the PIN code, which has one million possible combinations for the default 6-digit PIN. For normal humans, this means you would have to:
- Acquire someone’s iPhone
- Get and load the hacking software onto it, assuming it has somehow wafted out of Apple
- Then, by hand, try 6 digit PIN codes until you got to the one that worked
- On average, this would occur after entering half the possible codes, a total of 3 million digits. This would take more than 34 days of continuous one digit per second attempts.
- Or, if you really are a super-hacker, you could automate the process. Which I won’t go into here.
Cook then gets wilder:

Yes, the software, once created, could, would and should be used on "any number of devices." Devices that were provided to Apple at their offices with proper documentation and court orders. Most of these devices, as today, would have been lost by their owners, and Apple is helping the owners identify them so they can be recovered. Many of these devices, as today, would be evidence in criminal proceedings. And hundreds of these devices per year will be related to national security issues, as they are today.

I am very concerned about the FBI being blocked from tracking and stopping terrorists before they kill. But I'm equally concerned about the "merely" criminal aspects of this. For example:

Cook has more:

Because Apple built software used by Apple on specific phones delivered with court orders to Apple facilities, the government will now be able to listen to your microphone or camera. How exactly does this leap happen?

The fact is, Apple software was, is and will be chock full of security holes and other problems. Here is Apple's own list of the dozens of security problems that were fixed in iOS 7. After fixing all those problems, iOS should be secure, right? Apple then found more bugs, refused to fix them in user's devices, and instead released iOS 8 with no less than 53 additional fixes to security flaws. So how did iOS 8 go, with all those fixes? Not so well, according to Wired:

Finally, Tim Cook once more:

Apple products have been buggy and filled with security holes in every release. It's riddled with back doors, side doors and bottom doors, all because of Apple's ineptness. It's not getting better. Mr. Cook wants us to fear that the mean government will force us to walk around without privacy. Well, we already are! And it's Apple software that's responsible! Extending Apple's existing practice to iOS 8 will not create a new situation — it will maintain Apple's historic cooperation with the legitimate law enforcement operations of government, protecting us from terrorists and criminals.

What is this really about?

I wish I knew. But it's hard not to think of money and market positioning. There is a large portion of the public that thinks that Wall Street and Big Corporations are evil. Meanwhile, Apple makes products that are used by millions of people who think this way. Apple wants to market itself as being for the 99% of people.

But it has a problem. It's one of the richest, most valuable corporations in the world. It charges top dollar for its products, which are entirely made in cheap-labor countries. It plays games to avoid paying taxes. It's bigger and richer than Wall Street! It's even richer than the US Treasury:

It's quite reasonable to imagine that Tim Cook is following in the Steve Jobs tradition of marketing magic to divert its customers from looking at the numbers. Numbers that show that Apple is a corporate behemoth whose sales are slowing, whose new product initiatives have failed, and is desperate to bolster its brand and hold onto customer trust (and revenue) it does not deserve.
February 22, 2016
Large Organization Software Fails: the case of Microsoft Windows

Large organizations have trouble building software. This has been true since the dawn of software history, and shows no signs of changing. The decades-long, rolling disaster of Microsoft Windows is a great example of this. I've been hit personally with this. Recent experiences with Windows 8 have renewed my appreciation of the breadth and depth of the on-going awfulness of Windows.

Windows Screen Saver

I got a new computer. It had Windows 8. I was setting up my new machine and I wanted to do something simple. I had remembered that in some earlier version of Windows, you could get the screen saver to display the file name of the photo it was showing. This was useful if you wanted to get your hands on the photo that just flashed by. It's a pretty small feature, but one anyone who stores photos on their PC could find it useful.

So I drilled in to the screen saver.

I went into the settings, and didn't see the control I was hoping would be there.

So I clicked on Help, something I rarely do, but what the heck, that's what it's there for. Here's what I got: The content is missing!

It's a little thing. It's not like my computer crashed. In the world of books, it's like a footnote was missing — hey, that's an idea, let's compare the new edition of Windows to the new edition of a book!

Software and Books

Most of us know how to judge books. If a book is poorly produced, like the pages tear easily and the type is hard to read, most of us will toss it aside — it may have great content, but it's not worth reading. If we get past the first impression, we'll dive in and start reading. The next potential barrier is how well the book has been edited. If the book is full of spelling, usage and grammatical errors, many of us will think poorly of the author, the editor and the publishing house — the author shouldn't have made the mistakes in the first place, the editor should have caught and corrected them, and the publishing house shouldn't have put sloppy trash in print. Then and only then do we get to the style and substance of the book.

I read a lot of books from many publishers in many genres — fiction, history, science, etc. — and I'm happy to report that I rarely encounter a published book that has editing errors.

And by the time a particularly timeless book gets to later editions? There are never errors.

In that context, how is Windows 8?

I've got the latest version of Windows, 8.1, running on a new machine. It's hardly a first edition. Microsoft pours out updates, and I'm up to date. Here's a snapshot:

Note the scroll bar — there were hundreds more updates that had been applied.

The lovely option that lets you see the file name along with the picture was in an earlier version of Windows. Making a new edition of software isn't that much different than making a new edition of a book — basically, unless you add or change something, it stays the same. In this case, someone had to make a conscious decision to drop an isolated, harmless feature that gave value to many customers.

Why would someone do that? It's more trouble to drop a feature than just let it ride along on the next edition, so someone had to actively remove it. There is no conceivable objection to the feature. While not everyone would want it, since it's an opt-in feature, it harms no one. It's like someone deciding to drop a short appendix from a book — not everyone will want it, but those who do value it. In the paper publishing world, dropping it might save a page or two. But in the electronic world? There's no conceivable reason.

I don't claim for a second that displaying the file name on the screen saver is important. I simply claim that the decision to drop it exemplifies the pervasive anti-customer attitude of the Microsoft organization, which unfortunately is typical of large software-building organizations in general.

It's the missing Help file though, that really set me off. Again, it's a trivial error, like dropping a footnote. But why would you do it? How could it possibly slip though what should be a totally automated editing/QA process?? It may somehow be complicated in the labyrinthine world of Windows development, but it's a fixable thing. You have a program that assures that for each instance of Help there's a corresponding piece of content, and for each piece of content there's a way to reach it. There either is no such program or it's broken. In the overall scheme of things (Windows remains horrifically slow, it freezes and crashes, etc.) it's a small thing, but surely by the edition of Windows 8 I am suffering with it would have been found and fixed?

Conclusion

Software is all about productivity, attention to detail and automation. Unless you've got a de facto monopoly, software is also about meeting customer needs. Large organizations in general (for example government, big corporations) and Microsoft in particular don't get that, in spite of the billions they spend on development and (supposedly) quality. I would love to be able to say it's getting better, but most of the evidence is on the other side. Which is why, among other things, good software will continue to be produced mostly by organizations that are small and willing to do things the "wrong" way.

August 3, 2015
Facebook’s Software Quality: the Implications

I have pointed out Facebook's lack of desire or ability (who cares which?) to deliver software that actually works. I've pointed out that they're hardly alone in this respect. It's important to accept this observation as true, so that you can change behaviors that may have been unconsciously predicated on the supposition that Facebook delivers great software, effectively and efficiently. They don't. So don't hire their people and expect great things to happen, and don't mindlessly emulate their methods or use their tools!

The Unspoken Assumption

Facebook is a wildly successful company, worth over $200 billion. I'd like my company to be worth even 1% of Facebook. So I better find out what Facebook did, and learn from it. Facebook is a software company, so their engineers must be smart and effective. I better get some of them in so they can teach us the "Facebook way." And their tools — wow. If Facebook uses something, what an endorsement that is. My guys had better have a real good reason to use something else; I look at what FB's worth and what we're worth — don't we want to be like them? If a tool or method is good enough for FB, it should be plenty good enough for us.

The role played by software in FB's success

Here's the logic:

FB is wildly successful.

FB is built on software.

Therefore, FB software must be wildly excellent.

We already know by examining the quality of FB software that it's crappy. So we have reason to suspect that the virtues of FB software may NOT be a driver of FB's success. Consider this thought: What if FB is wildly successful IN SPITE OF its crappy software? If that's true, the LAST thing you'd want to do would be to infect your reasonably healthy engineers with disease vectors from FB.

Explaining FB's Success

There are lots of reasons software companies can become very successful other than having great software. In fact, by the time a company gets large, bureaucracy and mediocrity normally take over, and any great qualities in the software are normally eliminated. The most common reason a software company gets and stays successful is the network effect, the self-validating notion that "everyone" is using the software, therefore I should too.

The network effect becomes even more powerful when there's a marketplace. E-Bay is a great example. If you're a seller, you want to sell in the place that has the most buyers. If you're a buyer, you want the greatest choice of things to buy. Similarly, if FB is where all your friends are, you'd better sign up — which makes the network effect even stronger.

FB, by chance or plan, leveraged the network effect for growth brilliantly. Harvard already had a physical book with everyone's pictures in it, called the Facebook by students. The basic education and promotion problem was solved out of the gate: Harvard students knew what a "facebook" was; they all had a physical one, and used it, if only because their own information was there. For example, here's me in the 1968 edition:
However straight-laced those Harvard freshman looked, a fair number of them were hackers and troublemakers. Here's the very last page of the 1968 FB. Look at the last guy listed.

There's a similar entry, with a different photo, at the start of the book.

Zuckerberg was solidly in the long-standing Harvard hacker tradition. He had already illictly grabbed student photos for a prior application, which both got him in trouble and made him famous on campus. So when he launched "thefacebook," of course all the Harvard students would check it out. He did this in January. It was used by about half of all Harvard undergrads within a month.

His next smart move was to open it just to students at a couple more elite schools, and then Ivy League schools. Once established there, he expanded. He did NOT open the doors and let anyone join — he moved from one natural community to the next, letting the network effect do its magic before moving on. Finally, alumni were allowed to join, but only if they had a .edu address proving their affiliation. That's when I joined. Only after a whole generation of students had made it the standard did FB allow their parents to join.

The quality of the software had nothing to do with this. If people had to pay for it, FB would have flopped. Feature after feature came pouring out of the self-declared brilliant minds of the top people at FB, many of them flops, mixed in with scary experiments with privacy. But it was "good enough" most of the time, it's free, it's where your friends are, what can you do?

The conclusion is clear: FB grew to be a huge success IN SPITE OF having rotten software quality and development methods that are just horrible.

The FB environment and yours

Facebook software development methods and tools are NOT something a small, fast-moving, high-quality software shop should want to emulate. Their quality methods in particular are not only trashed by their users, but also by a fair number of ex-employees. The same thing goes for the computing and server environment.

If you find a talented ex-FB-er, by means hire him or her — but only after verifying that they're sick of how things are done at FB and want to work at a high-quality place.

Above all, don't emulate the actions of FB's leadership. It's the network-effect flywheel that continues to bring eyeballs to their applications, NOT their great software.

And think about this: if they're so brilliant and such great developers, why have they done about 50 acquisitions in their short life, a couple of which are important to their growth?

December 7, 2014
Facebook’s Software Quality: the Facts

Facebook is an incredibly successful company, one of the most valuable on the planet. It is natural to assume that a main reason for this is that they've got a boatload of great programmers who produce code that users love. This assumption is wrong. In fact, the widespread adoption of Facebook masks deep, long-term quality issues that are not getting better.

Facebook Success

Facebook recently passed $200 billion in market value. Amazing! It has billions of users world-wide and has no serious competition. No one can question FB's success in user count and market capitalization.

Facebook Mobile App

Mobile device use is going through the roof. We are in the middle of a massive, rapid migration from workstations and laptops to tablets and smart phones. This trend impacts FB just like everyone else. At the recent Money2020 conference, a top FB executive laid out the numbers, which are stunning; in short, FB mobile use nearly equals normal web use. If anything is important at FB, it's got to be getting the mobile app right.

Facebook Mobile App Quality

So how is FB doing, this premier, ultra-successful company with no lack of resources to do an excellent job? They've got to be doing way better than the rest of the industry, right?

Let's start by looking at user reviews:

Not too bad, 4 stars out of 5, right? But out of more than 22 million reviews, more than a quarter gave 1, 2 or 3 stars, more than 6,000,000 reviewers! Let's look at a few of those reviews. (I didn't scan for exceptionally bad reviews; I just picked off ones that were near the top of the Play store.)

Here are a couple reviews. Cindy gave 1 star because the app doesn't work at all, and Johnny gave 2 because he suddenly can't avoid being buried in notifications.

Here are a couple more reviews. The third reviewer gave 3 stars even though the app is basically disfunctional.

These are educational:

The 3 on the left describe things that worked on a prior release that no longer work, which is the cardinal sin of quality testing. Look at Bratty's review awarding 4 stars, even though he/she can't use the app at all. Makes you wonder if anything but 5 stars is good for FB. Jeremy's review sums it up: "you're still not listening to your users." If only 5 stars represents satisfied users, the ratings mean that about half of FB app users have a serious bone to pick. Which is quite a statement.

FB App Quality in Context

Compare the performance of the FB app to the performance of your car. Getting a new release of the app is similar to getting your car back from the repair shop, only with little trouble on your part and no expense. Most cars run pretty well — they start in the morning, run through the day, and rarely break down. When you get your car back from the repair shop, it's even better, even less likely to break down.

Not true for FB. Even though it's "in the repair shop" pretty frequently, the FB "mechanics" all too often find a way to break things that used to work, and fail to fix things that didn't work when it went into the "shop." FB programmers and managers think they're way smarter than auto mechanics, but if the car people performed even a little bit like the FB crew, they'd be out of business. The reality is that, with all their oh-so-highly-educated-and-smart mountains of cool (mostly) dudes, the FB crowd can't come close to delivering the quality that nearly every corner-garage mechanic delivers every day.

FB quality stinks, and it stinks for their fastest-growing, flagship product. In saying so, I'm simply summarizing the expressed experiences of literally millions of their users. There are ways to achieve high quality software. FB does not lack the resources. The fact that they don't deliver quality and aren't even embarassed about it tells us that they just don't care.

November 23, 2014
The Bogus Basis of “Trending on Twitter”

People write and talk about what's "trending on Twitter" as though the trend meant something. It doesn't. It's based on deeply flawed Twitter search software that gives random, widely varying results. I know the weatherman is often wrong, but what if he said it was going to be sunny in the 70's tommorow and as often as not there was a blizzard — would you keep listening? It's the same with Twitter, only worse.

Trending on Twitter is everywhere

It's amazing how widespread this useless stuff is. New York Times editors are in on the game.

It's even now got a prominent place on Wall Street!

You can not only follow what's trending in general, but you can narrow it down to different locations.

When a Twitter account is hacked, bad things happen.

And sure enough, the markets react.

We seem to care not only about what the Boston bomber says on Twitter:

But we also pay attention to the useless Twitter trends about it:

We've really got to stop this. It's not as though we've got reliable data here. It's just not. Twitter has been a technical joke for years, and there are no signs of improvement.

Trending on Twitter is meaningless garbage

I don't have the access to perform a universal test. But I did perform a test, and anyone else can reproduce my results. I did searches over a couple week period for the same term and saved the results. Sometimes the results were correct, but most of the time, items that were there before disappeared, only to pop up again on a subsequent search. Sometimes just a couple things were missing, and sometimes the gap was massive. Here is the evidence.

Then I took the search that appeared to have the most gaps, and performed the identical search about a week later. As I documented, one search had just 5 items and the other had 32, when they should have been identical. About 85% of the search results had been dropped by Twitter!

"Trending on Twitter" is based on comparing results of a search performed on one day to the same search performed on other days. If the number of results goes up or down, you've got a trend. Or so you think. But what if the results are really as bad as I have documented? I found that "blackliszt" went up or down by a factor of 6, like 600%! Wow!

Conclusion

Twitter software has always been bad. Management has learned to disguise the awfulness by suppressing the appearance of the "fail whale," but they clearly haven't actually, you know, made the software better. Anyone who takes its results as actually meaning something is depending on bogus data.

May 16, 2013
Twitter Software Quality: An Oxymoron

Twitter software quality Stinks. As I've demonstrated. On revisting and updating the facts, I've decided that "Twitter Software Quality" should be promoted to the status of oxymoron, joining the august company of terms such as "southern efficiency," "northern hospitality," and "government worker."

A Brief History of Random Awfulness

I took samples of searches for "blackliszt" on these dates: Apr 18, 19, 20, 22, 24, 25, May 1, 8. A total of 8 samples.

All searches were done as "All" to tell Twitter I wanted, you know, all the results, not just the ones Twitter felt like disclosing at the moment.

I only grabbed the first page from each search. I've shown the results in another post. Of the 8 searches, the one on May 1 is the most extreme. Here's a copy of the May 1 search for "blackliszt:"

You can see there are 5 tweets in the list of results, from Apr 11 to Oct 13. I decided to try to find out how many tweets there actually were between Oct 13 2012 and May 1, 2013, the date of the search pictured above.

I did this research on May 8. At least on May 8, Twitter was willing to admit that there were a total of 32 tweets in the same date range, although one of them (Feb 27) appears twice. Here they are:

A Twitter search for "blackliszt" performed on May 1 resulted in a list of 5 tweets going back to Oct 13. The same search for "blackliszt" performed on May 8 (above) resulted in a list of 32 tweets that should have been returned by the May 1 search. Maybe there are more! Given that one is double-counted (Feb 27), who the &*() knows?? What I do know is that on May 1, Twitter decided to discard 27 out of 32 potential results of a search. Roughly 85% of the tweets were gone!

Summary

I already knew that Twitter software quality was bad. It turns out that it's worse than I ever imagined. It's "Twitter-quality"-is-an-oxymoron bad.

You know all those "trending on Twitter" items you're seeing now that seem so modern and cool? They all assume that getting more or fewer results from a search means something. We now know that the results can easily go up by a factor of six, or drop by the same factor, just because of Twitter "quality." It's obvious that "trending on twitter" deserves to be the punchline of a joke, not something that anyone pays attention to.

May 15, 2013
Twitter Software Quality Stinks

There are big problems with software quality. The problems range from social apps to corporate to academia, include "mission critical" software, and everywhere in between. The social apps in particular have decided it's embarassing. But instead of actually, you know, fixing the problems, they seem to have decided to mask the problems! Twitter is a great example of this disease.

Two ways of Responding when you don't know the Answer

Suppose you're a kid and someone is demanding answers from you. Either you know the answer or you don't. If you know the answer, it's simple: just give the answer!

Q: When did Columbus sail the ocean blue?

A: 1492

If you don't know the answer, there are two ways to respond: the right way and the wrong way. The right way to respond is simple: Just say you don't know!

Q: When did Columbus sail the ocean blue?

A: I don't know.

The wrong way to respond is a little more complicated. You have to guess at an answer, state it as though you knew the answer, and hope no one cares or that the person asking doesn't know either so you can get away with it.

Q: When did Columbus sail the ocean blue?

A: 1542.

When the question you're asked has several answers, you can be wrong in a different way. For example:

Q: Name the ships in Columbus' voyage to the New World.

A: The Nina and the Santa Maria.

Q: Is that all of them?

A: Yes.

Twitter's Response when it doesn't know the answer

I never thought it would happen, but now I have fond feelings for Twitter's Fail Whale, which I haven't seen recently. You would think that the fail whale not showing up as often would be a good sign. It's not. It's a sign that Twitter has decided that it's better to lie than to admit it doesn't know the answer to the question you're asking. Instead of forthrightly saying "I don't know," Twitter now brazenly gives the wrong answer. Even worse, it gives a different wrong answer from one day to the next!

Twitter's Bogus Search results

Here are some screen shots of the results of the identical query, for "blackliszt," over a couple of weeks. I always selected "All results" to remove any excuse that Twitter was selecting the "top" results to help me out.

Let's go through time. Here's the result from the first day, Apr 18:

I tried again the following day, Apr 19, and was quite surprised with the result: the Rebelmouse tweet simply disappeared, pulling an older one into the results!

On Apr 20 I added a tweet and did the search again. My new tweet was there, and RebelMouse came back!

On Apr 22 I tried yet again and got another brand-new variation: this time Cadencia's tweet disappeared!

The results were unchanged on Apr 24 and 25. I gave Twitter a couple days to lose some data, and had my patience rewarded when I searched again on May 1. The first result was Rebelmouse; the most recent posts, my post on ballet, Cadencia and Rob Majteles, were all gone! Here's May 1:

Finally, look at this simple list of my tweets taken Apr 23, not a search:

Note that I had tweets on Apr 10 and Mar 25, both of which included "blackliszt," neither of which appeared in any of the search results!!

Sadly, I can't even claim that the folks at Twitter have it out for me. It's just the way things work there … uhhh, I mean, the way things don't work there…

Conclusion

Social Media software quality stinks. It's worth every cent you paid for it. Oh, you didn't pay anything for it, you say? Well, that's my point. When a program like Twitter gives you an interface, lets you do a search, gives you a result that's even worse than my "Nina and Santa Maria" answer, brazenly implies that it's the right answer and everyone just ignores the issue, something is wrong.

Q to Twitter exec: Why does your software randomly leave out results from searches? Why should anyone look at "trending tweets" or anything else when the data is randomly bogus?

A: I've never been asked that question before. The answer is simple: I do it because I can, because I don't care, because no one else seems to and because I'm worth a great deal of money and you're not. Next question please.

Thanks to MaryAnn Bekkedahl for inspiring me to write this up.

May 2, 2013
Internet Software Quality Horror Shows

Whether the software is a cool social app, an academic website or a real business, there is a common theme: the software is poorly designed and, even worse, it just breaks. As in falls flat on the floor, waves its arms in surrender, and just gives up. And not just once — it keeps breaking! As I've said before, we really need a revolution in software quality.

Cool Social Apps

Hey, social is where it's at — how can billions of Facebook users be wrong? Before long, there will be as many FB users as MacDonald's has sold hamburgers (billions and billions)!

Those guys must be great programmers, huh? I mean, just look at their office:

Here's one of them giving a talk at a conference:

See how cool he is? He's just wearing a t-shirt, not even "business casual."

The other social media are just as cool. Here's a "chill" Twitter office:

And Jack Dorsey, the Twitter CEO — quite the opposite of a buttoned-down financial guy, huh?

It's perfectly obvious that these guys must write just the coolest, most awesome code ever. There's no way people this cool could make elementary programming mistakes, particularly when their application is so very dead-simple, and hardly ever changes — they could spend practically all their time being cool and polish up some already-faultless code a couple times a day, and still be OK.

Except this little detail, which I scraped from my own screen, and which I personally have seen countless times:

Yes, the famous Twitter fail whale. I think Twitter got tired of all the publicity their "cute" failure message was getting them, so they reverted to something more discrete; here's an example:

FB is just as bad, of course, and they've always tried to minimize the message when they screw up:

Apparently, FB is incapable of keeping even the most recent day's worth of updates on-line — you should try going back in history and seeing how far you get. Oh, you thought the stuff you wrote was your data, did you?

Naturally, it makes sense to consider that you get what you pay for; all these cool social apps are, after all, free. You can hardly complain when something you didn't pay for is flakey — return it and demand a full refund!

So let's turn to a more promising field. Everybody's supposed to go to college and learn stuff, so…

Academia

Let's see if the universities do any better. I was just on a local college's website, and it was even worse than Twitter — Twitter's code knew it was screwing up and put up the fail whale. In this case, any number of links I hit encountered badly broken code:

Oh, alright. The colleges are perpetually underfunded, and putting up a website that works isn't a high priority compared to … all the other things they spend money on. I guess.

Probably a real business does it better, right?

Profit-making Big Company

Even more so, an essential public service, like the cable company! Those guys have the money, the funding, the experience and the mandate to do it right. Let's pick the case where their motivation is the highest: collecting money.

Oops.

Just a few days ago, I was on my local cable provider's site trying to access my account. Here's what I got:

Not just once, but repeatedly, for hours!

But maybe it's just TW that's got problems — surely all the other big companies do things great, with their huge staffs and policies and procedures and all, right?

Sadly, no. Here's just one personal example from Verizon:

Summary

There's no getting around it. Software is just bad. Everywhere. We can speculate about why this is the case, but let's agree on the facts: it's bad, and not getting better.

January 22, 2012