The Black Liszt

Author: David B. Black

Speed-optimized Software QA; or Cancer
QA is QA, right? Either you’re committed to quality or you’re not. If you are, you use the widely accepted tools and techniques and produce a high-quality product, or you accept the fact that you’ll churn out crap. What else is there to say?

Here’s what: if you accept this view and apply it in the usual software development environment, your QA processes metastasize, invade all parts of the process, and cause endless pain and suffering – not to mention crap software. Eliminating the cancer can only be done by leaving the cancer-causing environment. That means finally saying good-by to peace-time software and going to war.

Software Quality Assurance

Software QA is a BIG subject. There are lots of methods within a broad set of accepted practices. Experts assure us that “quality” goes way beyond testing; it infiltrates every aspect of the software SDLC.

But testing by itself, which is just part of quality, is a HUGE subject. You can get certified. Here are some of subjects involved in testing according a leading certification organization.

Here’s what’s involved just in test planning:

The SQA cancer

Most organizations aren’t aware of it, but the typical, expectations-based, peace-time software development organization is highly prone to QA and testing cancer.

Everybody is supposed to do good work. You’re supposed to take the flawless results of the previous group’s work, do what you do to it, and pass it on to the next group – error-free.

When it’s obvious you’ve been given crap, it’s easy to reject what you’ve been given and throw it back at the slugs who gave the crap to you. But all too often, the problems don’t emerge until you’re pretty far down the line of doing your own work. Then it’s upsetting and embarrassing. The schedule is at risk, and who’s at fault?

Once this happens a couple times, on the sending and receiving ends, most people respond by creating as many quality measure as they can, to be applied to the work they’re given. Then, having been blamed for producing bad work a couple times, they run all sorts of tests on the work they’ve produced before passing it on.
Why wouldn’t you? You don’t want to receive bad work, and you certainly don’t want to be blamed for giving bad work to someone else. But you can see how the QA/testing steps are reproducing.

Inside coding itself, it can be even worse. In addition to inspecting the code that's calling them and that they're calling, the programmers can "enhance" their code with additional code whose sole purpose it to check whether the inputs they're given — at run-time — are correct, and then again to test the outputs they're about to give.
As though that isn't enough, lots of quality-minded programmers are way into writing code that tests each individual piece of code; this can be before you write the code (i.e., test-driven development, which is currently fashionable, something programmers say they want to do to show everyone else how good they are) or after you write the code (i.e., plain old unit testing).

What's worse is, I've just skimmed the surface!

There is no escape

You think that's bad? Try to do something about it. Whenever anything goes wrong, you're likely to hear one of the following:
- I wasn't allowed to buy adequate test tools
- We weren't adequately trained on the tools
- I didn't have enough time to get enough coverage
- Lots of old tests had bugs, I had to disable them to meet the deadline
- We changed so much old code, I didn't have time to update all the tests for it.
- My regression testing has poor coverage, I need more time/money
- We have too much emphasis on testing — quality is something we need to build in
Then you'll find a group that isn't testing inputs or outputs. More time and money needed.

Get mad and remove any of these steps? You've just given the group a "get out of jail free" card if/when something goes wrong.

Cancer-free testing and quality

Testing cancer is one of the main reasons why peacetime software is a lengthy, organized expensive process … that produces disastrous results. Then, when you try to fix the problem so it doesn't happen next time — the cancer spreads. Current quality and testing methods are a cancer on software development, and there is only one known cure.

The cure is simple. It's documented. It's proven in practice, at many places over many years. But it's radical, and requires a shift to wartime software thinking. It requires shifting to global process optimization, and optimizing for speed. It takes less time and produces far superior results. Here are some of the key points:
- Forget "quality," whatever that is. Concentrate on testing.
- Move from correctness-based testing everywhere to change-based testing at just a couple key points.
- Move all testing from the lab to production.
- Move all testing from periodic to continuous.
There are a number of posts in this blog that spell this out. There are even books! The advance of software QA cancer is inexorable and unstoppable in environments that are friendly to it. There is only one cure: Wartime software and its associated methods.
December 8, 2015
The Fundamentals of Speed-optimized Software Development
How do you write software more quickly? How do you engage and win with Wartime Software? There is a set of techniques that enable dramatic speed-ups in producing effective, high-quality software, and they share a set of underlying principles. These fundamental principles of software development are practically never discussed, yet they are almost painfully simple. Understanding and applying them can help you adapt existing speed-optimized methods to your needs and evolve new ones.

Mainstream thinking about speed

Very few groups optimize for speed in software development. By far the most common factor that is optimized is expectations. Given that everyone involved in software knows about, believes in and practices expectation-based development, the options for speeding things up aren't great.

The most widely used method for speeding things up is simple: pressuring everyone involved to go faster. Do whatever it takes, darn it, we're going to bring this project in on time! What this amounts to is:
- People work harder, longer hours and extra days. Result: sloppiness, lack of care, increasing number of mistakes and omissions, greater risk.
- People squeeze or omit essential steps, particularly later in the project. Result: poor quality and even disasters encountered during roll-out.
- Fashionable new methods are introduced that re-arrange things or add steps. Result: see above.
The fundamentals of achieving speed

The fundamental principles of achieving speed in software are extremely simple: take fewer, more effective steps to the goal and eliminate overhead. When framed in all the stuff we know about software development, this can be abstract and difficult to apply in practice. But it's truly simple. Think about walking to a goal. The principles amount to:

Walk to the destination as directly as possible; avoid stopping and re-tracing your steps.

That's not so hard, is it?

I know, it is hard, because standard software development methods are worlds away from thinking about things in these terms. But that's because they're optimized for expectations and not for speed!

Replacing lots and lots with a little

Most of the steps in mainstream software development are there for a good reason. While concepts like project management may have been developed outside of software, they have been adapted to it. So if you're going to toss out some standard expectations-based activity, you'd better be sure there is speed-optimized activity that gets the same job done — at least as well, and with lots less time and effort.

With all the effort and work that's gone into software development methods over the years, there's not a lot of improvement to be made by tweaking this or enhancing that. Radical methods are required to accomplish radical change. So it's important to ignore all the intermediate steps and focus on the only goal that matters: customer satisfaction. Period. Everything else is a stand-in, an intermediary, or supposed to contribute to that end.

If this is the goal, one of the fundamental principles of Wartime Software, of optimizing for speed and results, is this:

DO globally optimize for the goal.

DO NOT sub-optimize for any intermediate goals.

Once you globally optimize for reaching your goal, then you realize that there is a HUGE amount of redundancy in the normal "rigorous" software development process. The redundancy isn't obvious or even visible because mostly it's many repetitions with slight variations, each confined in a silo, performed by different people in different environments using different tools. So it doesn't look redundant, when examined in isolation. But when viewed globally? MASSIVE redundancy, and an opportunity to replace lots and lots of overlapping stuff with a little bit, done right.

Conclusion

Wartime software is not achieved by doing a re-organization or tweaking a process. It's achieved by a total re-conceptualization of the optimal way to get from Point A to Point B in software. Once you understand the fundamental principle of global optimization, with the goal of eliminating overhead and redundancy, all the detailed speed-optimized techniques follow like Q.E.D.
December 1, 2015
The magic of block chain

There’s a shiny new toy on the block. It’s called block chain, i.e., the technology behind BitCoin. It’s going to solve lots of intransigent problems, the kind that have remained unsolved for years! Really! Everyone says so, from big banks to great investors to authoritative media with brainy journalists!

I love Bitcoin, and think that block chain is one of the cleverest technologies I’ve encountered in some time. But this level of enthusiasm is a bubble, and like all bubbles, it will burst. Meanwhile, loads of people are bubbling about all the problems worth billions, problems that have gone unsolved for years, that block chain will solve.

The stock transfer problem

Just for fun, I picked a clever 5 minute video produced by the Wall Street Journal that explains to us rubes in the back woods how BitCoin works, and how it will be used to solve the horrible problem of stock transfers. According to the authoritative folks at the WSJ, transferring stocks takes days, while Bitcoin makes it “nearly instantaneous.”

This is huge. Those authoritative WSJ folks, no doubt after consultation that was wide and deep in the world of Bitcoins, assure us that the result will be billions of dollars in savings.
Billions! And imagine, all you have to do is use BitCoin!

Little things

There are lots of things wrong with this picture. Let me just pick on a little thing. Bitcoin makes the transfer “nearly instantaneous.” Is that so? Because of all the computing the Bitcoin miners have to do to make the block chain secure, the actual waiting time varies, but a reasonable estimate ranges around 10 minutes. When you’ve gone to an ATM to withdraw cash, would you say getting the cash is “nearly instantaneous?” Well, no. You wait for the bills to count out. How long before the cash is in your hands? Ten seconds? Maybe twenty at the outside? What if it were a full minute? You would think the machine was broken. What if it were five minutes? You’d be sure the machine was broken, and you’d have left long before.

Why do people explaining things like this decide to change the facts on us? While I agree that, compared to “days,” ten minutes or so is a vast improvement, why exaggerate? What else have they got wrong, if they get non-crucial little things like this wrong?

Big things

There’s some magic that happens in the video. It’s not called out, but it’s clearly magic. And it has nothing to do with Bitcoin!

The magic is how the sender of the stock trade has all the data required for the transfer tied up in a neat little digital bundle, ready to send off to the block chain network – and equally magic, how the receiver of the stock trade is ready to receive said neat little digital bundle, unwrap it, store it, and be completely satisfied that it had received the stock.

Digitizing all the required information in a standard format all senders can send and all receivers can receive is the magic – and the genuinely hard-to-solve problem here. Why do things take days to settle? I don’t know, but the usual reason is that lots of departments are involved on both sides of the transaction, not everything is digital, and … it’s always been done that way!

Block chain is definitely a cool technology. But what it solves is not the problem here! The actual problem is completely unrelated to Bitcoin and block chain!

Once you’ve solved the genuinely hard problem – digitizing all required information in standard form on both ends and adapting all relevant systems to generate and accept it – there are loads of ways to transport the data from the sender to the receiver. There are even lots of peer-to-peer methods that could be used, thus avoiding all the hoo-haw of having your stock information stored many times in a distributed ledger all over the world. You could, for example, agree on a set of RESTful calls the sender could make on the receiver’s system that would work fine. The industry could set up a cooperative central place that used normal DBMS technology to make the transfer. There are loads of approaches, all of them viable, all of them faster than Bitcoin, and many of them no more expensive to implement.

Imagine you’re living in a house and you’ve bought a new one a couple of states away. You want to move all your stuff. As everyone knows, the hard part is packing and boxing up before you move, and unpacking after you’ve moved. That’s what takes the time and care, and that’s what Bitcoin-based solutions airily assume. In the real world, once you’ve packed everything, you load it into a truck, which drives to your new house, unload all the boxes, and then the “fun” of unpacking begins. In the Bitcoin world, you would still have to go through all the time and work of packing. Then you would call trucks (still!) and send your boxes to the various depots of the distributed moving service, which would move your boxes all over the place, let them be examined by loads of people, and put them on their shelves with exquisite care. Then you would … get ready … send your own trucks to the depot, pick up your boxes, move them to your house, and start the fun of unpacking. This added step in the middle would take at least 1,000 times longer than the moving van would have.

Isn’t Bitcoin great? See how it solves the moving problem so nicely?

There’s a pattern here

This misunderstanding of a technology and claiming it solves all sorts of long-standing problems is typical of the new technology hype cycle. It’s happened loads of times before, and will keep happening for the foreseeable future. A good current example is the Big Data mania, which is supposed to solve all sorts of long-standing problems and unlock the keys to the kingdom – except that it doesn’t. When people have trouble understanding the basics of relatively simple things like email, is it surprising that they get Bitcoin wrong?

Conclusion

I’m eagerly awaiting problems for which the super-cool block chain technology is actually relevant. I suspect they're out there. I've even started looking at a couple candidates. I’m ready.

November 19, 2015
Uber’s Evolution from the inside

Uber is a great success; you can tell because competition has sprung from all corners, and scads of new companies say they’re going to be “the Uber of X.” Along with great success comes great myths, which inform a set of assumptions about how Uber got to be Uber. The myths are all derivatives of the broadly accepted narrative of how tech companies start and grow. In every case I’ve examined, fact-based history contradicts these myths and explodes what “everyone knows” about Uber and every other tech company. I’ve explored this theme in a couple of cases, and I’ve just had the opportunity to question an early Uber driver, whose story provides fascinating insight on just a small part of what really happened.

Recruiting the first drivers

My driver was one of the first handful of drivers in the New York City area. Uber is a network-effect company, i.e., customers need access to a large pool of drivers, while drivers need business from a large pool of customers. How do you get that started? Of course I don’t know the whole story, but I did hear how my driver informant was recruited.

He was sitting in his car near Madison Square Park in Manhattan with a half hour to go before his ride would appear. Someone knocked on his window. He rolled it down, and it was two people who wanted to talk with him about a new way to get rides. It started to rain, and he invited the people to sit in his car, for which they were very grateful. He ended up signing up for their service, encouraged by a $1,000 signing bonus, and further encouraged by being paid $35 an hour just for keeping the app on his phone. He was then paid $1,000 for each additional driver he referred who signed up, and a further $1,000 when that driver took their first ride. That’s how a critical mass of drivers was recruited even when there was little work for them to do.

Here’s the part of the story I found most impressive: the two people who recruited him were … the head of Uber NYC and Travis Kalanich, the co-founder and CEO of Uber.
Yes, the big boss himself, personally walking the streets of NYC and hitting on strangers. Wow. For every 1,000 bosses who puts up one of those stupid pyramid-on-its-head charts
during a company talk to say how he’s just there to serve the important people on the front lines, there might be a couple like Mr. Kalanich who act on it and get into the trenches themselves.

Recruiting the first customers

Uber today is an easy way to use an app to call for an on-demand car service. It’s all “get me a ride now. As in, now.” At the start, it wasn’t that way at all! It was mostly for really long rides, like from NYC to Boston or Philadelphia, and the rides were often round trips with a big wait in the middle. This is a tiny minority of all rides, but one that was typically arranged way in advance by phone. With Uber, you could get someone to pick you up in half an hour or less, which was great compared to the alternatives. It also meant that having just dozens of drivers available was OK.

It was only as the population of drivers and customers grew that shorter trips with shorter lead times made sense – and the long-trip buyers were an ideal starting population as customers for that, so it evolved naturally.

Evolution of the service

Here are some tidbits about how the service has evolved.

At the beginning, drivers were screened thoroughly. For example, every candidate was shown a map of Manhattan with no street names. A pin was placed on the map, and the driver had to name the streets. If you didn’t get them, you were out. The screening has gone way down over time.

The cut taken by Uber has ratcheted up sharply. It started at just 10%, moved in a few steps and is now up to 38%. A huge difference.

As the number of drivers and customers grew, Uber provided a “heat map” in the driver’s app to show where drivers were needed and where there was an excess, which helped drivers position their cars to best effect. A subsequent version of the app dropped this feature and replaced it with congestion pricing. My driver misses it, and thinks a new version is coming out with the heat map restored.

The policies and standards for drivers have evolved quite a bit, something which is also visible to customers with the addition of Uber-X and evaluations of drivers – I learned that drivers can also evaluate customers! My informant says the evaluations are on the whole helpful – one incident reminded him he always has to inspect the passenger seat when someone leaves, for example, something he had been neglecting.

In spite of all the changes, my driver/informant loves Uber. The main reason is he has control. He can start and stop any time. He can work more and make more money, or take time off any time he needs to, without asking anyone’s permission.

Observations

It appears that Uber did a very smart thing I have seen other companies do: pick a sub-market in which they could get traction, and then evolve into the actual, larger market they probably had in mind all along. The sub-market enabled them to get to a critical mass of drivers and riders with less friction.

It's easy to focus on Uber as enabled by an app, or Uber as a marketplace. Those things are true. I think of them as being good choices (the app) and reasonable thoughts (marketplace), but somehow not essential.

For the consumer, Uber provides the convenience of hailing a cab (no advance planning, scheduling, committing) with cars that you can't see. It would work just as well if you used your phone to dial a number, gave your location, and then were told how long you'd have to wait for the driver. An on-demand car service.

For the driver, Uber provides control over time and money, as my informant made clear. When you drive a cab, you sign up for a shift. With a car service, someone schedules you. With Uber, the driver can opt in and opt out whenever it meets his needs, whatever those may be. Whether he got rides from an app or from someone calling him, the result would be the same.

Conclusion

Every company has a different story. Uber is no different. Even a narrow and limited glimpse into the factual history of Uber reminds us that every company ends up having a myth-based story that explains its success, and that if you’re going to learn something useful from their success, you have to find and pay attention to the facts. AND, the facts of similar companies that failed.

November 11, 2015
What E-mail teaches us about Bitcoin and Block Chain
E-mail is widely used, and everyone knows what it is. Bitcoin is a hot new techno-bauble, and Bitcoin technologies like block chain are getting lots of attention and money. It turns out that e-mail has a great deal to teach us about Bitcoin and its technologies. Here’s the punch line: in spite of its ubiquity, practically no one understands how e-mail works, and this causes huge errors with practical consequences! By comparison, Bitcoin and its spawn are incredibly complicated; most of the people who do understand e-mail have little chance of understanding Bitcoin. Think about the consequences of this, please.

Do You Know How E-mail works?

E-mail is simple, right? You login to your e-mail account, fill out the To and Subject fields, maybe add a couple people in the CC field, write your e-mail, and press send. Then some magic happens, and the e-mail shows up in the in-boxes of the people to whom you sent it. You can read your own e-mail by looking at the items in your in-box, and even go to your sent-mail folder and look at what you sent. It’s simple, wonderful and true! For the vast majority of the time, it’s fine to leave “then some magic happens” alone.

The trouble comes when trouble comes, i.e., when there’s some special circumstance that requires knowing something about how that “magic” in the middle works. That’s when it comes out that almost no one has a clue about what’s going on, even in something as simple and ubiquitous as e-mail.

The IRS e-mail case

There are lots of examples, but the issues involving e-mail at the IRS which have been in the news off and on for the last couple of years are a good case in point. Here’s the lead paragraph from Wikipedia on the subject:

Now, remember – I’m not talking about the merits of the issue on one side or the other. I’m solely talking about the knowledge exhibited of how e-mail works, and the practical consequences of that knowledge. Read this juicy lead from an AP story on the subject:

Here are the key points:
- In June 2011, Lois Lerner’s computer crashed.
- This resulted “in the loss of records”
- It was determined that the records on the hard drive, i.e., Lois Lerner's emails, were gone forever
I am aghast. Agog. At a loss for words. I’d like to be shocked at the “depth” of misunderstanding, but I think it’s more appropriate to be shocked at the “shallowness” of misunderstanding exhibited in this quote, and in the heads of all the IRS employees, FBI, Congressional staffers, the archivists, and all the journalists with their fancy degrees from fancy schools.

Here is the core concept that everyone involved on every side seems to agree on:

The e-mails Lois Lerner wrote are uniquely stored on the hard drive of her personal computer. If it is true that the hard drive is severely damaged, then the e-mails are “gone forever.”

The simple thing

Even from the simplistic view of how e-mail works, every e-mail is either a draft or is sent to someone. If it's been given an accurate address, it arrives. It's in the receiver's in-box, and perhaps eventually in their deleted mail folder. Since the issue involved e-mails not only received by Ms. Lerner, but ones sent by her, presumably to other IRS employees, there is an obvious strategy: do a search on the e-mail of every IRS employee to whom Ms. Lerner could have sent an e-mail, and see if she did send one. It's the magic of e-mail: the sender has a copy of what was sent, and the recipient has a copy of what was received. There are at least two copies: both sender and receiver have one!

Have you ever read that simple thought anywhere else? Neither have I.

The "deep" thing, requiring understanding of how it works

Now we get to the real point. An e-mail address has two main parts: the name, and the domain. The name is the part before the @ and the domain is the part after the @, for example Lois@IRS.gov. Similarly, all e-mail systems have two main pieces of software involved: a client and a server. Software by Microsoft is widely used in governments and corporations. Outlook is the client software, which runs on the computer on which you read and write e-mails. Exchange is the server software, which runs in a data center somewhere. Exchange is a program with a database holding the e-mails, address books and calendars for a whole bunch of users. A domain like IRS.gov is implemented with many Exchange servers, each with the e-mails of a particular collection of IRS workers, typically a couple for each physical location.

When Ms. Lerner wrote an e-mail, she used her computer running an e-mail client such an Outlook. When she hit the Send button, the e-mail immediately went to her Exchange server, which filed it away. It then found the Exchange server(s) of the recipient(s) and passed the e-mail to it (them), which it turn sent it to the user's Outlook clients. Shortly after Ms. Lerner sent an e-mail to her colleague Mr. Lowe, it was stored in no less than four places, including a couple servers. In addition, assuming the government had at least moderately responsible Exchange administration, the e-mails were further copied to replicas, on and off-site, and in addition periodically backed up to yet another medium and location.

There are other e-mail clients and other e-mail servers. I have no information about what the IRS actually used. But this is how e-mail works! There are clients. There are servers, which serve a number of users/clients. When a human writes an e-mail, it goes from her client to her server to the recipient's server to the recipient's client. As as result, it should have made no difference whatsoever that Ms. Lerner's computer "crashed." It wouldn't matter if it suddenly grew wings and flew off to Tahiti to frolic in the waves. Any e-mails that Ms. Lerner wrote were securely stored on her e-mail server shared with other users and in a data center, and on multiple replicas, backups and disaster recovery sites.

The fact that Ms. Lerner's computer crashed and people supposedly spent time attempting to recover e-mails from it, and when they failed, declared them "lost forever," and the fact that everyone else involved, including journalists and commentators and experts of all sorts, accepted that as the state of affairs ("well, if her hard disk crashed, what can you do, ya know?"), demonstrates that none of them has a clue about how e-mail works. It's like not knowing that cars have engines. It's that bad.

What e-mails have to do with Bitcoin and Block Chain

Compared to many other computer technologies, e-mail is simple. Compared to many other computer technologies, Bitcoin is complex. Even worse, what's interesting about Bitcoin isn't Bitcoin the crypto-currency — it's the block chain technology on which it's implemented. Block chain is getting all sorts of attention from financial technology people and investors. I won't review it here, but a brief look at the action will convince you it's frothy.

What if investors, financial industry executives and Bitcoin technology company leaders are as informed about block chain as everyone involved was/is about e-mail? What if they're making important decisions based on critical observations as sound as "well, the hard drive is kaput, so the e-mail is gone, and that's that?" If the understanding of important actors in the e-mail drama exhibit paper-thin understanding and wrong-headed conclusions, are we to understand that all the folks involved in Bitcoin and block chain are geniuses by comparison?

Place your bets, people. I know what I'm betting on.
November 3, 2015
Emails and Customer Respect: HP again

In an earlier post, I complained about software quality at big companies, illustrated by how they get the "little" things wrong, which of course sometimes builds up into getting really big things wrong.

HP is definitely on a roll in that regard. Just to pick an example, HP decided to buy Autonomy for $11.7 Billion in 2011, and about a year later, was forced to write off $8.8 Billion. In that context, little things like email marketing practices are trivial, things of no concern at all.

But that's the point! Quality starts at the top, it starts at the bottom, and attitudes about it pervade an organization. If no one is particularly concerned about quality, then stuff happens. And seems to be happening at HP (among others).

The importance of email

Some people may think that email is a tiny little unimportant thing. That's odd, because email is an important way that companies communicate with and interact with their customers. The fact that a company's many divisions may send out millions of emails tempts people inside the company to think how insignificant emails are. We send them out by the million!

But in the experience of any one customer, email may have a large role to play in forming their image of the company. Does it make good products that I want to buy? Does it listen to me? Does it respect my views and respond to my requests? When it makes an offer and I respond, what is the company's follow-up?

Email is like a salesperson, or a customer service representative, having an important role to play in the customer's future relationship with the company.

HP and email

Someone in power at HP badly needs to read this. The way HP currently handles their email is a sad example of dissing current and potential customers.

I recently received a couple more spam emails from HP, after many diligent attempts to follow their rules for unsubscribing.

Here's the bottom of the email:

Sounds good, huh? HP doesn't just protect my privacy, they're committed to it. I can choose (hah!) whether HP may communicate with me!

So, yet again, I clicked here to unsubscribe. This time HP put up a new hurdle.

They know what my email address is. All they have to do is what every responsible e-mail marketer does and put it in the unsubscribe URL in order to deliver what customers want, which is one-click unsubscribe. Not exactly a new idea!

But HP, that organization capable of rebuffing double-digit numbers of pleas from me to unsubscribe, has decided to up the ante. Now they demand that I type in my email address! Which they know! Just to show who's boss, and to show how much they respect me, and to hope that maybe I won't bother to type it in, forcing them to find another excuse to keep pounding me with spam.

The importance of email in customer relations and branding

An email interaction with a customer is like a sales person's interaction with a customer, i.e., an opportunity to show the company at its best, to make a good impression and to act in a way that inclines the customer to spend money buying the company's products and services, now or in the future. If the customer says "no, I'm not interested right now," the effective sales person bows out in a way that maximizes the chances that, in the future when the customer has a need, they will be favorably inclined to the company.

What if the sales person says "I really respect you and I respond to your needs, but if you want me to leave your office, you have to write on this piece of paper your exact name and job title. If what you write fails to match the information I have in any way, even a single letter, I will continue to walk into your office uninvited and demand your time and attention." That's the physical equivalent of the email interaction I had with HP. How do you think this would work out if physical sales people did it? Is there any reason to believe that the quality of the reaction is different when the HP representative is email?

More and even worse

HP has just announced that it's splitting into two companies. Naturally, having ignored all my requests to be dropped from their email lists, HP emailed me about it. Not just some division; central, home-office HP. Here's what I received:

Note that the subject of the email is "separation information pertaining to email subscription." Thus, the home office of HP is still convinced I'm an email subscriber. And this email is about being a legitimate, opted-in subscriber to HP email.

Naturally, I was curious to see how they'd handle the unsubscribe request. I clicked on the link above and got … nothing! For several hours I periodically tried, and the link led to a site that simply didn't respond. The next day I tried again, and got this page:

Again, I got that worst-practice of email unsubscription methods, the "we're making believe we don't have the email address that you clicked on to get here, so you have to guess to which of your potentially several email addresses we sent this and enter it correctly, otherwise we deem you insufficiently skilled to deserve being excused from being spammed by HP, and we will continue our periodic spam until you get it together."

That and a lawyerly "privacy" policy and five bucks will get you a coffee at the nearest Starbucks.

To all companies in the world anywhere that send email: take this as an example of what not to do. Your present and potential future customers sincerely request it, and will respond accordingly.

October 28, 2015
Big Data and Data Warehouses

There’s a rapidly growing movement to take all the data that’s scattered throughout an organization, rationalize it, bring it together, and make it available for analytics that will help management to understand and ultimately to transform the business.

This movement is taking place today. It’s explosive. It’s called Big Data and Analytics.

This movement also started in the 1970’s, took root in the 1980’s, exploded in the 1990’s and is with us today. It’s called Data Warehousing.

The fact that attention is slipping away from the thing called “data warehouse” and moving towards “Big Data” is a typical IT industry phenomenon. The problems are the same, the obstacles are the same and the solutions are the nearly the same – but the rhetoric and software are entirely different. Only a few savvy industry insiders are aware of the game that’s being played.

The Enterprise Data Warehouse

The Enterprise Data Warehouse, EDW, is the industry’s holy grail. It’s the place where all an organization’s data is stored for reading and analysis. All the data from the various operational and transaction databases is extracted, transformed as required and loaded into this database. Once there, it provides a “single source of truth” for the enterprise. Since the EDW is not running transactions, reports and analytics can be run against it at will, without harming ongoing operations. Single-purpose extracts can easily be made from it to support various projects.

The EDW makes common sense. It became a major goal for many organizations, and many are still marching towards that goal. There’s just this one little problem: getting there. Then there’s a second problem: realizing the potential value.

There are lots and lots or organizations that don’t have to worry about having an EDW that fails to fulfill its promise, because they just get bogged down along the way and never really get there.

Why don’t you read much about this? Simple: who wants to admit it? And if the road to the EDW ends up trapping those marching down it in impassable mud, who outside the organization is ever going to know it?

There’s a simple little acronym in EDW that is the tip of the mud-trap in which EDW gets bogged: ETL, which means Extract, Transform and Load. That’s what you do to get the data from where it starts to where it needs to be, in the EDW. Simple, right? Oh, if only it were…

Extract, Transform and Load

Before you even get to the E of ETL, you have to find the data. Then you have to get access to the data, with a properly jealous operational management group anxious that you avoid screwing them up. You have to get the whole thing to start with, and then a stream of updates.

The “database” could be nearly anything. It could be a set of ISAM files running under CICS on an IBM mainframe. In which case, you need to get your hands on the source copy books that contain the data definitions to have any hope of making sense out of them.

It could be something nice and modern, like Oracle. But you’d better start by getting a full dump of the schemas to have any hope of navigating among what could be many hundreds of tables. Then, without an E-R diagram that’s up to date, you’ll have little chance of making sense out of the tables. Then when you get down to it, you may discover a world of stored procedures initiated by access triggers, so that your innocent “just let me read the tables” turns out to have side effects. And then, getting the updates? You’ll soon find yourself either crawling to the DBA and begging for a change log to be shipped to you, or pleading to be allowed to program in some trigger-initiated stored procedures yourself, so you can get the updates with killing the performance of the DBMS, and avoid getting set upon by a mob of angry users.

Phew! Now you’ve gotten through the E part of one source. What if there are dozens, or hundreds?

And then the real fun begins. The also-innocent-sounding T phase of ETL. Because T doesn’t just mean simple, no-big-deal transforms, it also means take the customer names that are represented in different ways in different places, some of which have been updated or changed independently of the others, and make it so that when you end up with a customer in the EDW, that customer represents all of your relations with exactly one customer. Having three customer rows in the EDW for one customer kind of defeats the purpose of the EDW, after all.

I’m just scratching the surface here, but perhaps you can get a feel for why the glowing promise of the Enterprise Data Warehouse so often ends up with the participants hungry and wounded in various ditches along the path to the promised land.

Big Data

Forget I ever mentioned “data warehouse”, ETL or any of that other stuff. Bzzzzzzttt! New subject!! Brand new!!! NOT related to anything else in computing, completely WITHOUT history, stemming from this brand-new EXPLOSION of data that’s just EVERYwhere. It’s the Big Data movement! Where we take all these mountains of data that are just piling up useless and turn them into business GOLD. You’re already late – everyone else is already with it. There are books, conferences, experts, the whole nine yards!

You’ve got to get a Data Lake, and fill it up with data. Then you’ve got to rev up your Hadoop cluster and start cranking out those nuggets of business gold from all that data.

Except, hmmm. I’ve got to find the data. Get access to it. Get it once and then get a feed of the updates. All this data from different places, it doesn’t match up well, I’ve got to clean it up. Well, maybe I’ll just dump it into the Data Lake and let the Hadoop nerds worry about it. They’ve got all these servers at their disposal, maybe the servers can work at night cleaning everything up.

Gulp. I just looked at my nice, fresh, clean Data Lake. It’s a Data Swamp! There are snapping turtles and water moccasins swimming in there. Don’t. Like. This. Maybe I can get a transfer.

Conclusion

If “data warehousing” were a big success, it would have kept its name and would now handle what we now call “big data.” But no. “Data warehousing” projects are often classic IT projects that drift on forever, confronting obstacles and rarely producing results. Big Data is the new kid on the block. There aren’t (yet) decades of frustration and broken promises associated with it. Give it time. Every obstacle that DW projects encounter also rear up to challenge Big Data projects, and until solutions are found, returns will be equally elusive. And even then, there are conceptual flaws in most Big Data efforts.

October 19, 2015
Building Better Software More Quickly

How can you build software more quickly, without just cutting corners, taking risks and turning out crap? The prevailing view in the industry is that building software is a tough job, and it just plain takes a long time. Different factions heavily promote methods that are supposedly better, but in the end, the fancy new methods are either minor variations on standard practice or make things worse.

We don't need to speculate to decide that great software can be built much more quickly than nearly everyone thinks — there are people who are doing it today. I've written about this extensively. But in most environments, including government contracting and large corporations, the speed methods are forbidden. Yes, forbidden, as unthinkable as trying to walk in the office barefoot. The people who get it done don't just add some secret sauce to the standard recipes — they have whole new ways of framing the problem.

Getting from the start to the finish

Most programming jobs are like getting from Point A (where your software is today) to Point B (where you'd like it to be).
All the widely accepted methods for getting from Point A to Point B take what they sincerely believe to be the shortest path.

Standard ways to travel

There are standard, authorized, certified and audited ways to make the trip. Some people feel passionately that their way is better than the others, but they all take a long time, and there are frequent wrecks on the road. The journey is always an arduous one, and has many phases, each requiring specialists with lots of experience and training to get the job done while avoiding disaster. Which nonetheless happens often enough. Even when you make it safely to the destination, all too often you either arrive late or without all the stuff you expected to arrive with.

Everyone agrees that there are lots of steps in getting software done, steps that require education, experience and skill of different kinds, and each step builds on successful completion of earlier steps.

In the early stages of the trip, you do lots of planning, because you know how tough trips like this tend to be. You go back and forth over nearly the same ground, never making lots of progress. It’s climbing a mountain, and the switchbacks are just a necessary part of what you have to do to get to the peak. Once you’re there, you can finally see your goal at the bottom of the mountain. There’s a clear path to get there. The path continues to have lots of risks, but at least you have a definite plan, you’re know exactly where you’re going (you can see it!) and how you’re going to get there. It feels more like you’re going down the mountain.

When you’ve got point B in sight, it’s important to move deliberately and carefully. If you skip steps, you can easily wreck everything and fall off a cliff. But still, it’s easier going because everything is spelled out and written down. You’ve brought new members onto the team, specialists who have gone down mountains like this many times.

Even though you're going in a straight line, it turns out it's not a straight line on level ground! It's a straight line that goes over a very high mountain, one with vertical walls, gaps and all sort of progress-killing obstacles. There are all sorts of people promoting ways of getting to point B that are supposedly faster or less error-prone. But in the end, you still have to climb the mountain and then go down to the destination. All the same work ends up getting done, but just in different combinations and arrangements and orders.

Speed-optimizing methods

There are a couple wacko’s out there who talk about nothing but “speed.” Right. Going faster is definitely the fastest way to end up in the ditch.

Yes, that's what nearly everyone thinks, and especially people with experience, people who have been part of or been subjected to the wide variety of disasters caused by train-wreck software development efforts (oops, sorry, I'm being redundant here).

When you look at what the people who get things done quickly actually do, you find that they reject the whole paradigm of project-management-oriented software development. They think about the whole process in different terms. There's still a Point B, but instead of accepting the common delusion that you can figure out how comfortable a bed is before you lie on it, they realize that you have to get close to where Point B probably is, and then look around and refine it, maybe by quite a bit! Maybe Point B sounded wonderful based on the speculate reports received while standing on Point A, but maybe once you're in the general region of Point B, nearby Point B1 is clearly superior. So you shift to it. Look around. Figure something else out, and shift to Point B2. Wow, how great is Point B2 — let's build some comfy chairs and a roof so we can stay here a while — but be sure to build the roof this way, and with a skylight over there, which is pretty obvious while sitting on a camp chair in the right place, but simply couldn't have been known way over there at Point A.

Conclusion

The speedy programmers who want to get to a destination find ways to avoid making arduous, risky climbs over mountains; instead they zoom to the general place quickly, learn stuff and refine. The detailed methods that speed-oriented developers use can seem mysterious, until you see that they're looking at things from an entirely different point of view.

October 13, 2015
Secrets of Software Super-Developers

Outstanding software development groups are truly outstanding. In many sporting events, the winner is often determined by a margin of a fraction of a percent. In software, the best teams are at least 10X more productive than the competition, and sometimes 100X. Here's the most amazing thing of all: the way the super teams do it is not secret!

Discovering the secrets

For most of my software development career I was only vaguely aware of productivity differences. My job was rarely to judge and evaluate — it was to get stuff done, quickly and well. I figured out some basic stuff to the extent it helped me get stuff done, and hired people who would contribute. I quickly realized that education was irrelevant, as was experience in a particular technology or tool set. What mattered most was raw horsepower, ability to concentrate, willingness to work and the drive to learn.

I think my first wake-up call was a consulting project I did for Sallie Mae during the early 1990's, when I was one of three groups advising the company on a massive re-engineering project using document imaging and workflow technologies. One of the groups was me; the second was a local group with 5 people on the project full time; the third was a national consulting firm that had at least 15 people on the project full time with various senior people and "experts" floating around. I was the only person involved with any of the groups who had not only used the technologies, but created them.

To make a long story short, the other groups performed standard analysis and vendor evaluation, and proposed an expensive process that would result in a 30% productivity gain, if all went well. I dove into the actual work Sallie Mae was doing, noticed some things about the work, and proposed an inexpensive process that would result in a 5X productivity gain. I called the method I used "zero-based re-engineering," combining two fashionable-at-the-time concepts, "zero-based budgeting" and "process re-engineering." They put it to the test and demonstrated that it worked! I tell the whole story here. Did it catch on? Did the dramatic advantages of the approach lead to widespread use? Nope.

It took years for me to figure out that what I had experienced with Sallie was the tip of the iceberg, both with giant bureaucracies like Sallie Mae and with the "technology expert" consulting firms.

It's developer's gold — but you can't give it away

About ten years later (yes, I know, I'm slow), I started noticing similar stark contrasts among groups of technology companies and the way they built software. The things I noticed were at heart simple — but compared to existing practice, radical. And I now finally realize, revolutionary. Man, I'm really getting the formula for the secret sauce here! I've got to be really careful, and make sure I only whisper these secrets to our portfolio companies. I can't let them get out; if one of our company's competitors got wind of the methods and rolled them out before we did, we'd get killed.

So for the first couple of years, I was really careful with these "secrets." It was only spoken, never written, and only on a need-to-know basis. Some of the companies I would visit wouldn't say they had "secrets" — but they did things differently from most groups, and I eagerly learned from then. I'd visit other companies which could really benefit by employing one or more of these "secrets." I whispered to the smart, motivated people who are leaders in companies in which we've invested where the gold is — and it's really close, right over there!! — and they're ho-hum, thanks a lot, so glad you let me know, please tell me when you plan to visit again, good bye. In other cases, there was apparent enthusiasm, followed by nothing. I thought the problem might be understanding. So I followed up the verbal with e-mails to solidify the ideas.

Learning more and revealing the secrets

Meanwhile, I'm getting a broader knowledge of how the super-developers get it done. It appears there's a spectrum of techniques. Some people use lots of one and little of another, but they all optimize for speed and quality instead of expectations.

I did notice that putting the ideas in writing helped. So I graduated from over-long emails to PDF papers. I told people the ideas, then gave them the papers. Or the other way round. Having the papers seemed to increase the chances that our companies would take the ideas seriously. I kept augmenting and editing the papers as I learned things, and peppered them with real-life examples. Some of the papers went through a couple dozen editions. By about 5 years ago, I had over ten papers comprising over 800 pages.

Why won't people grab the gold?

As I saw more examples of great results, I became ever-more mystified about the laggards who just don't get it.

Now, to be clear, these "laggards" are so only in relative terms. Most of them are way smarter and more accomplished than their compatriots are in the giant legacy companies. In the eyes of the world, the vast majority are leading edge. I started noticing that what I saw as laggards were nearly universally practicing methods that the software industry as a whole has labelled "leading edge," things like unit testing, Extreme programming or Agile. So on a bell curve of established practices, they were definitely leading edge.

My dim light bulb finally started flickering on. Almost no one decides important things like software methods on their own! They pick whatever brand of recognized external authority they're most comfortable with and which best matches their self-image, and try to follow the established method.

Understanding the resistance

But these are programmers, darn it! The methods aren't exotic, requiring a thorough grounding in quantum string theory — in most cases, they're simpler than the standard stuff. So where does the reluctance come from?

This is a BIG subject. There are lots of aspects of it, and I'll return to the subject in future posts. But the bottom line is simple: if programmers want to advance in their careers, for the most part they have to accede to the demands of non-technical managers. The non-technical managers have no clue what programming is about, except that it's full of incomprehensible gobbledygook. Understandably, they want to feel confident of what's going on in the department they're in charge of, most of whose people are engaged in activities that are opaque to them. So they absolutely require the use of standard methods, measures and goals that are entirely consistent with other disciplines in which things get built, like buildings and cars. Anything else feels like a "lack of discipline."

Most of the "secret" methods did not follow the "makes managers comfortable" formula. So the vast majority of organizations would reject their use, regardless of their provable power and effectiveness. For me, keeping things secret was not the problem — it was getting the ideas acted on. There are indeed amazing methods used by software super-developers to get things done 10X or more better than anyone else. But because the methods are not among those talked about by the public "experts" in the industry and do not resemble how houses or cars are built, they aren't a candidate for being used by the vast majority of developers, who do not make decisions for themselves based on substance.

Going fully public

That's when I decided there was no harm and possible benefit in going fully public with the things I'd been noticing over the years. That's when I decided to fix up the papers, turn them into books, and publish them in paperback and on Kindle. Here is a little more background, some illustrative blog posts and links to the books that I've already published.

I am working on book versions of the remaining papers, some of which are more revolutionary than what I've already published. Going through the process of crystallizing what I've learned over the years has taught me a great deal about knowledge itself and innovation, since part of what I've done is observe dramatic advances and seen how the knowledge is acquired and spread.

October 6, 2015
Software Quality at Big Companies: United, HP and Google

I would love to avoid the issue of software quality — but it keeps finding me and biting me, as I'm innocently going about my business. I guess you can understand that there are issues at a giant company whose main business is flying airplanes. It gets more annoying when the company says it makes computers. It's even worse when it's an incredibly well-regarded software company. Here are just a couple personal examples. They seem small. But they're indicative of a pattern in practice.

United Airlines

I fly on United airlines a fair amount. I needed information about one of their flights, and not even on a day when a computer systems failure brought everything there to a halt. Just a regular day after they released some new software, software that no customer was pounding their fist on a table demanding — just regular old new software they felt compelled to release. Software that didn't work.

They point out that the new version isn't available — but neglect to point out that the old isn't available either! Sad. Pathetic. They put the effort into assuring that their error message would include an attractive picture of one of their planes flying — perhaps they could instead have put a bit more effort into keeping their software flying?

HP

This once-great company has been drifting for years. I'm amazed they still have as many customers as they do. Clearly some executive in some cushy suite is putting pressure on the marketing people to generate more leads. So I've been getting spam from HP like never before — yes, HP is "spamming" me.

Word has clearly also come down to keep the pressure on those recalcitrant would-be customers like me. So, like a nice, obedient spam target, I click the opt-out button at the bottom of the e-mail.
I have great expectations, because, after all, "HP respects your privacy." I go to the relevant page,

So I go to the form, and make sure the "unsubscribe all" box is checked before clicking the button.
Then, I get a re-assuring page saying it's all set, no more spam.

Everything is OK then, right, because HP respects me and everything about me; they say so.

Except: I've gone through this exact process or one similar to it ten times in the last month, and nothing changes! HP apparently is eager for me to receive their information, and they respect me as much as ever. Their software is broken and no one cares. Is this huge? No, of course not. But it's the small things that tell you what's really going on.

Google

For reasons that escape me, the general impression is that Google is great and everyone who works there is a genius. I get business plans telling me that everything is great with their software because they've hired a team from … Google! Case closed!

Except it's not, from big things to small. Here's a small personal example. I went onto Google+ (one of the many projects/services that is rarely on the short list of great Google achievements) to get my posts. Here's what I got:

I tried and tried. No luck that day.

Can you imagine something being down for a day? The recent American Airlines system outage that I had the pleasure to personally experience while caught in a system-wide ground halt lasted a couple hours. In that context, it's a good thing that Google+ is nothing but a free service for helping people waste time.

Conclusion

Software quality is a huge, on-going, unsolved (at most organizations) problem. There are ways to solve it. The overwhelming majority of practicing professionals and computer science academics prefer to ignore it. Meanwhile, the rest of us get the message loud and clear: we don't matter to them, and words to the contrary are nothing but propaganda.

September 29, 2015
The Building Better Software Better Books

I've been releasing the Building Better Software Better series of books for the last 3 years. Four are currently available. Another will be released in early 2016, and two more will follow. I'd like to briefly describe the origins and content of the books.

Writing lots of software

My first job, at 16, was pressing sweaters in a knitting mill during the summer for minimum wage. It was a motivating experience. Later in high school, I started programming computers. Before graduating with honors from Harvard College, I wrote code for oil refinery optimization and the ARPA-net. I then wrote code for compilers, composition systems, operating systems, DBMS internals and applications, large scale financial transaction processing, document processing, workflow and more. You can find details in my LinkedIn profile.

I was hands-on at each job, writing lots of code. I was often dragged into management, but struggled to rise to the level of mediocrity. I did fine interacting with customers and creating marketing stuff. And of course writing code.

Being on the Investor Side

I started working in the technology side of venture capital in the early 1990's. After helping to generate good returns for the fund, I became a GP at Oak Investment Partners in 2000, where I worked to improve the outcomes of our computer-based investments, from infrastructure and tools to the consumer internet. I saw lots of companies we didn't invest in, and of course spent lots of time with the ones we did. I concentrated on health care and financial technology investments with the Oak HC/FT fund, where I was a GP.

Like everyone else, I wanted our investments to succeed. But unlike most everyone else, I dove into the products and services, the code behind them, and the people who built the code. Sometimes the code made little difference. But other times, it really mattered. Regardless of its impact, I got to see how things went with the software and the business, sometimes over a period of many years.

Noticing the patterns

As I matched my own experience with that of the companies we invested in, I started to notice patterns. I first discussed the things I noticed with people in the companies. I found that writing them down was useful, so I put them into long e-mails. As the e-mails grew, they evolved into private-circulation papers. I made many additions and corrections based on feedback and new experience. I started posting about the ideas at my www.blackliszt.com blog. I'm finally releasing the material as the Building Better Software Better series of books.

The book series

Building Better Software Better describes how winning software people actually do what they do. It is often contrary to widespread practice, what is believed by many professionals, and what is taught in computer science and business school. In all too many organizations, failure to build the right software, quickly and well, is a major impediment to business success. Using even some of the methods described in these books can make software into a major driver of business success.

Wartime Software

Building bridges for civilian use takes years and thousands of people. Building bridges in wartime takes hours – and it’s done under enemy fire, and the load and quality requirements are higher. Most software is built the way bridges are in peacetime. Some groups screw up building bridges in peacetime, and their bridges fail, typically because they don't follow the established methods. Some groups — those who are “under fire” or just don’t have time or money to do it the “right way” — build software using wartime rules – software that is better than peacetime software. They do this not by cutting corners, but by following a different set of rules and methods.

Best to start: https://blackliszt.com/2012/03/bridges-and-software-in-peace-and-war.html

               https://blackliszt.com/2014/02/lessons-for-software-from-the-history-of-scurvy.html

General posts: https://blackliszt.com/warfare/

Book: http://www.amazon.com/Wartime-Software-Building-Matters-Better-ebook/dp/B00CUGHDT8/

Project management

The title is the Disease of Software Project Management, because it has in fact been a disaster for software. The idea that software development should be managed by project management techniques is accepted nearly universally — the only disputes are about which of the many minor variations to choose. This book is an extensive deconstruction of project management, its origins and its application to software development. If phrases like "that will probably take us three sprints to get done" spring from your lips without sarcasm, you need to read this book. The core idea is that all forms of project management optimize for expectations, while in wartime you optimize for speed.

Best to start: https://blackliszt.com/2010/10/software-project-management-dates-are-evil.html

               https://blackliszt.com/2013/03/software-comparing-waterfall-and-agile.html

               https://blackliszt.com/2012/10/software-project-management-book.html

General posts: https://blackliszt.com/project-management/

Book: http://www.amazon.com/Disease-Software-Project-Management-Disaster-ebook/dp/B009RQ6PUC/

Software QA

Things are done faster in wartime, but that's only possible if they're done differently. QA is one of the key areas of difference. There are core ideas to understand, and detailed methods that are different, involving different skill sets than are normally involved in QA.

Best to start: https://blackliszt.com/2012/04/a-simple-framework-for-software-quality-assurance.html

               https://blackliszt.com/2012/10/software-quality-assurance-book.html

General posts: https://blackliszt.com/software-quality/

Book: http://www.amazon.com/Software-Quality-Assurance-Building-Better-ebook/dp/B009TAMNHA/

People

Managers tend to apply the same templates to managing software people that they apply to everyone else. They foolishly think they can manage people who are doing something they don't understand and can't even see. This book covers some of the things that are unique to software people.

Best to start: https://blackliszt.com/2014/10/joe-torre-and-software-development.html

               https://blackliszt.com/2015/03/software-people-book-just-published.html

               https://blackliszt.com/2014/01/who-makes-the-software-decisions.html

               https://blackliszt.com/2013/11/people-and-substance-in-software-management.html

General posts: https://blackliszt.com/people/

Book: http://www.amazon.com/Software-People-Human-Building-Better-ebook/dp/B00VDIFP3U/

Future books

Three more books are in the works. The next one, Software Business and Project Strategy, will be published in 2016.

September 23, 2015
The Data Center of the Future

The elements of the data center of the future are mostly available today. Everyone is pretty used to the data centers they've been using for a while, which are thoroughly grounded in the past. So they keep building new copies of hoary architectures. But the parts and ideas are available to anyone who chooses to avail themselves…

Ancient Computing History

Back during the first internet bubble, say around the year 2000, data centers would likely have lots of Intel Pentium Pro microprocessor chips in their servers. It was an amazing device at the time. It had over 5 million transistors on the chip. It was so powerful that it was used in the first supercomputer to reach the teraFLOPS performance mark. Pretty amazing.

But building applications to support internet-scale applications was still hard. The clever software engineers of the time worked out ways to distribute the work among a collection of computers to get the job done, quickly and reliably. It was called, appropriately, "distributed computing."

Ancient Storage History

Back in the halcyon days before the internet, disks were just hooked up to the computers whose storage they maintained.

In internet-scale data centers, that wasn't good enough. There was always too much storage where it wasn't needed, and not enough where it was. Those bright computer guys got another good idea: we already have a local area network for connecting servers to each other. How about a storage area network for connecting computers to storage?

Ka-ching! Problem solved!

Today

Things have moved along. The latest microcomputer chips from Intel, for example the Xeon Processor E7 v2 family, have grown from millions of transistors to … Billions of transistors. That's Billion, with a "B," as in 1,000 times more transistors than the number in the Pentium Pro line. Instead of a single, single-threaded core, there are 15 dual-threaded cores, a total of 30 effective processors, each awesomely faster than the single core in the Pentium Pro. And it support about 25 times more main memory.

Each server with a Xeon E7 can handle at least 30 times the workload of the Pentium Pro, maybe 100 times. Here's a picture of the evolution:

The average data center architecture? Unchanged. Still emphasizing distributed computing, LAN and SAN, all the stuff invented to solve a problem of limited computing power that has long-since disappeared.

The Future (for most people) Data Center

This data center can be yours in 2015 — if you chose to build it.

The core principle is simple: use the cores! And everything else that's there! Get rid of obsolete architectures, chief among them that of "distributed computing" (with just a couple of exceptions), and drastically reduce the parts count. Here's what it might look like:

Note all the cores in the big (in power, but physically small) server boxes — plenty of room for "distributed computing," inside a single chip. There are loads of cores; devoting a couple to handling storage functions eliminates boatloads of parts and connections, the whole SAN, without loss. You can easily arrange that most of the time, the storage is connected to the box the apps that use it run on. If not? No problem — just a single hop to the storage software on the right box, and you've got yourself a virtual SAN, faster and for less money.

Conclusion

The data center of the future is still in the future for most people. The parts are there. The concepts are there. But old habits appear to be hard to break…

August 18, 2015
Human-Implemented Cognitive Computing in Healthcare

I'm pretty skeptical about "cognitive computing." It's hard even for people to "cognitively compute." In fields like medicine, only the brightest, most educated and experienced people are capable of producing useful results. But when they do, the results can be powerful, useful and save lives.

Cognitive Computing

I've worked closely with computers since I was in my teens. I've seen their impact on society, and the huge productivity gains when properly applied. But ever since the early days, a subset of the computer industry and the public has insisted on seeing computers as versions of human brains. Periodically, the computer industry gets excited about how the latest computer hardware and software will enable computers to do things that only the smartest and most educated humans can do. When the excitement gets frothy, the movement is called something new, so no one will be "confused," and think it's the same as all the earlier, essentially identical movements that have failed and quietly faded away. The latest such movement is called "cognitive computing."

Medicine

Doctors who specialize in a field of study both contribute to advancing the state of the art and stay on top of advances made by others. Action-oriented review articles are particularly valuable — they both summarize advances made by many people, and advance clinical practice by making those advances practical and actionable.

I'll take an example from emergency medicine, this journal in particular.

One of the articles that appeared in that journal earlier this year was about Horner's Syndrome in children.

There is lots of fascinating and useful information in the article. It's just amazing the things that go on in the human body. Knowing about all the things that sometimes go wrong makes it all the more impressive that most bodies work so well most of the time!

Here's a chart in the article that boils it all down. You've got a child presenting with Ptosis. What's going on here? There are a couple options, and the chart guides you to figuring it out.

One of the outcomes is Horner's Syndrome. What do you do next? Here's a chart that makes it all clear.

This example of human-implemented "cognitive computing" is similar in principle to many valuable intellectual and scientific results. It's the result of years of effort and study by many people handling many cases, and publishing the results. The advance here is boiling it all down and translating it into simple, unambiguous flowcharts that guide you to do the right thing, without leaving out anything important.

Can and should flowcharts like this be made available to front-line clinicians as they are seeing cases? Yes, of course, if only to enable the clinician to make sure something new hasn't emerged since last time he/she checked. Does it require fancy computing to make this happen? It does not.

Conclusion

Getting simple results like these is amazingly tough work for highly educated and motivated human beings. Meanwhile, computers are not yet capable of tying my shoes. When they are, I will gladly put them in the running for doing something more sophisticated and important, like picking who should have the lead role in the next James Bond movie. "Reading" the medical literature and figuring out how to respond when a child presents with ptosis? The crew that can't even keep a hospital's critical computer systems running is going to one-up humans? Maybe it's something you'll welcome for your child; for mine, I'll pass on the opportunity, thanks very much.

August 11, 2015
Large Organization Software Fails: the case of Microsoft Windows

Large organizations have trouble building software. This has been true since the dawn of software history, and shows no signs of changing. The decades-long, rolling disaster of Microsoft Windows is a great example of this. I've been hit personally with this. Recent experiences with Windows 8 have renewed my appreciation of the breadth and depth of the on-going awfulness of Windows.

Windows Screen Saver

I got a new computer. It had Windows 8. I was setting up my new machine and I wanted to do something simple. I had remembered that in some earlier version of Windows, you could get the screen saver to display the file name of the photo it was showing. This was useful if you wanted to get your hands on the photo that just flashed by. It's a pretty small feature, but one anyone who stores photos on their PC could find it useful.

So I drilled in to the screen saver.

I went into the settings, and didn't see the control I was hoping would be there.

So I clicked on Help, something I rarely do, but what the heck, that's what it's there for. Here's what I got: The content is missing!

It's a little thing. It's not like my computer crashed. In the world of books, it's like a footnote was missing — hey, that's an idea, let's compare the new edition of Windows to the new edition of a book!

Software and Books

Most of us know how to judge books. If a book is poorly produced, like the pages tear easily and the type is hard to read, most of us will toss it aside — it may have great content, but it's not worth reading. If we get past the first impression, we'll dive in and start reading. The next potential barrier is how well the book has been edited. If the book is full of spelling, usage and grammatical errors, many of us will think poorly of the author, the editor and the publishing house — the author shouldn't have made the mistakes in the first place, the editor should have caught and corrected them, and the publishing house shouldn't have put sloppy trash in print. Then and only then do we get to the style and substance of the book.

I read a lot of books from many publishers in many genres — fiction, history, science, etc. — and I'm happy to report that I rarely encounter a published book that has editing errors.

And by the time a particularly timeless book gets to later editions? There are never errors.

In that context, how is Windows 8?

I've got the latest version of Windows, 8.1, running on a new machine. It's hardly a first edition. Microsoft pours out updates, and I'm up to date. Here's a snapshot:

Note the scroll bar — there were hundreds more updates that had been applied.

The lovely option that lets you see the file name along with the picture was in an earlier version of Windows. Making a new edition of software isn't that much different than making a new edition of a book — basically, unless you add or change something, it stays the same. In this case, someone had to make a conscious decision to drop an isolated, harmless feature that gave value to many customers.

Why would someone do that? It's more trouble to drop a feature than just let it ride along on the next edition, so someone had to actively remove it. There is no conceivable objection to the feature. While not everyone would want it, since it's an opt-in feature, it harms no one. It's like someone deciding to drop a short appendix from a book — not everyone will want it, but those who do value it. In the paper publishing world, dropping it might save a page or two. But in the electronic world? There's no conceivable reason.

I don't claim for a second that displaying the file name on the screen saver is important. I simply claim that the decision to drop it exemplifies the pervasive anti-customer attitude of the Microsoft organization, which unfortunately is typical of large software-building organizations in general.

It's the missing Help file though, that really set me off. Again, it's a trivial error, like dropping a footnote. But why would you do it? How could it possibly slip though what should be a totally automated editing/QA process?? It may somehow be complicated in the labyrinthine world of Windows development, but it's a fixable thing. You have a program that assures that for each instance of Help there's a corresponding piece of content, and for each piece of content there's a way to reach it. There either is no such program or it's broken. In the overall scheme of things (Windows remains horrifically slow, it freezes and crashes, etc.) it's a small thing, but surely by the edition of Windows 8 I am suffering with it would have been found and fixed?

Conclusion

Software is all about productivity, attention to detail and automation. Unless you've got a de facto monopoly, software is also about meeting customer needs. Large organizations in general (for example government, big corporations) and Microsoft in particular don't get that, in spite of the billions they spend on development and (supposedly) quality. I would love to be able to say it's getting better, but most of the evidence is on the other side. Which is why, among other things, good software will continue to be produced mostly by organizations that are small and willing to do things the "wrong" way.

August 3, 2015
The Science of Drugs vs. the Science of Computers and Software

Prescription drugs are important elements of our lives. There is a strict, scientific, testing-based process to assure that drugs that become widely used are safe and effective, with known side effects. Computers and software are also important elements of our lives. There is a chaotic, fashion-trend-based process used to select the mixture of tools and techniques used to build, maintain and operate our IT systems, resulting in widespread failures, along with cost and quality problems. Worse, there is no recognition that this is the state of affairs, and no movement to correct the situation.

Pharmaceuticals

Everyone knows that drugs are important, and an important part of our economy. Here are some numbers from the CDC.
In 2013, we spent about $271 Billion dollars on prescription drugs. That's quite a bit, but just about 10% of national health spending.

I won't recount the process drugs go through to get approval from the FDA, but I think everyone knows it's an elaborate, multi-year and multi-stage process, with testing at each step to assure that we know how a proposed new drug will work in human beings. While I have my complaints, there is a process, and it's scientific and evidence-based.

IT

The IT industry is also a large one. Here's a breakdown of it worldwide.

There are conflicting estimates of its size in the US, but here's a representative one.

Note that the definition of IT does not include the activities of well-known IT-centric companies like Google.

I was fascinated to see that in 2013, IT was three times the size of the entire pharmaceutical industry. Amazing.

Drugs and IT

Drugs are developed by scientists. They are vetted by a strict scientific process. Only drugs that make it through all the tests are widely used. As a result, the vast majority of drugs are used safely and effectively by the vast majority of patients, with a few experiencing side effects that have already been identified.

IT is run by professionals and staffed with computer scientists and engineers, using tools and techniques developed over many years by scientists and engineers. No matter how high-profile and important the project, regardless of the involvement by government or private companies, a shocking fraction of IT projects end up late, too costly, ineffective or worse. Industry-accepted certifications seem to make no difference. New methods and techniques emerge, become talked about and are deployed widely without any evidence-based process being used to assure their safety and effectiveness. The industry is rife with warring camps, each passionately committed to the effectiveness of their set of tools and techniques. But there isn't even postmortem testing to see which ones were better at gaining its adherents admittance into IT "heaven."

Conclusion

I think the FDA-run drug acceptance process could be much better than it is. But the important thing is, everyone involved in prescription drugs understands and acts scientifically about the process. No one, including me, wants that to change.

The IT industry is at least three times the size of the drug industry. There are computer science and computer engineering departments in every major university, and their graduates staff the industry. It's hard to imagine that they don't understand science, scientific process and evidence-based reasoning. However: they adhere to faith-based processes and vendor-driven products that yield horrible results year after year. None of them say, "hey, this stinks, maybe we can apply that thing that Galileo, Newton and Einstein did, what's it called, science?"

The last thing I want is government involvement in IT, given how horribly government handles its own IT affairs, and I'm not suggesting it here. But it's a sign of just how bad things are in IT that the bureaucratic, government-run FDA does a more scientific job with drugs than anyone does with IT.

July 21, 2015
Cognitive Computing and Healthcare

They say that cognitive computing, the term-du-jour for Artificial Intelligence (AI), is in the process of transforming healthcare. Billions of dollars of investment are behind the effort. Sadly, there are good reasons to believe that little good will come of it.

Cognitive Computing

Whatever it is, people are pretty sure it's BIG. Here's what a major investor and the former GM of IBM's Watson unit says about it:

$80 billion dollars! Before long, we'll be talking serious money here!

Where's this money going? Lots of places. But there's one special target for the money. The same expert tells us:

Cognitive Computing in Healthcare

Is Cognitive Computing really happening in healthcare? You betcha. IBM's Watson by itself is making major inroads into healthcare, with terrific-sounding projects at Sloan Kettering, Cleveland Clinic, MD Anderson and others. Good things are coming! For example, C. Martin Harris, MD, chief information officer of Cleveland Clinic, says:

Cleveland Clinic's collaboration with IBM is exciting because it offers us the opportunity to teach Watson to 'think' in ways that have the potential to make it a powerful tool in medicine.

And here is how the CEO of IBM explained it in an interview:

It's so hot that IBM has created a separate division for Watson, investing more than $1 billion just to get it started, and will have a headquarters group employing more than 2,000 people.

So What's the Problem?

Big investments like this should mean that there's a big problem to be solved. What is it? Not enough doctors? The doctors are too expensive, and somehow automating what they do with this mega-expensive effort will help that? The doctors aren't as smart or educated as Watson will (by presumption) be?

Someone involved should let the rest of us know.

Meanwhile, count me a skeptic. The reason is simple: there is a decades-long history of researchers and big companies making claims more modest than the ones being made for "cognitive computing," and they've all failed, technically and/or in business terms. In the end, computers do get used for more and more, as we all know from personal experience. That's a trend that will certainly continue. But "cognitive computing," i.e., AI reincarnated and re-named? Uh-uh.

July 14, 2015
Fatal Flaws of Big Data

The message appears to be: if you're not way into Big Data, you're missing out on important things! For vendors and job seekers, I'm sure this is true, without reservation. For the companies that wish to benefit from the investment? Maybe not.

The Big Data Trend is BIG!

There's one thing for sure about Big Data: it's a Big trend.

We have been assured that Big Data is now the driving force in computing.

If you scan through the books, conferences and other things whose focus is Big Data, it's clearly a major fashion trend.

Whenever something like this catches first, everyone wants to jump on. Lots of people talk about their own "big data" that, on a closer look, isn't so big after all.

And generally when you look at it more closely, Big Data doesn't look quite so cool.

Is there something wrong here?

How much better is data that's BIG compared to MEDIUM or SMALL?

The killer assumption behind all the Big Data excitement is that Big is better than normal-size data — lots better. Makes sense, right?

Not so fast.

Let's spend a little time thinking about the core issues of data coverage, integrity/quality, and probability.

Probability

Today, we've got X data. Let's assume that, with Big Data, we've now got 100X data. Are we 100X better off?

Let's start with something simple and universal: flipping coins. Suppose we place ads. We make money when the coin comes up heads, and lose money when it comes up tails. Our data people tell use that the odds of getting heads are 0.5, with a certainty of 0.1 — i.e., the chance of it coming up heads is probably 0.5, but it might be as little as 0.4 or as much as 0.6. Now we have 100X more coin flips to apply to our measurement. Great, now we're really going to start marking money!

They come back, sweaty and proud with the answer: the probably of getting heads is 0.500, with certainty of 0.001 — i.e., the chance of it coming up heads is probably 0.5, but it might be as little as 0.499 or as much as 0.501. Wow, we've increased our level of precision massively! How much does that increase the money we make? Hmmmmm.

Quality/Integrity

Maybe the problem is that we just got lots more data points about the same thing. It didn't broaden our knowledge. Maybe we need to expand, check out the odds for not just nickels, but also dimes and quarters. Hmmmm. Let's get more ambitious. Let's track users, not just on our website, but also on 100 other websites. Tell the programmers to get going! We're going to be rolling in money from Big Data!

The programmers seem to be having trouble matching people over different web sites. Are all these people who claim to be David Black the same person? What about that David B. Black guy? And there appear to be two really different patterns of use coming from the same IP address — maybe someone else is sharing the computer? And I just discovered that there's a David Black who appears to use the internet from Manhattan, and a David Black who uses it from some place out in New Jersey. We already know there are multiple David Blacks. This could be one person or two. Which is it? This is getting hard.

Darn. I thought all I had to do was get loads more data and a Hadoop cluster and the money would start pouring in. Getting all that data to match up and make sense is harder than it looks. And then, when I've done it, is all I'm achieving increasing my level of certainty about what I already knew?

Data Coverage and lift

Alright. I've got my 100X more data. I've FINALLY sorted it out so it's high quality and matches up. Now I've got to make sure it really broadens my knowledge and gives me uplift in my results.

So far, all I've been doing is looking at my customers' actions. I bet if I look at demographics and social media — that's lots of data, surely it qualifies as "Big," I'll get better results. Big Data team — mush!

Darn, darn, DARN! Yeah, all this big data stuff changes what I offer to whom for how much — but it's not making a whole lot of difference in my results. And I'm getting hammered with complaints from people who want me to stop making offers to their kids, and old customers who wonder why we don't love them anymore. Yeah, we're getting 5-10% uplift, but we're losing at least that much from our old business, not to mention all the costs we've added.

Who's making money from the Big Data stuff? It must be the consultants, the vendors and the conferences. It's sure not me.

Of course, I could just patch it all up, start going to conferences, bragging about how I'm an expert, and maybe I'll get a great new job. But it would be based on a lie. I'm not that kind of person.

Conclusion

I love data. I love exploring it, analyzing it by all available means, and understanding it. Evidence-based solutions are the only ones I'm comfortable with. Everything else is just baseless faith. If I can use math optimization, machine learning or something else to do a better job than a person could do, I'm all for it. If I can get additional data and that data will help me get better results, bring it on!

But "Big Data" is not in principle better than "enough" data. Too little is not enough. More than you need to get the job done is a waste. Just like Goldilocks, you should want the amount of data that's "just right."

July 7, 2015
Innovations that aren’t: data definitions inside or outside the program

There are decades-long trends and cycles in software that explain a great deal of what goes on in software innovation – including repeated “innovations” that are actually more like “in-old-vations,” and repeated eruptions of re-cycled “innovations” that are actually retreating to bad old ways of doing things. Of the many examples illustrating these trends and cycles, I’ll focus on one of the most persistent and amusing: the cycle of the relationship between data storage definitions and program definitions. This particular cycle started in the 1950’s, and is still going strong in 2015!

The relationship between data in programs and data in storage

Data is found in two basic places in the world of computing: in “persistent storage” (in a database and/or on disk), where it’s mostly “at rest;” and in “memory” (generally in RAM, the same place where the program is, in labelled variables for use in program statements), where it’s “in use.” The world of programming has gone around and around about what is the best relationship between these two things.

The data-program relationship spectrum

At one end of the spectrum, there is a single definition that serves both purposes. The programming language defines the data generically, and has commands to manipulate the data, and other commands to get it from storage and put it back.
At the other end of the spectrum, there is a set of software used to define and manipulate persistent storage (for example a DBMS with its DDL and DML), and a portion of the programming language used for defining “local” variables and collections of variables (called various things, including “objects” and “structures.” At this far end of the spectrum, there are a variety of means used to define the relationship between the two types of data (in-storage and in-program) and how data is moved from one place to the other.

For convenience, I’m going to call the end of the spectrum in which persistent data and data-in-program are as far as possible from each other the “left” end of the spectrum. The “right” end of the spectrum is the one in which data for both purposes is defined in a unified way.

People at the left end of the spectrum tend to be passionate about their choice and driven by a sense of ideological purity. People at the right end of the spectrum tend to be practical and results-oriented, focused on delivering end-user results.

Recent example: Ruby and RAILS

The object-oriented movement tends, for the most part, to be at the left end of the spectrum. Object-oriented people focus on classes and objects, which are entirely in-program representations of data. The problem of “persisting” those objects is an annoying detail that the imperfections of current computing environments make necessary. They solve this problem by various means; one of the most popular solutions is using an ORM (object-relational mapper), which does the work of moving objects between a DBMS and where they “should” be nearly automatic. It also can automate the process of creating effective but really ugly schemas for the data in the DBMS.

A nice new object-oriented language, Ruby, appeared in 1995. It was invented to bring a level of O-O purity to shell scripting that alternatives like PERL lacked. About 10 years later, a guy was working in Ruby for Basecamp and realized that the framework he'd created for it was really valuable: it enabled him to build and modify things his company needed really quickly. He released it to open source and became known as the RAILS framework for Ruby, the result being Ruby on RAILS. Ruby on RAILS was quickly adopted by results-oriented developers, and is the tool used by more than half a million web sites. While "pure" Ruby resides firmly at the left end of the spectrum, the main characteristic of the RAILS framework is that you define variables that serve for both program access and storage, putting it near the right end of the spectrum.

Rhetoric of the ends of the spectrum

Left-end believers tend to think of people at the other end as rank amateurs. Left-enders tend to claim the cloak of computer science for their style of work, and claim that they're serious professionals. They stress the importance of having separation of concerns and layers in software.

Right-end practitioners tend to think of people at the other end as purists, ivory-tower idealists who put theory ahead of practice, and who put abstract concerns ahead of practical results. They stress the importance of achieving business results with high productivity and quick turn-around.

It goes in cycles!

There have always been examples of programming at both ends of the spectrum; neither end disappears. But what happens is that the pendulum tends to swing from one end to the other, without end up to now.

In 1958, Algol was invented by a committee of computer scientists. Algol 60 was purely algorithmic — it only dealt with data in memory, and didn't concern itself with getting data or putting it back. Its backers liked its "syntactic and semantic purity." It was clearly left-end oriented. But then practical business people, in part building on earlier efforts and in part inspired by the need to deliver results, agreed on COBOL 60, which fully incorporated data definitions that were used both for interacting with storage and for performing operations. It was clearly right-end oriented. All the big computer manufacturers, anxious to sell machines to business users, built COBOL compilers. Meanwhile, Algol became the choice for expressing algorithms in academic journals and text books.

There was a explosion of interest in right-end languages during the heyday of minicomputers, with languages like PICK growing in use, and niche languages like MUMPS having their day. The rise of the DBMS presented new problems and opportunities. After a period of purism, in which programmers struggled with using left-end languages that were supposed to be "easy" like BASIC to get jobs done, along came Powersoft's Powerbuilder, which dominated client-server computing because of the productivity win of integrating the DBMS schema with the language. The pendulum swung back with the rise of the internet and the emergence of "pure" java, with its arms-length relationship to persistence, putting it firmly in the left-end camp. Since Java wasn't pure enough for some people, "pure" Ruby got out there and grows — and some Ruby user is under pressure to deliver results and invents RAILS, which then spreads to similarly-minded programmers. But "amateurs" who like Java but also like productivity invented and are spreading Grails, a derivative of Java and RAILS that combines advantages of each.

So is RAILS an innovation? Java? I don't think so. They're just minor variants of a recurring pattern in programming language design, fueled by the intellectual and emotional habits of groups of programmers, some of whom push towards the left end of the spectrum (with disdain for the amateurs) and others of whom push towards the right end of the spectrum (with disdain for the purists).

Conclusion

This spectrum of the relation between data and program has persisted for more than 50 years, and shows no signs of weakening. Every person or group who, finding themselves in an environment which emphasizes the end of the spectrum they find distasteful, tries to get to the end they like. Sometimes they write a bunch of code, and everyone thinks they're innovative. Except it would be more accurate to say they're "inno-old-ive." This is one of the many patterns to be found in software history, patterns that account for so much of the detail that is otherwise hard to explain.

June 30, 2015
What Baseball can teach us about OPM, Anthem and other Cyber-Thefts

In baseball, teams play against each other. Each half inning, one team does its best to attack the other and score, while the other does its best to stop them. The teams are similarly staffed, and they alternate playing offense and defense. In computer security, teams also play against each other. The "home" team always plays defense, while the "away" team comes to town and tries to score against their hosts.

Tiny, often remote "visiting teams" in cyber-war score massive victories against huge, well-funded organizations like OPM and Anthem. These are rarely quick "hit and run" attacks — they are more often months-long penetrations, during which massive amount of information gold is marched out of the "well-guarded walls" of the clueless behemoth. What's worse, most people don't seem to care — imagine if a single gold bar were secreted out of Fort Knox: heads would roll! How can this happen? Why does no one seem to care?

Baseball and Cyber-war

First and foremost, baseball is visible. We can see it and understand it. Loads of fans come to stadiums to watch it.
Cyberwar? It's largely invisible. It's as though the stadiums were empty.

In baseball, we can actually see the team at bat competing against the defenders.
It's pretty exciting! For the vast majority of people, there is no equivalent in cyber-war.

The fans and managers understand the game; those closest to it have normally played it. They have strong opinions, for example, about the defensive shift maneuver, which is sometimes used against a pull hitter. Even if you've never heard of it, a simple diagram makes it easy to understand.
In cyber-war there are also strong opinions, but the way most managers think about cyber-defense is simply inappropriate and ineffective. Not only is there no defensive shift, there is a complete lack of awareness when the enemy has been inside your walls for weeks, ransacking away. Because no one understands what's going on, including those in charge, the ineffective methods continue to be standard practice, even when there are better approaches available. Retail stores, who actually care about loss prevention, generally have better theft prevention measures.

Above all, there's this. The people who play baseball care about it.
So do the people who watch baseball. Cyber-war is way more than a game, but people just don't take it seriously. They don't even give the passion to it that they give to games! The individual computer users don't know or care, and neither do the managers.

Conclusion

Nothing will change in Cyber-war until we understand it, start caring about it and apply methods that work. In a fight between the smart and motivated against the clueless and unmotivated, the outcome is preordained.

June 23, 2015
Systemic Issues Behind the Cyber-Security Disasters at OPM, Citi, Anthem, etc.

Our personal data is stored in the computers at large corporations and government organizations. We now have abundant proof that these large organizations are incapable of protecting our data. This is not a string of bad luck that will soon pass. These large organizations never had good security — they just weren't being attacked. Unfortunately, the security flaws are a direct outcome of the dysfunctional technical and management practices that lead to large-organization IT failures across the spectrum.

Recent Security Disasters

The security disaster at the government Office of Personnel Management (OPM) has been in the news recently. Here is a summary, and here is a timeline. OPM knew all about security, and tried its darndest to be secure, spending over $4.5 Billion dollars on a system to prevent breaches, including a recent $218 million upgrade on the security system known as Einstein. All for naught.

In the private sector, there was the breach at Anthem, preceded by a string of security disasters at major banks and retailers involving tens of millions of consumer records.

The Response to the Attacks

We're seeing the usual responses to the problems.

First and foremost, try to avoid letting anyone know there's a problem.

Second, try to draw attention to all the attacks that were thwarted. The OPM is actually bragging about all the attacks they defend against! That's like, when the bank has been totally cleaned out, bragging about how many attempts had been thwarted.

Finally, talk about how much you care, offer completely counter-productive services to consumers, and spend even more money on the stuff that didn't, doesn't and won't work. Ignore the fact that the incentives are all wrong, that in fact no one cares.

No one is losing their job. No significant changes are being made. No one is running around like their hair's on fire. Ho-hum, it's business as usual.

Systemic Issues are behind the Disasters

Security in large organizations is broken. But that's just a side effect of the fact that IT in large organizations is broken. Not in detail — in principle. When the foundation of a building is made out of jello instead of concrete, you don't fix it by adding more jello, trying a new flavor of jello, or getting everyone to walk slowly and carefully. You replace it with reinforced concrete — pronto! When the foundations are the wrong kind of stuff, making new foundations out of jello will never help. Even if it's jello that costs billions of dollars.

The Systemic Issues

This is a subject that is long and deep. All the problems come down to two simple core thoughts: (1) computers are just like all the other things to which management techniques are applied, so standard-issue "good management" will solve any problems; and (2) computer security is just like all the other computer issues, and can be managed using the same standard techniques.

Wrong and wrong.

Computers and software in general are radically different than anything else we encounter in our normal lives, and evolve more quickly by orders of magnitude than anything else in human experience. Managing a software building project as though it were a home building project leads to results that are, at best, 10X worse than optimal methods, and at worst, complete disaster.

Computer security in particular is not just another issue to be managed using standard techniques, which in any case yield horrible results. In computer security, we're dealing with smart and motivated attackers who are at war with us, and naturally use the latest "weapons" in a rapidly evolving arsenal. While our attackers are at war with us, we plod along at a peace-time pace, scheduling security issues like just the other items in prioritized lists. When the armed gang breaks through the back door of the warehouse, we eventually discover the break-in and schedule a response for sometime in the next couple of months. By the time we've installed new alarms, the gangs are already on their third generation of tools for defeating them.

Computers are different than the other things we manage

Computers evolve at a pace that is completely unprecedented in human experience.

Most of the things that managers do to manage computers is modeled on what they do for everything else, and make things worse.

Computers are incredibly complex! But somehow, we imagine that people with no actual experience with computers can manage them, when we would never let someone who never saw a baseball game manage a team, or someone who never wrote an article manage writers.

The vendors of hardware, software and services have evolved to provide incredibly expensive, ineffective products and services that are packaged to make top managers feel great.

Computer security requires war-time actions, not peace-time ones

Translating from physical security, managers insist that security is about walls, guards and kevlar vests. The bad guys are out there, our job is to keep them out. Wrong. The vast majority of security breaches result from either conscious or unknowing cooperation of insiders. Including OPM.

The bad guys are at war with us. By the time we've figured out that we've been robbed, the bad guys are long gone. By the time we're just wrapping up the requirements documents for our response, the bad guys have cleaned us out again.

Once we finally deploy our best defense, the art of war has advanced and our defenses are useless, just like the Maginot Line in World War I.

Conclusion

We all know that the definition of insanity is repeating the same actions and expecting different results. In that sense, the approach that large organizations, private and public, take to computer security is insane. All the people in charge propose is doing what they've always done, only somehow harder and better. The alternative approach, while radically different from the current one, is simple, clear and actionable. The people in charge actively resist it today. They've got to embrace it if there is to be any chance at all of improvement in cyber-security.

June 18, 2015