When a kind of software isn’t available, it makes sense to build it. If it is available to buy, everyone says you should buy it. But in a surprising number of cases, building makes sense even when you can buy software.

Here's an example of a company that wasn't in the software development business decided to build what they needed, and grew an amazing business from there.

There was a little bank in Georgia that somehow got the ambition to process credit cards for their customers when that was a new thing. There was no software available to buy, so they built some primitive software. It worked well enough. Then they improved the software and sold some local banks on processing credit cards for them (not necessarily in that order). One thing led to another, and the software evolved into the second-largest credit card processing company in the US, TSYS. Particularly when software categories are new, that’s what people do – they build software.

As the software becomes available, it starts to make sense to buy it. There are always issues at the beginning (see this about developing functionality on a new platform) about the time and effort to make the software work for you, and whether it has all the features you need. Sooner or later, the surviving vendors reduce the pain of customization and implementation, and develop a rich enough set of functionality so that the needs of nearly everyone in a business segment are met. At some point on that spectrum, it becomes kind of nuts to build it when you could buy it. It might take you years to build what you need, and then you’ll have to support it, and you’ll always be behind your competitors who are using off-the-shelf software, not to mention the risk that the software you try to build doesn’t end up working or really meeting your needs. So the mantra in the corporate offices becomes “buy, buy!” And only if you really can’t buy it should you build it; and even then, you should look real hard for excuses to put it off. If nothing else, there is less risk associated with buying software, and corporate managers nearly always vote for the least-risky of alternatives.

However, as these products have been getting built, tools and components for building software have also been advancing, and the ability to make your software systems adapt to changing business conditions has becoming increasingly important as a business advantage. Meanwhile, the major products in any category tend to get bloated, as pressures from various customer groups result in feature after feature being added to the product. In the early days of a product, you are most concerned that the features you need are in the product; as the product ages, the concern shifts to the burden on the software and users from the vast majority of the features in the product that someone may need, but not you. It may be that you only need a tiny fraction of the available features. An example of this that is close to home for many people is the Microsoft Office products – who even knows what most of the menu items in Word mean, much less uses them?

The case against purchased products gets stronger when you need several of them, and they all have to work together. Then you have the problem of application integration, which is typically compounded by different release and upgrade schedules from the vendors, who, try as they might, don’t seem to be able to release a new version without screwing something up that worked in the older version. If, on top of it all, you have an ambitious and low-cost programming group at your disposal, then you’re a good candidate for the final stage in this evolution, which is, “OK, I bought it and I’m using it, now when do I get to turn it off?” “Programming” shops whose main purpose in life is baby-sitting purchased products can’t even imagine doing something like this, which is probably just as well, because they are highly unlikely to pull it off. But groups who are itching to write code, and do stealth projects to prove what they can do, are another matter.

So the evolution returns to the starting point, only typically at a much more advanced level, with more advanced tools and a knowledgeable view of the business and the required functionality. “Why pay the vendors their ransom and put up with the crap they dish out, when I can write it, own it, change it any time I need to, and control my own destiny?” If the numbers work out, why not, indeed?

The story of the evolution of spreadsheets is well-known, though rarely studied. Spreadsheet evolution illustrates the patterns of software evolution. It shows the power of those patterns; if the leaders of Apple, Lotus and Visicalc had known and acted on the predictive power of those patterns, Office would not have been a Microsoft product, but one of theirs.

As a reminder, here are the basic patterns of software evolution:

Software is created for a new platform.

Software grows more capable on the platform with increasing abstraction.

The automation provided grows deeper.

The breadth of automation grows wider, doing more things.

The Spreadsheet products in evolutionary terms

The first popular spreadsheet program appeared on the then-new Apple computer. It was Visicalc. In terms of the stages of evolution, it was this:

• Emergence on a new platform: Conceptual Breakthrough
• Development on a platform: Basic product
• Automation depth: Recorder
• Automation breadth: Point product

If the platform became dominant and remained stable, we would expect competition by increasing the depth of automation on the Apple, increasing the breadth of automation e.g. by creating a collection of related products, or by moving along the same-platform development dimension. Instead, a powerful and popular new platform emerged, the PC. Visicalc themselves should have become the leaders in spreadsheets for the new platform, but as so often happens, the emergence of the new platform is an opportunity for a new, focused company to do it.

The company that won was Lotus, with their 1-2-3 product.

• Old platform: Apple I/II
• Old functionality: Spreadsheet, like Visicalc
• New platform: IBM PC/DOS and clones
• New functionality: Spreadsheet, much like Visicalc, along with some hype of other applications
• Outcome: They won big time

All the other categories (Basic product, etc.) remained the same.

Meanwhile, back in the land of Apple, the Mac came out, the first practical GUI on a personal computer platform. You would think this would be the big chance for Apple themselves to jump into the business, or for Lotus to convert their category-killing program. And of course, it was their big chance, but they blew it. Instead, Microsoft of all unlikely companies, jumped in.

• Old platform: Apple I/II, PC/DOS
• Old functionality: Character mode spreadsheet, like Visicalc and Lotus 1-2-3
• New platform: Apple Mac GUI
• New functionality: Excel Spreadsheet, but with attractive fonts and graphics
• Outcome: They won on the Mac

It wasn’t easy to get a spreadsheet right for GUI, but Microsoft kept plugging away and really took advantage of the graphics and the mouse.

At the same time, Microsoft was plugging away at their own clone of the Mac operating system for the PC, Windows, which they announced in 1975 and finally came out with a usable release, 3.1, in 1982 (seven loo-oo-oong years later – but that’s another story). Here we have a GUI platform for the PC, Lotus’ home base. The GUI had shown its attractiveness elsewhere. There was a GUI backer who was clearly going to see it through. You would think that the savvy business executive who was in charge of Lotus at the time, surrounded by a heavy-weight group of Cambridge thinkers, would have been all over this. Didn’t happen. While Apple built the GUI but failed to build the spreadsheet for it, and while Lotus built the spreadsheet but failed to convert it to a GUI on their own dominant platform, Microsoft built both the GUI and the spreadsheet.

• Old platform: Apple Mac, PC/DOS
• Old functionality: Mac GUI spreadsheet; PC/DOS character spreadsheet
• New platform: PC Windows
• New functionality: Same as on Mac
• Outcome: Huge win/win: PC/Windows helped Excel, Excel helped Windows

At the time, Lotus had much more revenue than Microsoft, and seemed reluctant to rewrite its industry-leading spreadsheet for Microsoft’s Windows, which was seen as a move that would help the success of that platform, and thus help Lotus’ competitor. But once it was clear that Windows was here to stay, Lotus finally did the re-write, and got 1-2-3 onto Windows. So now 1-2-3 and Excel were competing as point products on the same technology platform.

But now Lotus had another problem: 1-2-3 was a point product (I know, I know, they said they had three applications, thus the “1-2-3,” but who used the “2-3”?), and by this time, Microsoft had an actual product suite: Microsoft Office, combining spreadsheet with word processing and presentations via OLE and drag-and-drop. On the movement from point product to product collection to product suite, Microsoft was consistently a step ahead, and thus had more market pull than Lotus. While Lotus was plugging the virtues of 1-2-3 vs. Excel, Microsoft was promoting the value of its product collection. Then, while Lotus was promoting the value of its acquired collection of office products, Microsoft had a real product suite. The end result, we know today, is the complete dominance of the Microsoft Office product suite.

Conclusion

The evolution of the spreadsheet fits nicely into the general patterns of software evolution I have described. If strategists and company leaders had been aware of these patterns and acted on their proven predictive power, the outcome of the spreadsheet wars would have turned out differently. Leaders today would be well-advised to understand and apply the patterns of software evolution.

Some software is changed over time. But there is a great deal of software that is simply part of the landscape or part of the underpinning of the software we see. This largely ignored category of software is important  to understand.

Keeping up with Software

I had a nice conversation with the non-technical CEO of one of our software-based companies. He expressed the widely-held view that the process of developing software is undergoing change at a dizzying pace. Just in the last couple of years, new languages and tools have emerged and are being used and talked about. How can anyone possibly keep up, he wondered?

That certainly is how things seem to most people, both inside and outside of the computer industry. We had desktop machines, personal computers, 25 years ago, and we have them today; of course, they’re less expensive and have way more power and capacity than they did back then, but they’re still personal computers, we put them on or beside our desks, and use them. Software, on the other hand, undergoes constant change. For programmers, we’ve gone through an incredible changing landscape of programming languages and tools during that time. I’ll just give a short list: assembler, BASIC, turbo Pascal, Smalltalk, C, Objective-C, 4-GL’s too numerous to list, C++, Java, C#, PERL, javascript, php, python, ruby and on and on. It’s no wonder that it seems like it’s hard to keep up! And of course, in a way it is – it does take work. But most of that language change is little but noise. The claims of advances don't hold up when examined.

That’s how it seems. Hardware is hardware. Yes, it gets better and cheaper, but software, wow – there’s always something new!

The reality is different. True software change, whether in methods or bodies of code, is slower than glacial.

The Non-Evolution of Software Categories

It’s hardware that undergoes dramatic change, not software. Think Moore’s Law. Think about mobile phones just ten years ago. Here are details. By contrast, change in software is actually glacial. Yes, the surface weather of software changes rapidly and unpredictably; but the underlying principles and structure of software don’t even move as fast as glaciers – it’s more like tectonic plates.

Things are changing all the time.  Software is certainly “soft” compared to hardware. But programmers spend most of their time in their minds, not wandering around touching up the hardware. What do they spend time thinking about? It’s not the hardware – it’s the software environment in which they work, and in which the software they create will live.

Think about kids in a sand box. The sand is something they can push around, pile up, makes holes in, and generally have fun with. How about the box the sand is in? It’s hard. It keeps the sand in one area. No kid thinks about it much – it’s just something that’s there when you’re playing.

It works the same in software. The “part” of the software you are playing with (a.k.a. “programming”) seems “soft” and easily changeable, like sand. The “part” of the software you’re not playing with seems “hard” to you. It’s just part of the landscape. This effect is strongest when the programmer has no access to the source code of the landscape software. An example is a DBMS or an operating system. These things feel like immutable reality. They are what they are, and you have to deal with them. Hardware speed and storage capacity has grown so much, for example, that DBMS's are a data storage solution to a problem that no longer exists. But most software design continues to be built on them.

Just as other pieces of software feel like given realities if you’re a programmer, so does the programming language in which you’re working. The verbs and the syntax are as real as the grains of sand. Just as when you play with sand, you have no sense that each grain is actually a very large aggregation of molecules, so when you use a verb in the programming language, you have no sense that a compiler or interpreter will actually use many machine language instructions to execute each verb.

The “hardness” of software to which you have no source code access (or otherwise no ability to change) is a strong psychological effect.

The Glacial Evolution of Software Applications

Programmers have a special view of software. Regular people also use software personally and/or professionally. While society gives us the impression that the software people use changes like crazy, the reality is that people learn new things slowly, and successful software evolves very slowly. The changes we see are largely incidental things that are enabled by the astounding advance of hardware.

The Bedrock of Production Systems

Large bodies of working, production code are the bedrock on which fancy new layers may be built, but change very slowly. Here's an example of the impact of that hard-to-change code.

It isn’t as though people don’t want to change these systems. Often they are desperate to change them! Sometimes managers will put huge investments into changing or replacing them, only to quietly admit failure and stop trying. Until the next bunch of ignorant-of-history managers come along.

These bedrock systems aren't just systems software like DBMS's. They are also plain old applications. For example, the software that handles the largest number of credit card accounts in the world is written in the prehistoric language COBOL. Several attempts were made to re-write card processing software using fancy new languages. Failed. See also this and this.

Conclusion

Hardware changes with amazing speed, giving no signs of slowing down. While there's always something new to talk about in software, it changes much more slowly than hardware, and is remarkably hard to change.