Category: Software Evolution

When a kind of software isn’t available, it makes sense to build it. If it is available to buy, everyone says you should buy it. But in a surprising number of cases, building makes sense even when you can buy software.

Here's an example of a company that wasn't in the software development business decided to build what they needed, and grew an amazing business from there.

There was a little bank in Georgia that somehow got the ambition to process credit cards for their customers when that was a new thing. There was no software available to buy, so they built some primitive software. It worked well enough. Then they improved the software and sold some local banks on processing credit cards for them (not necessarily in that order). One thing led to another, and the software evolved into the second-largest credit card processing company in the US, TSYS. Particularly when software categories are new, that’s what people do – they build software.

As the software becomes available, it starts to make sense to buy it. There are always issues at the beginning (see this about developing functionality on a new platform) about the time and effort to make the software work for you, and whether it has all the features you need. Sooner or later, the surviving vendors reduce the pain of customization and implementation, and develop a rich enough set of functionality so that the needs of nearly everyone in a business segment are met. At some point on that spectrum, it becomes kind of nuts to build it when you could buy it. It might take you years to build what you need, and then you’ll have to support it, and you’ll always be behind your competitors who are using off-the-shelf software, not to mention the risk that the software you try to build doesn’t end up working or really meeting your needs. So the mantra in the corporate offices becomes “buy, buy!” And only if you really can’t buy it should you build it; and even then, you should look real hard for excuses to put it off. If nothing else, there is less risk associated with buying software, and corporate managers nearly always vote for the least-risky of alternatives.

However, as these products have been getting built, tools and components for building software have also been advancing, and the ability to make your software systems adapt to changing business conditions has becoming increasingly important as a business advantage. Meanwhile, the major products in any category tend to get bloated, as pressures from various customer groups result in feature after feature being added to the product. In the early days of a product, you are most concerned that the features you need are in the product; as the product ages, the concern shifts to the burden on the software and users from the vast majority of the features in the product that someone may need, but not you. It may be that you only need a tiny fraction of the available features. An example of this that is close to home for many people is the Microsoft Office products – who even knows what most of the menu items in Word mean, much less uses them?

The case against purchased products gets stronger when you need several of them, and they all have to work together. Then you have the problem of application integration, which is typically compounded by different release and upgrade schedules from the vendors, who, try as they might, don’t seem to be able to release a new version without screwing something up that worked in the older version. If, on top of it all, you have an ambitious and low-cost programming group at your disposal, then you’re a good candidate for the final stage in this evolution, which is, “OK, I bought it and I’m using it, now when do I get to turn it off?” “Programming” shops whose main purpose in life is baby-sitting purchased products can’t even imagine doing something like this, which is probably just as well, because they are highly unlikely to pull it off. But groups who are itching to write code, and do stealth projects to prove what they can do, are another matter.

So the evolution returns to the starting point, only typically at a much more advanced level, with more advanced tools and a knowledgeable view of the business and the required functionality. “Why pay the vendors their ransom and put up with the crap they dish out, when I can write it, own it, change it any time I need to, and control my own destiny?” If the numbers work out, why not, indeed?

The story of the evolution of spreadsheets is well-known, though rarely studied. Spreadsheet evolution illustrates the patterns of software evolution. It shows the power of those patterns; if the leaders of Apple, Lotus and Visicalc had known and acted on the predictive power of those patterns, Office would not have been a Microsoft product, but one of theirs.

As a reminder, here are the basic patterns of software evolution:

Software is created for a new platform.

Software grows more capable on the platform with increasing abstraction.

The automation provided grows deeper.

The breadth of automation grows wider, doing more things.

The Spreadsheet products in evolutionary terms

The first popular spreadsheet program appeared on the then-new Apple computer. It was Visicalc. In terms of the stages of evolution, it was this:

• Emergence on a new platform: Conceptual Breakthrough
• Development on a platform: Basic product
• Automation depth: Recorder
• Automation breadth: Point product

If the platform became dominant and remained stable, we would expect competition by increasing the depth of automation on the Apple, increasing the breadth of automation e.g. by creating a collection of related products, or by moving along the same-platform development dimension. Instead, a powerful and popular new platform emerged, the PC. Visicalc themselves should have become the leaders in spreadsheets for the new platform, but as so often happens, the emergence of the new platform is an opportunity for a new, focused company to do it.

The company that won was Lotus, with their 1-2-3 product.

• Old platform: Apple I/II
• Old functionality: Spreadsheet, like Visicalc
• New platform: IBM PC/DOS and clones
• New functionality: Spreadsheet, much like Visicalc, along with some hype of other applications
• Outcome: They won big time

All the other categories (Basic product, etc.) remained the same.

Meanwhile, back in the land of Apple, the Mac came out, the first practical GUI on a personal computer platform. You would think this would be the big chance for Apple themselves to jump into the business, or for Lotus to convert their category-killing program. And of course, it was their big chance, but they blew it. Instead, Microsoft of all unlikely companies, jumped in.

• Old platform: Apple I/II, PC/DOS
• Old functionality: Character mode spreadsheet, like Visicalc and Lotus 1-2-3
• New platform: Apple Mac GUI
• New functionality: Excel Spreadsheet, but with attractive fonts and graphics
• Outcome: They won on the Mac

It wasn’t easy to get a spreadsheet right for GUI, but Microsoft kept plugging away and really took advantage of the graphics and the mouse.

At the same time, Microsoft was plugging away at their own clone of the Mac operating system for the PC, Windows, which they announced in 1975 and finally came out with a usable release, 3.1, in 1982 (seven loo-oo-oong years later – but that’s another story). Here we have a GUI platform for the PC, Lotus’ home base. The GUI had shown its attractiveness elsewhere. There was a GUI backer who was clearly going to see it through. You would think that the savvy business executive who was in charge of Lotus at the time, surrounded by a heavy-weight group of Cambridge thinkers, would have been all over this. Didn’t happen. While Apple built the GUI but failed to build the spreadsheet for it, and while Lotus built the spreadsheet but failed to convert it to a GUI on their own dominant platform, Microsoft built both the GUI and the spreadsheet.

• Old platform: Apple Mac, PC/DOS
• Old functionality: Mac GUI spreadsheet; PC/DOS character spreadsheet
• New platform: PC Windows
• New functionality: Same as on Mac
• Outcome: Huge win/win: PC/Windows helped Excel, Excel helped Windows

At the time, Lotus had much more revenue than Microsoft, and seemed reluctant to rewrite its industry-leading spreadsheet for Microsoft’s Windows, which was seen as a move that would help the success of that platform, and thus help Lotus’ competitor. But once it was clear that Windows was here to stay, Lotus finally did the re-write, and got 1-2-3 onto Windows. So now 1-2-3 and Excel were competing as point products on the same technology platform.

But now Lotus had another problem: 1-2-3 was a point product (I know, I know, they said they had three applications, thus the “1-2-3,” but who used the “2-3”?), and by this time, Microsoft had an actual product suite: Microsoft Office, combining spreadsheet with word processing and presentations via OLE and drag-and-drop. On the movement from point product to product collection to product suite, Microsoft was consistently a step ahead, and thus had more market pull than Lotus. While Lotus was plugging the virtues of 1-2-3 vs. Excel, Microsoft was promoting the value of its product collection. Then, while Lotus was promoting the value of its acquired collection of office products, Microsoft had a real product suite. The end result, we know today, is the complete dominance of the Microsoft Office product suite.

Conclusion

The evolution of the spreadsheet fits nicely into the general patterns of software evolution I have described. If strategists and company leaders had been aware of these patterns and acted on their proven predictive power, the outcome of the spreadsheet wars would have turned out differently. Leaders today would be well-advised to understand and apply the patterns of software evolution.

Some software is changed over time. But there is a great deal of software that is simply part of the landscape or part of the underpinning of the software we see. This largely ignored category of software is important  to understand.

Keeping up with Software

I had a nice conversation with the non-technical CEO of one of our software-based companies. He expressed the widely-held view that the process of developing software is undergoing change at a dizzying pace. Just in the last couple of years, new languages and tools have emerged and are being used and talked about. How can anyone possibly keep up, he wondered?

That certainly is how things seem to most people, both inside and outside of the computer industry. We had desktop machines, personal computers, 25 years ago, and we have them today; of course, they’re less expensive and have way more power and capacity than they did back then, but they’re still personal computers, we put them on or beside our desks, and use them. Software, on the other hand, undergoes constant change. For programmers, we’ve gone through an incredible changing landscape of programming languages and tools during that time. I’ll just give a short list: assembler, BASIC, turbo Pascal, Smalltalk, C, Objective-C, 4-GL’s too numerous to list, C++, Java, C#, PERL, javascript, php, python, ruby and on and on. It’s no wonder that it seems like it’s hard to keep up! And of course, in a way it is – it does take work. But most of that language change is little but noise. The claims of advances don't hold up when examined.

That’s how it seems. Hardware is hardware. Yes, it gets better and cheaper, but software, wow – there’s always something new!

The reality is different. True software change, whether in methods or bodies of code, is slower than glacial.

The Non-Evolution of Software Categories

It’s hardware that undergoes dramatic change, not software. Think Moore’s Law. Think about mobile phones just ten years ago. Here are details. By contrast, change in software is actually glacial. Yes, the surface weather of software changes rapidly and unpredictably; but the underlying principles and structure of software don’t even move as fast as glaciers – it’s more like tectonic plates.

Things are changing all the time.  Software is certainly “soft” compared to hardware. But programmers spend most of their time in their minds, not wandering around touching up the hardware. What do they spend time thinking about? It’s not the hardware – it’s the software environment in which they work, and in which the software they create will live.

Think about kids in a sand box. The sand is something they can push around, pile up, makes holes in, and generally have fun with. How about the box the sand is in? It’s hard. It keeps the sand in one area. No kid thinks about it much – it’s just something that’s there when you’re playing.

It works the same in software. The “part” of the software you are playing with (a.k.a. “programming”) seems “soft” and easily changeable, like sand. The “part” of the software you’re not playing with seems “hard” to you. It’s just part of the landscape. This effect is strongest when the programmer has no access to the source code of the landscape software. An example is a DBMS or an operating system. These things feel like immutable reality. They are what they are, and you have to deal with them. Hardware speed and storage capacity has grown so much, for example, that DBMS's are a data storage solution to a problem that no longer exists. But most software design continues to be built on them.

Just as other pieces of software feel like given realities if you’re a programmer, so does the programming language in which you’re working. The verbs and the syntax are as real as the grains of sand. Just as when you play with sand, you have no sense that each grain is actually a very large aggregation of molecules, so when you use a verb in the programming language, you have no sense that a compiler or interpreter will actually use many machine language instructions to execute each verb.

The “hardness” of software to which you have no source code access (or otherwise no ability to change) is a strong psychological effect.

The Glacial Evolution of Software Applications

Programmers have a special view of software. Regular people also use software personally and/or professionally. While society gives us the impression that the software people use changes like crazy, the reality is that people learn new things slowly, and successful software evolves very slowly. The changes we see are largely incidental things that are enabled by the astounding advance of hardware.

The Bedrock of Production Systems

Large bodies of working, production code are the bedrock on which fancy new layers may be built, but change very slowly. Here's an example of the impact of that hard-to-change code.

It isn’t as though people don’t want to change these systems. Often they are desperate to change them! Sometimes managers will put huge investments into changing or replacing them, only to quietly admit failure and stop trying. Until the next bunch of ignorant-of-history managers come along.

These bedrock systems aren't just systems software like DBMS's. They are also plain old applications. For example, the software that handles the largest number of credit card accounts in the world is written in the prehistoric language COBOL. Several attempts were made to re-write card processing software using fancy new languages. Failed. See also this and this.

Conclusion

Hardware changes with amazing speed, giving no signs of slowing down. While there's always something new to talk about in software, it changes much more slowly than hardware, and is remarkably hard to change.

I have described the concept of automation depth, which goes through natural stages starting with the computer playing a completely supportive role to the person (the recorder stage) and ending with the robot stage in which the person plays a secondary role. I have illustrated these stages with a couple examples that illustrate the surprising pain and trouble of going from one stage to the next.

Unlike the progression of software applications from custom through parameterized to workbench, customers tend to resist moving to the next stage of automation for various reasons including the fear of loss of control and power.

Automation depth in Information Access

Each of the patterns of software evolution I've described are general in nature. I’ve tried to give examples to show how the principles are applied. In this section, I’ll show how the entire pattern played out in “information access,” which is the set of facilities for enabling people to find and use computer-based information for business decision making.

Built-in Reporting

“Recorder” is the first stage of the automation depth pattern of software evolution. In the case of information access, early programs were written to record the basic transactions that took place; as part of the recording operation, reports were typically produced, summarizing the operations just performed. For example, all the checks written and deposits made at a bank would be recorded during the day; then, at night, all the daily activity would be posted to the accounts. The posting program would perform all the updates and create reports. The reports would include the changes made, the new status of all the accounts, and whatever else was needed to run the bank.

At this initial stage, the program that does the recording also does the reporting. Reporting is usually thought to be an integral part of the recording process – you do it, and then report on what you did. Why would you have one program doing things, and a whole separate program figuring out and reporting on what the first program did? It makes no sense.

What if you need reports for different purposes? You enhance the core program and the associated reports. What if lots of people want the reports? You build (in the early days) or acquire (as the market matured) a report distribution system, to file the reports and provide them to authorized people as required.

Efficiency was a key consideration. The core transaction processing was “touching” the transactions and the master files; while it was doing this, it could be updating counters and adding to reports as it went along, so that you wouldn’t have to re-process the same data multiple times.

Report Writers

The “power tool” stage of automation depth had two major sub-stages. The first of these was the separation of reporting from transaction processing. Information access was now a key goal in itself, and was so important and done so frequently that specialized tools were built to make it easy, which is always the sign that you’re into the “power tool” phase.

This first generation of power tools were specialized software packages generally called “report writers.” The power tool was directed at the programmer who had to create the report. Originally, the language that was used for transaction processing was also used for generating the report. The most frequent such language was COBOL. The fact that COBOL was cumbersome for this purpose was reflected in the fact that specialized syntax was added to COBOL to ease the task of writing reports. But various clever people saw that by creating a whole new language and software environment, the process of writing reports could be tremendously enhanced and simplified. These people began to think in terms of reporting itself, so naturally they broke the problem into natural pieces: accessing the data you want to report on, processing it (select, sort, sum, etc.), and formatting it for output.

The result of this thinking was a whole industry that itself evolved over time, and played out in multiple environments and took multiple forms. The common denominator was that they were all software tools to enable programmers to produce reports more quickly and effectively than before, and were complete separate from the recorder or transaction processing function.

At the same time, data storage was evolving. The database management system emerged through several generations. This is not the place for that story, which is tangential to the automation depth of information access. What is relevant is that, as the industry generally recognized that information access had moved to the report writer stage of automation, effort was made to create a clean interface between data and the programs that accessed the data for various purposes.

Data Warehouse and OLAP

Report writers were (and are) important power tools – but they’re basically directed at programmers. But programmers are not the ultimate audience for most reports; most reports are for people charged with comprehending the business implications of what is on the report and taking appropriate action in response. And the business users proved to be perennially dissatisfied with the reports they were getting. There was too much information (making it hard to find the important things), not enough information, information organized in confusing ways (so that users would need to walk through multiple reports side-by-side), or information presented in boring ways that made it difficult to grasp the significance of what was on the page. And anytime you wanted something different, it was a big magilla – you’d have to get resources authorized, a programmer assigned, suffer through the work eventually getting done, and by then you’d have twice as many new things that needed getting done.

As a result of these problems, a second wave of power tools emerged, directed at this business user. These eventually were called OLAP tools. The business user (with varying levels of help from those annoying programmers) had his own power tool, giving him direct access to the information. Instead of static reports, you could click on something and find out more about it – right away! But with business users clicking, the underlying data management systems were getting killed, so before long the business users got their own copy of the data, a data warehouse system.

In a sign of things to come, the business users noticed that sometimes, they were just scanning the reports for items of significance, and that it wasn’t hard to spell out exactly what they cared about. So OLAP tools were enhanced to find and highlight items of special significance, for example sales regions where the latest sales trends were lower than projections by a certain margin. This evolved into a whole system of alerts.

Predictive Analytics

OLAP tools are certainly power tools, but the trouble with power tools is that you need power users – people who know the business, can learn to use a versatile tool like OLAP effectively, and can generate actions from the information that help the business. So information access advanced to the final stage in our general pattern, the “robot” stage, in which human decision making is replaced by an automated system. In information access, that stage is often called “predictive analytics,” which is a kind of math modeling.

As areas of business management are better understood, it usually turns out that predictive analytics can do a better, quicker job of analyzing the data, finding the patterns, and generating the actionable decisions than a person ever could. A good example is home mortgage lending, where the vast majority of the decisions today are made using predictive analytics. Many years ago, a person who wanted a home mortgage would make an appointment with a loan officer at a local savings bank and request the loan. The officer would look at your information and make a human judgment about your loan worthiness.

That “power user” system has long since been supplanted by the “robot” system of predictive analytics, where all the known data about any potential borrower is constantly tracked, and credit decisions about that person are made on the basis of the math whenever needed. No human judgment is involved, and in fact would only make the system worse.

Predictive analytics is the same in terms of information utilization as the prior stages, but the emphasis on presenting a powerful, flexible user interface to enable a power user to drive his way to information discovery is replaced by math models that are constantly tuned and updated by the new information that becomes available.

Sometimes the predictive analytics stage is held back because of a lack of vision or initiative on the part of the relevant industry leaders. However, a pre-condition for this approach really working is the availability of all the relevant data in suitable format. For example, while we tend to focus on the math for the automated mortgage loan processing, the math only works because it has access to a nationwide database containing everyone’s financial transactions over a period of many years. A power user with lots of experience, data and human judgment will beat any form of math with inadequate data; however, good math fueled with a comprehensive, relevant data set will beat the best human any time.

Conclusion

All these stages of automation co-exist today. One of the key rules of computing is that old programs rarely die; they just get layered on top of, given new names, and gradually fade into obscurity. There are still posting programs written in assembler language that have built-in reporting. In spite of years of market hype from the OLAP folks, report writing hasn’t gone away; in fact, some older report writers have interesting new interactive capabilities; OLAP and data warehouses are things that some organizations aspire to, while others couldn’t live without them; finally, there are important and growing pockets of business where the decisions are made by predictive analytics, and to produce pretty reports for decision-making purposes (as opposed to bragging about how well the predictive analytics are doing) would be malpractice.

Even though all these stages of automation co-exist in society as a whole, they rarely co-exist in a functional segment of business. Each stage of automation is much more powerful than the prior stage, and it provides tangible, overwhelming advantages to the groups that use it. Therefore, once a business function has advanced to use a new stage of information access automation, there is a “tipping point,” and it tends to become the new standard for doing things among organizations performing that function.

Computers and software are all about automation. See this for the general principles of automation. When you dive into details and look at lots of examples, patterns emerge. The patterns amount to sequences or stages of automation that emerge over time, with remarkable consistency. When you apply your knowledge of the pattern to the software used in an industry at any given time, you can identify where the software is in the sequence. You can confirm the earlier stage in the sequence and predict with accuracy the stage that will follow. This gives you ability to say what will happen, though not when or by whom.

The patterns of automation play out in multiple dimensions. I have described what may be called the depth of automation, in which software evolves from recording what people do through helping them and eventually to replacing them.

In this post I will describe another dimension, which may be thought of as the breadth of automation. The greater the automation breadth the more functions are incorporated into the automation software and the more highly integrated the functions are with each other.

The Dimension of automation breadth

Automation breadth has these basic levels, independent of the automation depth of the products involved:

Components rarely appear first in historical terms, but they are the beginning of this sequence. Typically, functionality that is very difficult to write or whose requirements change rapidly is separated out and delivered in the form of a component. The quality of the component may become so important that it becomes an industry standard.

Example: The Ocrolus service (disclosure: Oak HC/FT is an investor) is a classic example of a valuable, narrow component. It takes images of documents of nearly any kind, recognizes them, extracts their data and returns the data to the component user. This functionality is so challenging to write, and the consequence of errors so great, that most applications that need the functionality are likely to use the component, which is delivered as a service, rather than write their own.

In a time of rapidly emerging functionality, point products are usually first, because they can be gotten to market quickly. Buyers typically talk in terms of “best of breed,” but have the problem of negotiating and maintaining relationships with multiple vendors, who frequently have conflicting interests. The buyer also has to take responsibility for integrating the various products he has bought. Buyers reasonably worry about whether their typically small vendor will stay in business and continue to invest in their product.

Example: Captura (now part of Concur) provided a point product to enable a company to automate the process of entering, approving and paying expense reports.

Product collections solve some of these problems. There is now a single vendor; the vendor is probably much larger and more likely to stay in business; the products will be maintained and there is no conflict of interest. Once a product collection becomes available in a category, buyers will typically prefer an adequate product from a collection to a superior point product. Product collections can be formed by acquisition.

Example: CA, Computer Associates, is a typical vendor that acquires products in a category and puts them together into a product collection.

It is often desirable to share data among the products in a collection. Frequently, they maintain copies of essentially the same information, have essentially the same security roles defined, etc. Users have to go through considerable time and effort to accomplish this on their own, and then again when there is a new release, and desirable integrations are sometimes not even possible. Product suites solve these problems to a large extent, since a single vendor performs the integration and sells the integration as part of the product collection. Once a product suite becomes available in a category, buyers will typically prefer an adequate product suite to a product collection whose products are superior, because the cost of installation and maintenance is lower and the benefits of integration often outweigh the benefits of individual product features. Building a product suite typically requires source-code level coding and functionality design changes, but the code bases of the products can be separate.

Classic example: Microsoft Office was one of the first product suites for office products. While the benefits of integration are not overwhelming in this functional area, users clearly benefit from having a suite rather than individual products. Once the benefits of a suite became accepted, buyers no longer wanted individual office products.

Recent example: The market for business formation has long been dominated by the point product Legalzoom. A new, rapidly growing product suite called Zenbusiness (disclosure: Oak HC/FT is an investor) is taking classic advantage of moving to the next stage of automation breadth, by offering small business customers not only business formation services, but also services for websites, banking and accounting.

Not all functionality areas benefit from having an integrated product, but for those that do, integrated products are better for vendors (reduced costs due to elimination of redundancy) and for buyers (a simpler, more unified product that is easier to learn and operate, and has deep, fully-automated function integration), and typically win in the marketplace.

Example: Everyone thinks of SAP’s R3 as being the first client/server ERP product, but its real break-through was being the first truly integrated product on which a large enterprise could run its business. Using a single body of code and a common shared DBMS, its many modules each ran different parts of the enterprise business. In spite of the high cost of implementation and operation, the advantages of running your business on a single integrated product were overwhelming. Like most projects to build a unified product, the vendor had deep domain knowledge, it took a long time to get it right, and the early installations were painful.

Once functionality markets reach a level of maturity, the competition becomes intense, and organizations often have to choose areas of distinction, outsourcing the functions they choose not to compete in to a low-cost provider. Products that enable organizations to selectively outsource in this way typically take business from products that must be fully staffed by the buyer.

Example: FDC is the leading processor for credit cards in the US. With a billion cards outstanding, the market is huge and highly competitive. While the technology base of the FDC product is decades old, the functionality it delivers is highly sophisticated. In addition to delivering a completely integrated, multi-department product, FDC offers off-site staffing for the various departmental functions, where the staff sounds on the telephone as though they worked for you.

Conclusion

The dimension of automation breadth helps us understand the evolution of software and the progression that naturally takes place in a given market space. The companies that win are most often the ones that dominate a narrow market segment with a particular product and then broaden the range of software they can sell to a given organization. They typically move step by step according to the progression I have described here.

I have described the concept of automation depth, which goes through natural stages starting with the computer playing a completely supportive role to the person (the recorder stage) and ending with the robot stage in which the person plays a secondary role. I will illustrate these stages with a couple examples that illustrate the surprising pain and trouble of going from one stage to the next. Unlike the progression of software applications from custom through parameterized to workbench, customers tend to resist moving to the next stage of automation for various reasons including the fear of loss of control and power.

I will use companies to illustrate this progression that are known to me personally but not widely known. Software history generally is the history of the winners as written by the winners, like most history. However if you want to guide your own software strategy by taking advantage of the clear patterns of history, it is essential to include examples of companies that are rarely discussed.

Nextpage

NextPage was active from 1999 to around 2010. It had a distributed document management product that was prominent in service industries, for example lawyers and accountants. The product operated at a “recorder” level in terms of automation depth. It wasn’t directive, it just sat there waiting for a document request, and when it got one, responded with a list of applicable documents and lets you view the documents.

They and some of their larger customers saw that, in fact, many of their engagements weren’t so very different. As a high-powered, highly paid lawyer, you may think that every job is unique and your ability to handle the differences very important, but it was obvious to people involved that certain transactions involved very similar documents and workflows. It made sense to attempt to capture and automate the similarities.

With the cooperation of one of the world’s largest law firms, NextPage built a new product on top of the existing distributed document platform; this new product was automation at the “power tool” level. While recognizing that each transaction of a particular type would end up producing a unique set of documents, for example, it operated like a series of factory distribution belts, assuring that all lawyers working on an M&A transaction, for example, worked on the right things in the right order with the right base documents, reviewed in the right order by the right people, etc. etc. Lawyers continued to craft their documents. But instead of treating each (for example) M&A transaction as though it were the first one the firm had ever handled, the new system treated them like a repeatable process, with variations.

But the lawyers didn’t think of it that way at all! They were used to thinking of themselves as powerful, autonomous, self-directing agents. Now they were to be subservient to an assembly line in a factory, having their work and output measured as though they were hourly workers? No way – I’m a professional – there’s no way I’m going to put my head in that yoke!

NextPage had a good position in the market, existing customers, a prestigious lead customer for the product, a strong business case, and tangible results. But the resistance to moving to a “power tool” level of application was stronger than the force NextPage was able to bring to bear at that time, and the effort failed to bear fruit. Anyone who looks at the application knows it’s going to happen, the benefits are definitely there. But anyone with sense would be reluctant to predict even the decade when it would be likely to actually happen.

The pattern of resistance encountered by Nextpage was strongly correlated with the prestige and power of the people whose work would be affected. The same resistance is seen elsewhere; for example, how eager are the highly paid and prestigious category of medical doctors to have their work automated?

Captura/Concur

My VC firm invested in Captura, which is now part of SAP Concur, the leader in on-line expense management systems. These applications basically automate the process of filling out expense reports, getting them checked and approved, and finally paid.

If you think about expense reports, you imagine a form that you fill in with your expenses during a trip. You then attach your receipts, drop it off, and eventually get reimbursed for out-of-pocket expenses. So it seems like it would be a “recorder” type application. In fact several companies were started and eventually failed with products that weren’t much more than recorder type applications, with some automatic routing added on. This kind of application didn’t provide enough value to offset the expense of installing and running it.

Captura went much farther. While parts of this application operate at the “recorder” level, particularly the entry of un-automated expense information, much of it actually operates at the “robot” level, which is why the payback for its use is so high. It has built-in rules for approval levels and documentation requirements and many other things. It knows whether and when and to whom expense reports should be routed for approval from the HR system. It gets feeds from the corporate card system. It kicks out exceptions. It allocates expenses to the right accounts. It feeds directly into the accounts payable system to cut checks. When it needs something it can’t get from a computer, it asks a human for it, and otherwise does its job without help.

Unlike other “power tool” type applications, this application tended not to threaten people who were powerful in the adopting organizations. The most powerful people who used it tended to regard it as saving them time, and everyone got their reimbursements more quickly and accurately. The administration group saw it as saving thankless clerical work, and the accounting executives saw automatic enforcement of all their standards. Still, this application category didn’t really take off until the cost of implementation was reduced to a painless level; in other words, it had to wait for nearly universal access to the web to be practical and affordable. And it worked much better as a secure service than as an enterprise application, because the burden of installation and maintenance on IT added so much friction that only the most motivated potential customers would go for it.

ClickSoftware

ClickSoftware provides a classic example of a “robot” level application. It takes the problem of a field service operation, in which there are calls for service from companies in different locations, with different equipment to be repaired, with varying levels of urgency and service level agreements, with different hours of operation and times to travel to them, and finally with different service technicians who have varying skills and qualifications and constraints about hours of work. Who gets sent to which location to fix which problem in which order?

Without something like Click Software, this problem is solved in a rudimentary way by the service manager, who uses some combination of experience and common sense to work out the best solution. However, for a large service organization, the difference between a good solution and the optimal solution can be worth a great deal of money. Click Software takes all those inputs and produces the optimal solution, and asks the human manager for help when the problem simply can’t be solved; in response, for example, the human may ask a critical resource to work overtime, or might call a customer and see if their request can be deferred.

Click illustrates another characteristic common to robot-type applications. It is typical that “power tool” type applications do not become robot type ones by incremental addition. A robot application requires a whole new approach to the problem, and a level of math programming that is likely to be entirely foreign to the software group that wrote the power tool application.

Tape backup systems

In tape backup, whenever a new platform has come out, tape backup software products tend to emerge in the same order, starting with products that just record what the backup operator does, moving to power products that give the operator hints and suggestions and automate the repetitive operations, and ending with true robot products, which are driven by a set of rules and which request operator intervention at only and exactly those times when human intervention is required. For example, Palindrome and Cheyenne (now part of CA) introduced robot-level backup products in the early 1990’s for the PC server platform, at a time when similar products had been in general use on earlier platforms (e.g. IBM mainframe) for decades.

Having cycled from primitive to robot level automation several times as new platforms came out that had tape backup, the whole category is dying, frozen in time as the latest computer systems use the kind of disks that have become incredible cheap and tape has become obsolete.

Conclusion

It seems inevitable that an industry would move, step by step, towards robot-level automation. The historical fact is that while the incentives for advancement are always there, advancement happens only when an industry is "ready" to move, which can be decades after it is technically possible and economically practical.

I have discussed the fundamental concept of automation.

Software automates human effort to varying degrees. In doing so, software emerges that performs this automation to an increasing extent. In this post I'll describe the basic stages through which automation progresses. In later posts I'll give specific examples.

Automation depth

Automation depth can be understood in terms of the extent to which human effort, observation, knowledge and control is replaced by the computer system.

Stages: help a human do something; do something a human would otherwise have to have done; make a decision a human would have made; gather information about all the work that is arriving, in process and completed; make work and resource allocation decisions. There are a few different  ways automation depth can be understood. Here are a couple of them:

• Stages: As depth increases, the human does things faster and has less work to do.
• Automation depth is correlated with moving from an “open loop” system to a “closed loop” system.
• Effort. In the earlier stages the computer augments or replaces the person’s efforts. This is like power steering in a car, in which everything is the same except turning the wheel takes less effort.
• Knowledge. As automation depth increases, less knowledge is in the heads of people and more in the computer.
  
• Control. As automation depth increases, the human loses control and eventually becomes controlled by the computer.

Automation depth has these basic levels:

Unlike some of the software evolution progressions I've described, iIt is not generally a quick win to get to the next level of automation depth. An industry can stay stuck at a level of automation depth for decades. Normally, the people directly involved with an application strongly resist moving to the next level of automation depth, because they see it as deeply threatening to their autonomy, power or skills.

If a company tries to go to the next level, it is essential that it understand its position, and have the backing to last out the transition and the flexibility to keep trying until it establishes a beach-head from which it can expand. The benefits have got to be dramatic and tangible. Feel-good stuff tends not to work here. Some people are going to get bent out of shape, and managers of managers have got to impose the solution. The only way that typically happens is if things are arranged for their risk of failure to be low and the rewards great, and indisputable when they actually happen.

I will give examples of this progressions in future posts.

I have described the way that once an application appears on a software platform, there is a consistent way the category of applications evolves on that platform. In this post I'll describe examples of this pattern. Briefly, the sequence starts with a prototype, and goes through these stages:

• Custom application
• Basic product
• Parameterized product
• Workbench product

There can be several levels of sophistication at each of these levels. The sequence is important because each step increases the speed and decreases the cost of making a body of code meet a set of customer needs.

Metasys (bought by Optum), Syntra (Clearcross)

These were medium-sized software companies, with revenues of tens of millions of dollars a year, which practically no one would have any reason to know existed today. Both these companies had a “custom application” that they were attempting to upgrade to a “basic product.” Both code bases evolved from large projects that were done on a consulting basis for a particular customer. Because both companies marketed what they had as though they were real products and failed to put the intellectual energy and money into properly upgrading their code bases, they were always coming up short in terms of features in sale situations, and when they were able to get sales, the implementation and customization efforts were harrowing to everyone concerned. The plain fact was, everyone was excited by the vision of how lovely it would be if their code were at the level of “basic product” and built a sales effort as though it were, but in the end, what each had was a “custom application” that had been severely hacked and “marketectured.”

Aurum

Aurum had a “basic product” for the emerging CRM market. At the time I looked at them with my VC firm, they were trying to bring it to the parameterized level. Because of the need for extensive customization in the CRM market, most of their implementations involved source code modifications, which made support and upgrading to new releases a major challenge. We initially chose not to invest because the company failed to recognize this. Later, with new management, the company at least recognized the issue, took steps to correct it and made business arrangements to minimize the impact on customers. We then tried to invest, but didn’t have the opportunity. The company went public and was a good win for its investors, but failed to get its product to the next level, and ended up selling to what at the time was one of the major ERP companies, Baan.

Paysys VisionPLUS

This credit card processing product was a “parameterized product,” and met all the relevant criteria. The company gained ground against the competition before and during the time I was CTO of the company because it fully met the criteria for “parameterized product,” while the competition was somewhere between that and “basic product.” Customers were able to make the product perform many more functions without source code changes due to this fact, an advantage the market appreciated. If the parameters that are made available correspond to the kind of changes you want to make, a parameterized product is ideal, and worlds better than a basic product.

The VisionPLUS product had its origin in two earlier products that were created by the company. The first of these products was CardPac, a fairly parameterized product for processing bank cards, like Visa and MasterCard. The product enjoyed a good deal of success at a time when most such card processing was performed by custom applications, which only the largest institutions could afford to write. Cardpac was sold to smaller banks, and parameters were introduced to reduce the cost of customization, maintenance and support.

A couple of retail institutions approached the company and asked it to modify Cardpac so that it could process retail (closed-loop) cards. After a couple of failed attempts, the company produced a completely new product for this purpose, Vision21. This product enjoyed considerable success among high-end retailers.

Finally, there was very large processor, Household International, that was running multiple copies of both products, separate because they had been customized for a variety of reasons, for example to support methods of credit that were unique to a market (for example “hire-purchase” in South Africa). While Paysys had failed to create a generic bank/retail product when confronted with the generic problem, unifying multiple bodies of related code into a single, highly parameterized code base proved to be a far more tractable problem, particularly with a single important customer who insisted that these variations were the only ones to worry about. The unified product was called VisionPLUS.

The industry quickly rallied to this new product that could be directed at so many different problems with such relative ease. While “parameterized product” may sound like an abstract concept, it translates directly into business advantage compared to more primitive product types, by enabling the product to be customized, installed, upgraded and maintained with less labor, less time and lower risk of error.

Paysys was bought by a major card processor, First Data, when First Data needed to expand into international markets with tough requirements. First Data's code base at the time was written in assembler language and was pretty much at the basic product level with minimal parameterization. Buying the Paysys COBOL code with its parameterized power was a major step forward for them. The number of accounts managed by VisionPLUS expanded by a factor of four under its new owner, to over 600 million.

Pivotal

Pivotal was early in the CRM market with a workbench-type product. Not all markets value this implementation method equally. CRM is one of the markets that values it most highly, because nearly every installation needs to undergo substantial customization. The benefits to customers were the ability to migrate their applications to new platforms as they emerged (e.g., Pivotal automatically converted Windows apps to browser apps) and the ability to extensively alter data, screens and application logic without source code changes. These characteristics took a good deal of effort on the company’s part to explain to customers. Their early efforts were much too technical, and didn’t resonate. Once they reached a level of market penetration, however, reference selling was the key. A potential customer would talk with an existing Pivotal customer who had first installed someone else’s application that seemed similar to Pivotal’s, which in fact it probably was, as it came out of the box. Then the reference would talk about all the time, effort and money to make simple-sounding customizations. Out would go the competing product, and in would come Pivotal. Suddenly customizations were fast, low-effort and low-cost. End of sales effort. Take order.

This implementation worked in a text-book manner for Pivotal, and is typical of the pattern. The company went public in 1999. However, its further growth was greatly limited by the fact that they chose to operate exclusively within the confines of the Microsoft Office product suite.

ERP products

The major ERP products have been at various stages of parameterized and workbench for a number of years. At one point, SAP seemed to be on the way to dominating the field, and armies of consultants were engaged in years-long projects to use SAP’s proprietary language, ABAP4, to modify and customize the base system to meet customer needs. Then along came PeopleSoft, originally just a vendor of HR software, with a complete ERP suite. While the software was not as richly functional out of the box as SAP’s, everyone knew by this point that no one used the software out of the box anyway – you had to wait for years while endless customization took place. What PeopleSoft had that was different was a set of development tools, PeopleTools, which was a generation ahead of SAP’s. SAP’s ABAP4 was at the time your basic 4GL, and you could program nearly anything with it, but you did have to spend a great deal of time programming. The PeopleSoft alternative, while no technical break-through, was still miles ahead of SAP in terms of ease of use and programmer productivity; they actually had a screen painter!

PeopleSoft convinced many buyers to care more about the implementation level of the product than the base functionality, and convinced those buyers that they were farther along the scale to a truly workbench product (although that term was not used) than the alternatives. As a result, they enjoyed years of outstanding growth. They were bought by Oracle for over $10 billion.

Conclusion

While not the only factor in success, companies whose products are higher up the tree of abstraction than their competitors enjoy a clear strategic advantage over their customers. As products evolve on a platform, the ones that appear or migrate to new levels of product abstraction tend to be more successful than ones at a lower level. While the discussion within a company is often about this or that customer or feature, raising the discussion to a new level of abstraction and acting on it can provide a huge competitive boost to a company's fortunes.

This is the fourth in a series of examples to illustrate the way that functionality that had been implemented on an older platform appears on a newer platform.

See this post for a general introduction with example and explanation of this peculiar pattern of software evolution. This earlier post contains an example in security services software, this earlier post describes an example in remote access software and this earlier post describes an example in market research.

Unlike the prior examples, this one is well known and I had no personal involvement.

Example: BEA systems

This is a classic example of the pattern of functionality emerging on a technology platform, and then emerging in pretty much the same way in the same order on other platforms.

IBM mainframes were originally used in a batch processing mode. They had operating systems that were really good at it. Groups of users increasingly wanted on-line access, and they wanted transaction processing control and security. The operating system didn’t provide it. So they built what the operating system thought of as an application, but which special applications that ran with it thought of as a specialized kind of operating system, a “transaction processing monitor.” Over time, the value of the main IBM TP monitor, CICS, became recognized, and it evolved to quasi-operating system status.

UNIX gets developed by a bunch of smart people at Bell Labs, and eventually becomes widely deployed on many kinds of computers. It was originally developed with interactive use in mind, and the early UNIX people would have been offended by the idea that it would need to be augmented by a primitive thing like a TP monitor. But that’s exactly what happened, and the UNIX-oriented TP monitors were built along very similar lines and for similar reasons as CICS!

The original authors of Unix ignored TP monitors when they built it in the 1970's, but AT&T had its own version of a TP monitor that ran on IBM mainframes to control its network. AT&T decided it would be cheaper to use their own computers instead of IBM mainframes to run its control software. Their computers had the Unix operating system, but transaction wouldn't work properly without a transaction monitor. So they set about building their own specifically to run on top of Unix, Tuxedo starting in 1983. It was successful and became widely used.

The founders of BEA were ex-Sun Microsystems employees who were amazingly prescient in seeing this need. But they didn't actually build anything! They started by buying Tuxedo, the Unix-centric TP monitor that had been owned by Novell since 1993. They played with other middleware companies but most importantly bought WebLogic, a TP monitor built specifically for Sun's Java language.

It was the availability of Unix-based TP monitors that helped enable the massive growth of the Unix platform during the internet explosion in the late '90's and early 2000's. I was active during this period, both with CICS applications and internet applications. The whole internet tech world, techies and investors, became convinced that certain things were required to build an application with that all-important characteristic, "scalability." It had to run on Sun computers, with the Sun version of Unix. It had to use the Oracle DBMS. And it had to be written in the Java language using the J2EE (usually spoken as jay-two-double-E) library, intended to ride on … a TP monitor! Oracle ended up being the winner; it first acquired BEA in 2008 and shortly after acquired Sun.

The key, just as it was for CICS, was the integrity of transactions when many people are accessing a shared set of data. UNIX simply did not provide this capability, and just like on the mainframe, the capability was added on top of the operating system, with an application that the operating system thought of as a normal application, but which the applications that made use of it thought of as a specialized kind of operating system.

You might well think that this whole set of functionality is dead today. Who talks about TP monitors? Many companies showed that you don't need all the complexity and overhead of a TP monitor to build scalable applications.

In reality, it's a clear example of another strong, repeating pattern: bad ideas in software rarely die; they just fade for a bit and then re-emerge with a new name in slightly different form. Here's a good example of the morphing of data warehouse into Big Data.

I won't go into detail here, but while J2EE and classic Unix TP monitors are on life-support, the same idea of distributed applications for scalability lives on today in the form of microservices woven together with an enterprise message bus. Even though, just like 20 years ago, they're not needed for building scalable applications!

This is the third in a series of examples to illustrate the way that functionality that had been implemented on an older platform appears on a newer platform.

See this post for a general introduction with example and explanation of this peculiar pattern of software evolution. This earlier post contains an example in security services software and this earlier post describes an example in remote access software.

This example is known to me personally because my VC firm was an investor, and I was involved with them through the life of the investment. 

Example: Knowledge Networks

Organizations that depend a great deal on the opinions and actions of a large number of people sometimes conduct market research to help them shape the details of a product, an advertising campaign, a political campaign or other relevant effort. This kind of research long pre-dates computers. It normally starts using informal methods for selecting the people to ask and evaluating the results. But then it moves in stages towards increasing amounts of scientific control and analysis in order to reduce costs and improve accuracy.

Market research was already well-established on the prior technology “platform” of the telephone when the internet started to spread quickly in the second half of the 1990’s, when a substantial and growing fraction of the US population got internet access. People in the field were familiar with the issues of selection bias, people without telephones and random digit dialing methods of assuring statistically valid panels. But when the web (the new platform) started spreading quickly, did market research transfer its knowledge, methods and techniques to the new platform? It didn’t (I think you probably guessed the answer), because brand-new people put together the first web-based market research systems. It was quick, easy and had the advantage of being inexpensive – but it was as scientifically primitive as telephone-based surveys were prior to the introduction of statistical methods.

Knowledge Networks was started by a couple of professors with stature in market research in the pre-internet world, with the goal of keeping the cost and speed advantages of the internet, but bringing it up to pre-internet scientific standards.

While there has definitely been a migration of internet-based market research to higher levels of scientific standards, which Knowledge Networks has both led and benefited from, their experience is an example of the danger of getting too far ahead of the pattern. One of the key facts about the “emergence of functionality on a new platform” pattern is that the functionality emerges in the same order as it did on the earlier platforms – but it doesn’t skip steps or leap right to the end! These professors knew that internet-based market research would evolve to greater scientific integrity – and they were right! – but they didn’t fully appreciate that the market would get there in its own sweet time, and that it would insist on dawdling on intermediate steps. By insisting that Knowledge Networks provide only the best, highest-integrity market research methods, up to the standards of the best available on earlier technology platforms, the original leaders of the company caused it to be “out of step” with the market. They were “ahead” of the market, which is a great place to be if you want to be a business “visionary,” but is rarely a good place to be if your goal is to build a substantial business.

I have to say that this is a really hard one to get right in practical situations. I personally was involved with Knowledge Networks at the time some crucial decisions were made, but I didn’t know enough about or appreciate the power of the pattern to help make the company as successful as it could have been. In fact, I was probably part of the problem. The professors who started the company were really smart, and they were on top of all the issues of market research. I knew this, appreciated it, and was excited by the possibilities of benefiting by translating the best practices from traditional market research to the internet. What’s painful is that I also knew in general terms the dangers of being ahead of a market. But that’s exactly what we were, and yet again, for the umpteenth time, I didn’t see it and didn’t call it. Arghhh!

Lesson: being too early is just as bad as being too late.

This is the second in a series of examples to illustrate the way that functionality that had been implemented on an older platform appears on a newer platform.

See this post for a general introduction with example and explanation of this peculiar pattern of software evolution. This earlier post contains an example in security services software.

This example is known to me personally because my VC firm was an investor, and I was involved with them through the life of the investment.

Example: Aventail

Aventail built functionality for remote access that has been implemented over and over again, each time a new technology platform has emerged. But they rode what was at the time the latest wave (internet protocols and SSL encryption), and so were participating in a growing market.

I remember using teletype paper terminals running at 110 baud in the late 1960’s for remote access to computers. Whenever a new platform would come out, the new technology wouldn’t support remote access, but for some strange reason, people would want it! So, focused entirely on getting something working in the new environment, and either ignoring or simply being ignorant of earlier solutions to the same problem, someone would build a remote access solution. But then inadequacies would be found, and a release two would come out. All in what appears to be ignorance of solutions built on prior platforms, blind to their lessons-learned..

A good example is the identification and access control system for remote access. The system you want to connect to has some system for user ID’s and passwords, and then some method of access control based on user groups. The remote access is normally first built in the simplest possible way, having its own system administration, user identification and access control. As the use of the system grows, this parallel administration is a burden, and so some level of integration with the core security system is then implemented. The pattern is that the separate system is normally built first; the need for integration is “discovered;” the integrated control systems are supplied in a later release.

When you see this pattern of stupidity and ignorance for the first time, you scratch your head. These are programmers and experienced product people! How could they have missed such an obviously valuable feature of the same functionality built on an earlier platform? Well, that's the pattern, as I described in detail in the first post in this series. It's a wonderful pattern — it enables anyone who understands it to predict the future with great accuracy and precision!