The Black Liszt

Category: Software Evolution

  • Software Evolution: Functionality on a New Platform: Security Services

    A whole book could be devoted to spelling out the “natural” emergence of features on a platform, and identifying and accounting for the minor variations from platform to platform (the emergence sequences don’t repeat exactly for a variety of reasons). However, the similarities are obvious and universal enough that anyone with longitudinal familiarity with a couple of comparable platforms would recognize them.

    This is the first in a series of examples to illustrate the way that functionality that had been implemented on an older platform appears on a newer platform. All the examples illustrate the point that, even though the functionality is there for anyone to see on the older platform, working and delivering business value, it only appears at its “proper time” on the newer platform.

    See this post for a general introduction with example and explanation of this peculiar pattern of software evolution.

    I have mostly selected companies that most readers may not be familiar with, to demonstrate the strength of the pattern and the way it affects all participants in a market, not just the well-known, mainstream companies. Also, I know them personally, and so can tell their stories from personal knowledge, instead of just repeating things I’ve read.

    Example: Securant

    Old platform

    IBM mainframe

    Old function

    Security services: ACF2, RACF

    New platform

    Web server

    New function

    Security services for Web applications were extremely basic, and focused on message encryption. Securant implemented mainframe-class user and application security. The same kind of companies that depended on mainframes for transaction processing were very attracted to Securant’s approach to Web applications.

    Outcome

    Terrific product and sales traction despite product inadequacies and the company’s internal problems. The company was acquired for a good price by RSA Security in 2001.

    Securant was started by a couple of young programmers in their 20’s as a services business. They were hired by a large financial institution that was adding internet/web infrastructure to their IT infrastructure. Naturally, the web applications needed access to the programs and data on the mainframe systems, and those systems were protected by state of the art security systems. The financial company would have preferred to be protected by a security facility for the web just like it did for the mainframe, but none was available. So they agreed with these guys to build one.

    The young programmers knew nothing about mainframes, and never bothered learning anything much about them. As far as they were concerned, mainframes were pretty useless things well on the way to becoming obsolete – why become an expert in steam engines in the 1930’s when internal combustion engines are obviously the future? So they focused completely and solely on what they knew, and built a ground-up application that met the business security needs as they understood them from the users, who were also barely familiar with what the mainframe had to offer.

    Before long, the one-off services project became a very hot, rapidly growing product company. One of the many ironies in this project is that the father of one of the two founding programmers had run one of the mainframe security companies! But he had been estranged from his son for many years, and neither knew that the other was or had been involved in computer security. The father decided to come back from the retirement that had been funded by the sale of his security company, re-connected with his son, and you can imagine his amazement when he found out what that son had accomplished. The father saw the functionality he had on the mainframe, re-imagined and re-implemented for the new environment.

    One of the powerful things about this pattern is that it’s like the tide or the current in a river – at the right point, it just pushes the vendors and functionality in the direction of the “flow,” and everything ends up moving in the same direction. The people involved tend to think they’re inventing things – but what they invent is pretty predictable, because they’re responding to the same kind of needs that the previous bunch of inventors were responding to.

  • Software Evolution: Functionality on a New Platform

    When a new “platform” emerges (UNIX, Windows, Web, Apps), if you look at any application area and see how it evolved on prior platforms, the application’s functionality will emerge on the new platform in roughly the same order, though often on a compressed timescale. The functionality that is relevant depends on the particular application area. This concept applies both to system and application software.

    The pattern is: functionality emerges on a new platform in roughly the same order as it emerged on earlier platforms. The timescale of the emergence may be compressed; the important aspect of the pattern isn’t the timing but the order. The pattern means that functional steps are rarely skipped – what was next last time is also next this time. The pattern also means that when someone tries to introduce functionality too soon, before the functionality that preceded it on prior platforms is generally available, the market will not accept it.

    While this pattern takes a good deal of knowledge and judgment to notice and apply, I have consistently been impressed by its predictive power. By following the pattern, you can be pretty confident that you’re building proven functionality, and that you’re following a pattern of success.

    I have noticed a couple of danger points here. When the company is too aware of the pattern, it is easy for them to “get ahead of themselves” and more importantly ahead of the market, by solving problems that certainly will become important, but problems that the market doesn’t know it has yet. The “same order” part of the pattern is important; building ahead of the market appears to win few business benefits.

    On the other hand, without knowledge of the pattern, it is easy to make up all sorts of things you think people might want, and build them, only to find out later that you wasted time and money, because the functionality you built is never part of the accepted function set.

    Really great companies who have lots of creative people, the ability to execute, and listen closely to market reactions to what they’re doing, don’t “need” to know about this pattern. However, for the rest of us mortals down here, having a “cheat sheet” to what important features the market will be demanding next can prove awfully helpful.

    Basic Example: operating systems

    IBM famously created an operating system for their mainframe line of computers, OS/360. It had the capability of running multiple jobs at once. Its multi-tasking abilities grew and became more sophisticated through the 1970’s.

    450px-360image001

    Eventually a transaction monitor, CICS, was written and became a de facto part of the operating system for applications with lots of interactive users. As the operating system was used, it became evident that various access methods for storage and communications needed to be separate from the core, and so clear interfaces were created, and the notion of completely self-contained access methods (for example, a file system) as replaceable units was supported. A strong security system was not part of the early versions of the operating system, and the need for one became critical, and so strong external modules were written and support for security was added to the core. While there was a “main” operating system, alternative operating systems were written for various purposes, and a virtual operating system was written to actually run on the “bare metal.” With VM (the virtual OS), you could devote most of a machine’s resources to the production users, while letting some of the users running on a completely different operating system.

    While all this was taking place, people were studying and experimenting in university environments, deciding just what an operating system was and was not, what the best ways to build one were, and so on.

    Before very long, mini-computers were invented; these were basically mainframes on the cheap, with all sorts of features and functions missing – but they were cheap. And, since each had a unique instruction set, each minicomputer needed an operating system. Programmers were hired, and those programmers, of course, ignored the mainframe operating systems, and built simple, cheap OS’s to go along with the cheap hardware. Surprise, surprise, those cheap OS’s resembled nothing as much as – the first generation of mainframe operating systems! But people quickly discovered the limitations, just as they had before, and set about making the same set of enhancements that the previous generation of pioneering programmers had made. Within ten years, they had re-invented many of the important mainframe OS concepts, and were on the way to building the rest.

    With all this knowledge of operating systems floating around and pretty easily available, what do you suppose happened when people took the early micro-processor chips and made them into micro-computers? Naturally, they understood the state of the art of operating systems theory and practice and adapted an existing OS (which were starting to be built in high level languages) or built one that took all this knowledge and applied it with appropriate adjustments to the new environment, right? Bzzzzt! Of course not!

    What the kids who were faced with the task did was start from scratch, not only in terms of code, but also in terms of knowledge. They didn’t stand on the shoulders of giants; they didn’t learn from the experiences of the many that preceded them; they built OS’s as though they were the first programmers who ever tried to do such a thing. And the result was pretty much like early mainframe (even pre-360!) operating systems. There was no serious memory protection or address mapping; there was no real concept of multiple users, multiple levels and types of users, or any real security; no multi-tasking; the access methods were hard-wired in, and so on. The limitations and problems emerged pretty quickly, and so did add-on rubber band and baling wire patches, just like in earlier days.

    It’s a good thing that IBM came along at this point, and brought commercial order and education to the emerging microcomputer market. When they came out with the IBM PC, they not only legitimized the market, they had deep history with mainframes and minicomputers. They employed true experts who knew operating systems inside and out. They had a research division, where there were people who could tell you what the operating systems of the future would look like. So it makes sense they would get those experts together, and they would create a small but efficient micro-tasking kernel, common interfaces for installable access methods, and many other appropriate variations on all the modern operating systems concepts. The last thing such a smart, educated and astute major company like IBM would do was make an exclusive deal with a small company that had never built an operating system, who had just bought the rights on the cheap to a half-baked excuse for a primitive first-generation OS, and make that the IBM-blessed … Wait! … that’s what they did do! Arrrgggghhh!

    Explanation

    One might well ask, how can a pattern like this continue to have predictive power? Why wouldn’t the people who develop on a new platform simply take a little time to examine the relevant applications on the older platforms, and leap to the state of the art? Why wouldn’t customers demand it?

    It is hard to know for sure, but I think there are a couple main factors at work, and there is evidence for the relevance of each of the factors.

    The first factor is the developers. It is well known that most developers learn a platform and then stick with the platform they’ve learned for an extended period of time, basically as long as they can. The reason is simple: they are experts on the platform they already know, and therefore have prestige and make more money than they would as novices on a platform they’re just learning. I speculate that this is one of the many contributing factors to the rapid migration of ambitious programmers into management, where they can advance without being tied to a platform, at least as much. So who learns the new platforms? With few exceptions, new people. If you’re just entering the industry, you are counseled to learn the hot new languages; you tend to be sensitive to where the demands and rising salaries are. Still, you expect and are paid entry-level wages, along with most other people (except managers). Why should the experienced programmers jump to the new platform? They would have to compete with hard-working young people, their knowledge of the older platform will be considered a liability, and on top of everything else, they’d have to take a pay cut.

    The result is that no one working on the new platform has an in-depth, working knowledge of the applications on the older platform, and at least in part because of this, everyone considers knowledge and use of the platform to be vastly more important than knowledge of an old application on an obsolete platform. So they ignore it! As a result, they dive in and attempt to automate the application area “from scratch.” Their attempts are usually quite close to every first generation program for that application on past platforms, because it turns out that the determining factor isn’t the platform, it’s the business problem. They proceed to re-discover, step by step, the reasons why the first generation was inadequate and had to be supplanted by a second generation, etc.

    The second factor is the buyers. When a new platform emerges, most buyers simply ignore it. Why pay attention? It’s a toy, there are no good applications, etc. The few buyers who do pay attention tend to be crazy, early adopter types who just love the experience of trying new things. Like the programmers, they also tend to care about the platform more than the application – otherwise, they wouldn’t even consider buying what is typically a seriously immature application on the new platform. But they can only buy what’s being sold, and so they choose among the inadequate applications. Because they don’t care about applications as much as platforms, they don’t even ask for features they know could only be present in the mature applications for older platforms – they press the applications vendors for the “next” obvious cool feature, in the narrow universe of the new platform.

    The reason why application evolution repeats itself on the new platform, then, is that nearly everyone involved, builder and buyer, is ignorant of the past and wears what amounts to blinders. It’s as though they are building and buying the application for the first time. Therefore, they respond most strongly to the same business pressures that people involved in computer automation tend to see first, and then next, and so on. It’s as though there’s a “natural” most climb-able path up a mountain, and successive waves of climbers approach the climb from the foot of the mountain, completely ignorant of the experiences of those who came before them, but confronted with the same options they tend to make the same choices, and so everyone takes roughly the same route up the mountain.

    Why don’t smart groups who know all this leap-frog their way to the most advanced application functionality? I have seen this happen, and it’s the buyers who tend to rain on this kind of parade. The buyers tend to be familiar with the range of applications in a category, and those applications tend to address a highly overlapping set of problems in ways that vary only slightly. The “far-seeing” builder then comes along with all sorts of solutions to problems that the buyers don’t even know they have! The buyers look around in confusion – why isn’t anyone else talking about this? I kind of understand what you’re talking about, but I feel silly thinking that something’s important when no one else does. I think I’ll wait. And they do. So getting too far ahead of the buyers is just as much of a problem as being too far behind the competition. The result: the application evolution repeats itself on the new platform, in roughly the same order each time, and no “cheating” or “running ahead” is allowed.

    This all sounds incredibly common-sense when I read it written down, but I have to admit that this particular piece of common sense is not only uncommon, it took me personally decades and multiple failures to finally get this simple thought into my thick skull. The key thought is this one: you may think you know a person has a problem; the problem might be a severe one, and cost the person a great deal of money and trouble; you may even be entirely right in this judgment. However, if the person in question does not think he has a problem, why should he pay for a solution – in fact, why would he go to the trouble of implementing a solution even if it were free?  Even worse, why would he even waste time talking with you, once he got the idea that you thought he had a problem he didn’t think he had? Why would he listen to a chimney sweeper’s pitch if he lives in a house without chimneys? The air quality in Los Angeles in 1970 was terrible. But there was no market for catalytic converters on automobile exhaust systems at that time. The problem that converters solve existed, and was getting worse. It was obvious for anyone to see. It was even talked about. But in peoples’ minds at the time, having a car that contributed to air pollution was not a problem most people accepted they had (even though we know that, objectively speaking, they did have it).

  • The Progression of Abstraction in Software Applications

    This post describes a little-known concept for understanding and creating software architecture that small groups use to defeat large, powerful incumbents and nimble competitors. It is one of a small number of powerful, repeating patterns that help us understand and predict the evolution of software. Understanding these patterns can help entrepreneurs direct their efforts; if they do it well, they greatly enhance their chances of success. Understanding the patterns can also help investors choose to invest in groups that are walking a path to success.

    Evolution of Applications Towards Abstraction on a Platform

    One of these patterns is the stages that applications naturally evolve through on a technology platform. Each step or stage brings a big increase in the power of the software, decreasing the effort and increasing the speed and effectiveness of being deployed to meet customer needs.

    A category of software applications may well get “stuck” at a particular stage for a long time, sometimes even decades. During this time, the software may appear to move forward, and of course the marketing people and managers will always put things in the best possible light. But it’s always vulnerable to being supplanted by a next-stage version of the functionality.

    While there aren’t clear lines of delineation between the stages, it’s nonetheless useful to understand them roughly as:

    • Prototype. A hard-coded body of code.
    • Custom Application. Does a job reliably, but most changes require changing source code.
    • Basic Product. The code now has parameters, maybe user exits and API’s. Real-life implementations tend to require extensive professional services, and the cost of upgrading to new versions tends to be high.
    • Parameterized Product. The level of parameterization is high, with interface layers to many things outside the core code, so that many implementations can be done without changing source code. There may be some meta-data or editable rules.
    • Workbench Product. A large portion of the product’s functionality has migrated from code to editable meta-data, so that extensive UI, workflow, interface and functionality changes can be accomplished via some form of workbench, which could be just a text editor. The key is that the details of application functionality are expressed as editable data, meta-data, instead of code. Nonetheless, all fundamental capabilities are expressed in highly abstract code.

    As a body of code goes through this sequence of abstraction, it is increasingly able to meet the needs of new customers and changing needs of existing customers, with decreasing amounts of effort, risk and changes to source code. At the same time, the more abstract a program, the more functionality is expressed as data that is not part of the software itself, a.k.a. meta-data, and the more the software implements generic capabilities, as directed by the meta-data.

    The pattern applies both to individual bodies of code and to collections of them. It applies to code built internally for an organization and to code that is sold as a product in any way.

    I defined the stages above as a convenience; in reality, the categories are rarely hard-and-fast. A body of code could be given a big transformation and leap along the spectrum, or it could take a long series of small steps. One body of code could remain stuck with little change in abstraction, while other bodies of code doing similar things could be ahead, or progress rapidly towards abstraction.

    The Driver of Abstraction Evolution

    In biological nature, competitive pressures and external change appear to drive evolutionary changes. Similarly, when we look at categories of software, if little changes to make the software change, it doesn’t change – why take the trouble and expense to change software that meets your needs or the needs of your customers?

    In reality, someone always seems to want changes to an application. A prospective user would gladly use the software if it did this, that or the other thing. A current user complains about something – it’s too slow, too hard to use, too complicated, or it just doesn’t work for X, Y or Z. How often does a piece of software not have a “roadmap?” If it doesn’t, it’s probably slated for retirement before long. Brand-new software is rare. The vast majority of software effort goes into making changes to an existing piece of software.

    How much time and effort is needed to change a particular body of software? That is the key touch-point between techies and business people. This is the point at which the level of abstraction of the application comes into play. Regardless of the level of abstraction of the application, the “change” required either has been anticipated and provided for or it has not.

    • If the change has been anticipated, the code can already do the kind of thing the user wants – but not the particular thing. Doing the particular thing requires that a parameter be defined, a configuration file changed, a template created or altered, workflow or rule changes made, or something similar that is NOT part of the program’s source code. This means that the requirement can be met quickly, with little chance of error.
    • If the change has not been anticipated, then source code has to be changed in some way to make the change. The changes required may be simple and localized, or complex and extensive – but the source code is changed and a new version of the program is created.

    This is what the level of abstraction of a software application is all about: the more abstracted the application, the more ways it can be changed without altering the source code and making a new version of the program.

    This is the fundamental driver of applications towards increasing abstraction: as changes have to be made to the application, at some point the technical people may decide to make the change easier to make in the future, and create the appropriate abstraction for the kind of change. This may happen repeatedly. As more complex changes are required, the program may just get more complicated and more expensive to make further changes, or more sophisticated abstractions may be introduced.

    While historically it appears that outside forces drive applications towards increasing abstraction, smart programmers can understand the techniques of abstraction and build an application that is appropriately abstract from the beginning. Similarly, the abstracting methods can be applied by smart programmers to existing bodies of code to transform them, just because it's a way to build better code and meet ever-evolving business requirements.

    Conclusion

    The hierarchy of abstraction in software is one of the most important concepts for understanding a specific piece of software, or a group of related software. Over time, software tends to become more abstract because of competitive and business pressures, or because of smart programmers working to make things better. The more abstract a piece of software is, the more likely that it can respond to business and user demands without modifications to the source code itself, i..e., quickly and with low risk.

    The hierarchy of abstraction is certainly a valuable way of understanding the history of software. But it is more valuable as a framework for understanding a given piece of software, and the way to evolve that software to becoming increasingly valuable. It is most valuable to software developers as a framework for understanding software, and helping them to direct their efforts to get the greatest possible business impact with the smallest amount of time and effort.

  • The Fundamentals of Computer Automation

    The principles underlying computer automation are clear and strong. They account for most of the automation we see. They tell us clearly what will happen, why it will happen, and what the benefits will be. What the principles do NOT tell us is who will first apply automation to what process in what sector.

    Understanding the principles lets you predict the rough order of the sequence of automation.

    The principles are extremely simple. Perhaps that's why they're rarely stated and appear not to be taught in schools or understood by practitioners. So much the better for people who want to innovate and win by automating!

    The Principles

    You can probably guess the principles of automation from the definition of the word: "automate" means to do by machine or computer what a human would otherwise do.

    At core, there's really a single simple principle:

    • Replace time that a person would have spent with work done by a computer.

    The core principle is demonstrated by a before-and-after comparison: the principle requires that human time/value doing whatever activity is being automated, net of the investment in the computer hardware and software, is reduced after the addition of the computer and software. Lots of words, same meaning: apply the computer stuff, and there's less total human time/effort.

    Expanding on the replace theme, "replace" means anything that reduces human time to do something:

    • Replace time that a human would have spent with work done by computer; REPLACE!
    • Reduce the amount of time spent by a human; REDUCE!
    • Arrange the human's time more efficiently, eliminating waste; RE-ARRANGE!
    • Decide Who does What work, How and Where they do it; RE-ORGANIZE!

    That's it! 

    This notion of what software was REALLY about struck me quite early in my career. The thought was simple: the ultimate purpose of most of the software I wrote is to replace humans. The thought made me uncomfortable. But after a while, I connected the thought with all the rest of the mechanization and industrialization in society for hundreds of years. A new machine is valuable only to the extent that it somehow reduces the total amount of human labor to reach a given result, everything taken into account! This was true for the Jacquard Looms to which the luddites violently objected in the early 1800's, and the same principles are at work today with computers, resisted by modern-day luddites.

    In other words, computers are the next step in a long sequence of people figuring out how to get stuff done with less effort by themselves and/or other people.

    Here are some simple examples:

    REPLACE

    Replace what humans do with computers doing it is the core principle of automation. E-mail is a super-simple example, since computers automate the entire process of delivering the result of typing to another human being, eliminating the many steps involved in doing it physically. The replacement can be low-level and mechanical, like delivering email, or it can be high-level, like the work of deciding who should do what work in what order.

    REDUCE

    Electric light is a pre-computer example, since it reduces all the time required to, for example, deal with the oil lamp, and replaces it with flicking a switch. A modern example is spreadsheets, which automate the time people used to spend making calculations with adding machines.

    RE-ARRANGE

    Introducing chat into customer call centers eliminated huge amounts of wasted waiting time, so that phone operators could handle chat when they weren't on the phone.

    RE_ORGANIZE

    Routing certain kinds of calls to the optimal customer service person (Who does What work), giving them scripts and hints.

    The Principles apply to Manual and Mental work

    At the start of automation, the computer and software automate work that it's pretty plain that people are doing. As it gets more advanced, a couple interesting things happen.

    1. Once the work is done by computer, it sometimes happens that better ways of getting the work done are possible. This is the same principle we see in mechanical flight: even though all birds have wings that flap, airplane wings don't flap. Instead of flying the ways birds do, engineers have figured out of the core principles of flying and reflected those in airplane design. Similarly, human work is often first translated to computer automation simplistically, and later rebuilt without reference to the human origins.
    2. Once computers are a significant part of the landscape, computer people figure out valuable things to do that have little to do with what humans have done in the past, though this is uncommon.

    The Principles are recursive

    At the beginning, getting computers to do things is incredibly labor-intensive. As time goes on, some people look at the work humans do to get computers to do things — and automate the automation process! Yes, the very programmers whose job is to put people out of work can be put out of work by computers. And on and on.

    The Principles applied to human interaction

    We can see the principles in action in the sequence of changes that have taken place in normal interactions between human beings.

    Prior to automation, the only way humans could communicate was by voice, and by being in each other's physical presence. Here are the major automation steps that have taken place:

    • Writing a letter.
      • Automation is making and acquiring paper, pen and ink, and transporting letter
      • Eliminates the travel time of the sender and/or receiver
    • Publishing books, newspapers.
      • Automation is printing and distributing the books or papers
      • Eliminates the travel time of all the receivers
    • Telephone
      • Eliminates the travel time for the parties to get together
      • Eliminates the time spent writing and reading, and the delay in transmission of prior methods
    • Mobile phone
      • Eliminates the need to travel to a phone to make and/or receive calls
    • Voice mail
      • Eliminates the requirement that the receiver be available when the caller calls, for those cases when it's applicable
    • E-Mail
      • Writing a letter with near-real-time delivery and the ability to read when ready, like letter reading and voice mail.
    • Chat
      • Like email, only with a different interface, making it better for real-time, written exchanges, with minor delays acceptable.
    • Scripting
      • In customer service centers, reduces training, reduces human error, and reduces the time to produce an optimal answer to a question.
    • VRU, chatbot
      • A recorded and/or machine-generated party in a conversation.
      • Applicable to all of the above methods.
      • Eliminates a human giving repetitive responses or messages.

    This all seems obvious, right? Like we know all this stuff, what's new?

    For the steps above that are computer-based, what's interesting is that the technical capabilities were often in place considerably before they were brought into production. The gap was usually not decades-long, like it has been with more esoteric technologies. And in each case, knowing the technology and this fundamental principle could enable you to predict that the automation would be built and deployed.

    Conclusion

    If you want to predict the future of technology and evolution, study the evolution of software and the fundamental principles that drive its application. The principles are obvious! But they are nonetheless rarely discussed or applied.

     

  • Laser Disks and Workflow Illustrate the Insane, Fashion-Driven Nature of Software Evolution

    Computers are so exact and numbers-based that we tend to think of the people in charge of them as white-jacketed scientists with esoteric knowledge driving towards calculated optimal results, much the same way we imagine that the scientists and engineers calculated the path of the Apollo capsule to the Moon. If the Apollo program were run the way most computer projects are, it would have been a miracle if the capsule made it off the launch pad, much less traced an incredibly exact path from Cape Canaveral to the chosen landing spot on the Moon, a journey of over a quarter million miles.

    The reality is that the application of computers and software to automation is painfully slow, usually failing to achieve optimal results by large margins, and often lagging behind what is achievable by decades. The reasons appear to be a semi-random mixture of commercial interests, technology fashion trends, ignorant and risk-averse managers, and technical specialists rooted in the past and wearing blinders. That’s all!

    Here’s the story of a typical example.

    Microfilm, document imaging and laser disks

    Long before computers, microfilm was the preferred medium of librarians and archivists to preserve paper documents for future use. Microfilm was more compact than paper, and lasted much longer. What happened when computers came along is a curious and educational story, a prime example of how computer use and software evolve.

    Disclosure: I lived and worked as a participant in this history during the late 1980’s and early 1990’s. The story I will tell is not systematic or comprehensive, but like a traveler’s tale of what he did and saw.

    A side show in the long evolution of computer storage is the laser disk. A laser disk has data recorded onto it either as a whole, during manufacturing, or incrementally, as part of a computer system. The laser disk had quite a run in the consumer market in the form of CD’s, and then DVD’s. They were an improvement on existing methods for distributing digital content of all kinds.

    In the computer industry, they took the form of WORM (write-once-read-many) drives, and were used to record archival data that should not be updated, but could be read as often as needed. The technology, as often happens, went searching for problems it could solve. In an effort to reduce human labor more than microfilm could, jukeboxes were invented. Each jukebox had a disk reader and many shelves for disks, with a mechanism to load a selected disk into the reader. FileNet, now part of IBM, built one of the first of these systems in the mid-1980’s.

    At around the same time that WORM drives became practical, paper document scanners were evolved from systems to capture images of the paper onto microfilm. In each case, light was shown onto paper and focused onto a target; a scanner replaced the film with an optical sensing device.

    How could we get people to buy WORM drives and jukeboxes, the creative marketing people wondered? The departments that put documents onto microfilm weren’t going for it – the new systems were much more expensive, microfilm wasn’t accessed often enough to make it worth connecting it to computers, and the records retention people were understandably skeptical that any computer disk would be readable 100 years from now, as microfilm will be.

    Adding workflow to document imaging

    I have no idea who made the audacious leap, but some creative person/group came up with the idea of doing computer document scanning at the start of the process, when paper arrived at a location for processing, instead of microfilmed at the end for archival purposes. But WHY??? The creative types scratched their heads until they were losing hair rapidly. Why would anyone make responding to loan requests or whatever even longer than it is today by introducing the step of scanning?

    GENIUS TIME: LET’S INVENT “WORKFLOW”!!!

    Everyone has an image for what “workflow” is. For documents, it’s the path of processing steps from arrival to filing. In larger organizations, there are often multiple departments involved in processing a document, so there are in-boxes and out-boxes and clerks transporting documents from one to the other.

    Rubbing their hands and cackling, the scammers worked out their strategy: We’ll scan the documents and convert them to images so that we can move images from place to place electronically instead of piles of paper! We’ll call it “document imaging workflow.” It will deliver massive benefits to the whole organization, to everyone who touches the paper, not just to the tiny group who files and archives at the end – and storing the scanned documents onto WORM drives will do their job too! Hooray! We will leap-frog the old-fashioned, paper-flooded back office into the modern electronic age. What’s not to like?

    It was audacious and brilliant. It was a scam that would wither if confronted with just a little common sense or cost accounting. Which, true to the pattern of such fashion-driven trends, it never had to confront!

    Implementing workflow

    A typical target environment for implementing workflow was large offices with large numbers of documents, often forms of some kind, arriving every day. The forms would go from the mail room to the relevant people’s desks for processing. The person handling the form would often have an IBM 3270 terminal for interacting with a mainframe computer. When the person was done with the form, they would place it in one or more outboxes for further processing at other desks, or for filing. Sometimes documents would go into file cabinets for a period of time, but all would end up being archived, and sometimes microfilmed for permanent storage.

    Implementing workflow first of all required scanning the documents as a first step, and storing them immediately on normal computer disks, and ultimately on WORM drives, thought to be the equivalent of microfilm. Once scanned, the documents would be managed by workflow software, which would route them to the appropriate desk in the appropriate department for processing. Every desk needed to have the old computer terminal replaced or augmented with a large, high-resolution image terminal, capable of handling at least the display of the document, and sometimes also the computer screen. The old, slow connections between terminals and mainframe needed to be replaced with a fast local area network, and there needed to be substantial servers for handling the local image handling and workflow routing.

    Of course, lots of analysis and programming was involved in addition to the equipment. Workflows had to be analyzed and programmed, and all the management, reporting and exception handling taken care of.

    The benefits of workflow

    When a movement like document imaging workflow has a head of steam up, no one seems to ask whether it’s a good idea, and if so, how good of a good idea it is – quantitatively. When I was involved, everyone threw around there would be at least a 30% productivity improvement due to workflow, and that would make all the expense and disruption worth it. I never encountered a single study that measured this.

    Think about it for a minute. Would looking at an image of a document make you work faster than you would looking at the original paper? Would picking the next paper from the in-box be slower than getting the system to show you the next image? What about the labor of the clerks moving stacks of paper from one person’s outbox to the next one’s inbox? It would certainly be saved, but chances are a single clerk could handle the paper movement for many dozens of people. And remember, there’s the scanning and indexing that’s been added and the massive computer and software infrastructure that has to be justified.

    It’s obvious why no one EVER did a cost-benefit analysis of workflow – the back-of-the-envelope version easily shows that it’s a couple zeros away from making sense.

    I remember a couple of out-of-it, tech-fashion-disaster know-nothings mildly thinking out loud about how workflow could possible be worth it financially. The immediate response from workflow fashion leaders was upping the ante – don’t you know, the wonderful workflow tool from (for example) FileNet lets you program all sorts of efficiencies for each stage of work; it’s something we really need to dive into once we get the basic thing installed. End of subject!

    So who fell for this stuff? Just a who’s-who list of major organizations. Each one that fell for it increased the pressure for the rest to go for it. None of them did the numbers – they just knew that they couldn’t fall too far behind their peers.

    Conclusion

    It’s hard to think of a field that’s more precise and numbers-oriented than computers. Who has a clue what actually goes on inside those darn things? And the software? There are millions of lines of code; if a single character in any of those lines is wrong, the whole thing can break The impression nearly everyone has, understandably, is of a field that is impossibly precise and unforgiving of error. When the software experts in an organization say that some new technology should be adopted, sensible people just agree, particularly when there’s lots of noise and other major organizations are doing it. It can be even more gratifying to get known as pioneering, as one of the first organizations to jump on an emerging tech trend!

    Somehow, the smoke and noise from the software battlefield is so intense that the reality is rarely seen or understood. The reality, as I’ve illustrated here, should result in everyone involved being sent to the blackboard by a stern teacher, and being made to write, over and over: “I pledge to try harder to rise above the level of rank stupidity next time.”

  • Simple Data Entry Technology Illustrates How In-Old-Vation in Software Evolution Works

    Since when is “data entry” (entering data into a computer) a pivotal, innovative technology? When the difference between doing it the normal way and doing it with advanced technologies is a …ten-to-one productivity difference … that’s when.

    I’ve described how the Operations Research algorithm of Linear Programming is fifty years into an agonizingly slow roll-out through different applications, from scheduling oil refineries in the 1960’s to scheduling retail sales in the 1990’s, and now scheduling medical infusion centers and operating rooms in the late 2010’s. In each case, laborious and error-prone human scheduling was replaced by the algorithm, with improvements ranging from no less than 10% to over 50%. This is major! Why did such an innovation wait for decades to be applied, and for many applications, is still waiting!!?? This is the mystery of how what’s called “innovation” works in reality, and why it should be called “in-old-vation” instead.

    You may think that part of the cause is that LP is an exotic algorithm – even though it’s a standard part of the Operations Research engineering curriculum, most so-called normal people haven’t heard about it. While it appears that even the hosts of people who wax eloquent about AI and ML are clueless about LP, it’s not exactly a secret. So let’s see if obscurity is the reason why LP remains a “future innovation” in many potential applications, by examining the super-plain, ordinary, completely-understandable-by-normal-people case of data entry.

    Data Entry

    Just as process optimization is done by hand or stupid methods for decades until some genius comes up with the brilliant idea of applying tried-and-tested LP to the problem and dramatically improves it, so is Data Entry widely performed by primitive methods until some “innovator” comes along and applies “Heads-Down Data Entry” (HDDE) methods to the process – and typically gets improvements of 3 to 10X! The only difference is that while LP is taught in Engineering departments and studied by math nerds, Heads-Down-Data-Entry is “just” a collection of common-sense techniques that require no math and no professors to understand or implement. It’s so “common” that it doesn’t even have a generally accepted name – though it’s been implemented in many places and been thoroughly proven in practice. It’s far too humble to merit an academic department – and yet, when applied, has delivered truly massive gains, far higher proportionately than exotic Linear Programming has!

    The methods of HDDE were first implemented in places that had huge volumes of repetitive data to be entered into computers. Banks were early users for check entry, and so were the credit card companies who, at the start, had huge volumes of paper charge slips to process. Simple ideas like minimizing keystrokes and eye movement were implemented, and then taking advantage of the eye-to-fingers pipeline, when people noticed that showing clerks the next item to be entered before the current one was complete led to a big jump in speed. Other methods like double-blind techniques were invented, so that entry clerks just entered – whether their work was original or used to check someone else’s work was entirely handled by the Data Entry system.

    As soon as scanning and image display became practical, HDDE adopted them. That led to another jump in productivity, enabling large, complex forms to be broken up into pieces, so that a clerk would see the image on a screen of the same piece of data from a whole set of forms instead of entering a whole form from start to finish. No HDDE shop would even consider having the entry clerk think about anything on the form, stuff like “if this field is missing, do this instead of that,” because it would just slow them down.

    Finally, there’s ICR (Image Character Recognition), which is having the computer “read” the image instead of a human. This technology has existed for many decades. Once you’ve got HDDE in place, phasing in ICR is a natural, so that the proportion of entry done by humans gradually decreases as the effectiveness of ICR increases.

    Remember, applying LP to scheduling might result in a 30% improvement, which in most cases, is major to the point of being revolutionary. What about HDDE? Entering data from a paper form into a computer using primitive methods might get between 1,000 and 1500 KPH (keystrokes per hour). There are lots of stages of improvement, including things I’ve mentioned like eliminating the thinking and breaking up the form, but levels of 10,000 to 15,000 KPH in a professional environment are widely achieved – with superior quality. That’s a minimum of 5X! Typically much more. Of course, as you incorporate ICR into the process, it gets even better, gradually reducing the human factor, so that most fields are entered with no human involvement. At this point, the technology is probably best called ICR+HDDE, though there is no generally accepted term.

    Given all this, it would be insane to handle computer data entry by anything other than HDDE methods, right? Welcome to the software industry, where insanity of this kind is the accepted state of affairs. And where almost no one practices the most simple and basic of computer fundamentals, such as counting.

    How HDDE gets implemented

    I described how Linear Programming went from problem domain to domain, each time acting like an innovation, as indeed it was in that area of application. Once it gets established, it tends to stay. The case of HDDE is different, I think because it’s not a recognized “thing” in the halls of academia, or among the poo-bahs of big business. It’s the kind of thing that no self-respecting Professor of Computer Science would stoop to consider, assuming he ever encountered such a low-status thing – you know, the kind of thing that “merely” makes common sense and, well, works.

    HDDE has appeared in competitive, high-volume service businesses, where it has a major role to play in delivering results for the customers of the business. There have been software products that directly support HDDE, so that all you have to do is buy and implement them. It’s neither obscure nor hidden. But it’s never been talked about at conferences as the “coming thing.”

    Case Study: HDDE rejected

    In the early 1990’s, when document imaging and workflow technology were hot and something people talked about the way they talk about AI/ML and innovation today, the government-backed student loan organization, Sallie Mae, decided to apply the technology to improve the operations at the handful of processing centers they had at the time, employing many thousands of people and processing millions of documents a year. The popular thing to do at the time was to scan documents on receipt, and then send them to the same places the paper was sent, so that workers could process the images of the documents displayed on new big screens instead of paper. The job was basically to type the data into the right places of the software application they used.

    Everyone at the time said that converting the documents was important ONLY because it enabled wonderful workflow, the elimination of inboxes and outboxes for paper. And the bits of other stuff you could do, like having a group of people taking from a common inbox instead of each having their own. The common “wisdom” was that you could gain 30% productivity improvement by implementing this marvelous new technology.

    I got involved, since I was a recognized expert on document imaging technology at the time, and had personally coded one of the early workflow systems. I figured out and showed in detail that by canning the workflow and implementing HDDE techniques, they could gain a minimum of 5X productivity improvement. No one disputed my thoughts or detailed plan. They just ignored it, and proceeded to implement the standard stuff. I strongly suspect that after considerable time and expense, there were walk-throughs of the Sallie Mae sites showing visitors the big screens and absence of paper – what a big success the project was!

    Case Study: ICR-HDDE applied with success

    There are two current cases I know of where ICR-HDDE is being applied and winning. Each are classic, narrow service businesses where converting forms to data is the key value of the business, and where the companies buying the service just want fast, accurate data delivery, they don’t care how it’s done. Disclosure: each of these is an Oak HC/FT investment.

    At Groundspeed, insurance forms and reports are captured and the relevant data is extracted from them by the most effective relevant means, often involving forms of ICR-HDDE. There is lots of forms recognition from documents and images that are often computer output, with the relevant data appearing at varying places on a page. Nonetheless, Groundspeed is able to deliver the data stream the customer needs, quickly and accurately, The results are so powerful that new levels of analytics are enabled by the newly available stream of structured data.

    At Ocrolus, financial documents of all kinds including bank statements and pay stubs are converted to data in a standard format to enable fast and effective operations like giving loans for business and personal use, along with a growing list of other operations that also need good data. An effective combination of ICR-HDDE techniques are applied to get results for companies that need accurate data to make fast decisions.

    Conclusion

    HDDE is a collection of methods that have been proven in practice for many decades. The technology that is behind it continues to deepen and reduce human effort even more, with the addition of ICR. But it remains a niche technology, ignored by the numerous places that could benefit from it, even more than LP.

    The big difference between LP and HDDE is that LP is a formal piece of magic that’s in academia. HDDE is nowhere. In fact, it’s really just an example of classic industrial engineering applied to computer software and the people who use it. Which makes it all the more mysterious that it's largely ignored.

    In-old-vation is real. Most “innovations” are minor variations on things long-since proven and demonstrated in practice, but are unimplemented in the many situations that would benefit from them until some mysterious combination of circumstances arises to let them explode into practical reality.

  • Understanding Software Evolution Helps you Predict the Future of Software

    There are patterns in software evolution. Because almost no one studies software history, and even fewer study software evolution, these patterns are almost never discussed.

    The patterns are amazing. In some cases, you can be pretty sure that a trend that “everyone” says is going to be the future will fizzle out. In other cases, you can predict with a high degree of certainly that software of a certain definite kind and description will be built – even though it hasn’t been built yet – it hasn’t been built anywhere by anyone, but nonetheless you know it will be built.

    The reason I started to study software evolution is that it’s just plain interesting. The patterns that emerge from it are striking and educational. But most important, knowing software evolution can help you predict the future!!

    Knowing software evolution can help you predict the future for the simple reason that software does not evolve the way pretty much everyone thinks it does. It is often the case that, when a new software system appears on the market that has a big impact, people who know the patterns were sitting around waiting for it to happen! They knew it was coming!

    I remember living in New Haven CT in the early 1970’s. I was a vegetarian. A new restaurant had just opened, a kind I’d never been to, an Indian restaurant. I went, and it changed my culinary life. Indian food – made by vegetarians for vegetarians, healthy and wonderful flavors!

    The new restaurant was definitely new in Connecticut, a true innovation. The owners had to go through quite a bit to open it up, get supplies, etc. But was it otherwise innovative? Did the owners invent a single one of the dishes or recipes? No, of course not. Just like with a piece of software, they had to create it from scratch in a new setting, making appropriate adaptations, and so to the locals, it was new. But in reality, it was a newly adapted instance of a proven winner. Software works the same way – a piece of software that is a big new winner, the first of its kind, has most often been proven elsewhere.

    The first Indian restaurant in New Haven could have opened years earlier or later than it did, with different people doing the work. There was no way to predict when the Indian restaurant would have opened, or who would have done it. But isn’t it obvious that, in the context of ethnic restaurants enjoying increasing success and the growing Indian population that someone would have opened an Indian restaurant there? Shockingly to most people, software works exactly the same way! In a surprising number of cases, you can’t predict WHO or WHEN, but you can sure as heck predict WHAT will be built and even WHERE (i.e., in what business domain).

    Here is a specific example of software that resembles in important ways the example of Indian restaurants. Instead of Indian food, the star is a specific optimization method within the field of Operations Research (OR) called Linear Programming (LP). While most people know there are loads of Indian people in India and elsewhere and they have a unique cuisine, understanding this is helped by the fact that we can picture India on a map of the globe and we all eat some kind of food. How many people have even heard of OR or LP, or have seen and understood software of any kind? To be clear, what I mean by "having seen software" is seeing the source code, not the results of it on a screen. This LP software sits around, widely used in some domains, and completely unknown in others, just like there were no Indian restaurants in New Haven before a certain date.

    Knowing this helps you predict the future of software, because in a shocking number of cases, it doesn't need to be invented, but "merely" imported into an area of use where it is not currently in use. That is a big reason to be interested in understanding software evolution.

  • Barriers to Software Innovation: Radiology 2

    Value-creating innovations are rarely the result of a bright new A-HA moment, though an individual may have that experience. A shocking number of innovations are completely predictable, partly because they've already been implemented — but put back in the vast reservoir of ready-to-use innovations, or implemented in some other domain. This fact is one of the most important patterns of software evolution.

    Sometimes the innovation is created, proven and fully deployed in production, like the optimization method Linear Programming, which I describe here. In other cases, like this one, the innovation is built as a functioning prototype with the cooperation of major industry players — but not deployed.

    In a prior post I described how I went to the San Francisco bay area in the summer of 1971 to help a couple of my friends implement a system that would generate a radiology report from a marked-up mark-sense form. We got the system working to the point where it could generate a customizable radiologist's report from one of the form types, the one for the hand. Making it work for all the types of reports would have been easy — we demonstrated working software, and wrote a comprehensive proposal for building the whole system. It was never built.

    True to the nature of software evolution, the idea probably pounded on many doors over the years, always ignored. But about 10 years ago, a pioneering radiologist in Cleveland came up with essentially the same idea. Of course, instead of paper mark-sense forms, the radiologist would click on choices on a screen, and would usually look at the medical image on the computer screen. This enabled the further benefit of reducing the work, and letting doctors easily read images that were taken in various physical locations. Tests showed that doctors using the system were much more productive than those who worked in the traditional way. Finally, they decided that mimicking the radiologist's normal writing style was a negative, and that the field would be improved by having all reports follow a similar format, with content expressed in the same order in the same way. This was actually a detail, because the core semantic observations would be recorded and stored in any case, enabling a leap to a new level of data analytics. It also, by the way, made the report generation system much easier to build than the working prototype we had built decades earlier, which enabled easy customization to mimic each radiologist's style of writing.

    The founding radiologist was a doctor, of course, and knew little about software. He did his best to get the software written, got funding, and got the system working. Professional management was hired. My VC group made an investment. Many people saw the potential of the system; it was adopted by a famous hospital system in 2015. But in the end, the company was sold off in pieces.

    Nearly 50 years after software was first written that was able to produce medical imaging diagnostic reports quickly and reliably while also populating a coded EMR to enable analytics, the system is sitting in the vast reservoir of un-deployed innovations. It can be built. It saves time. It auto-populates an EMR.

    Many people have opined on why this particular venture failed to flourish. It's a classic example of the realities of software innovation and evolution. The reasons for failure were inside the company and outside the company. For the inside reasons, let's just say that the work methods of experienced, professional managers in the software development industry lead to consistently expensive, mediocre results. Nonetheless, the software worked and was in wide production use, delivering the advertised benefits. For the outside reasons, let's say that, well, the conditions weren't quite right just yet for such a transformation of the way doctors work to take place.

    The conditions that weren't right just yet for this and uncountable other innovations add up to the walls, high and thick, behind which a reservoir of transformative innovation and "new" software awaits favorable conditions. In other words, the reservoir of innovations wait for that magic combination of software builders who actually know how to build software that works, with a business/social nexus that accepts the innovation instead of the standard no-holds-barred resistance.

    Corporations promote what they call innovation. They are busily hiring Chief Innovation Officers, creating innovation incubation centers, hanging posters about the wonders of innovation, etc. etc. They continue to believe the standard-issue garbage that innovation needs to be invented fresh and new.

    The reality is that there is a vast reservoir of in-old-vations that are proven and frequently deployed in other domains. All that's needed is to select and implement the best ones. HOWEVER, a Chief Innovation Officer is STILL needed — to perform the necessary function of identifying and breaking down the human and institutional barriers that have prevented the in-old-vations from being deployed, in many cases preventing roll-out for — literally! — decades!!

  • The Comes-before is not Causation Fallacy in Software Evolution

    Practically no one understands software, so why should we expect anyone to understand the broader and in some ways more demanding subject of software evolution? But while rarely (as in, almost never) studied, thoughts about software evolution are often found lurking in the backs of people's minds, and pop out in the things they write.

    In biological evolution, putting aside nasty things like DNA and all the stuff it can and can't explain, there is a now-obvious pattern in the fossil record of how species have evolved. There is a clear progression from simple to complex, less capable to more capable and so on. Cold-blooded lizards are pretty neat, but there's little doubt that warm-blooded mammals rule, and that mammals appear in the fossil record after the cold-blooded animals. The picture of this nearly all of us share is that the later creatures evolved from earlier ones, in a more-or-less continuous progression. No leap-frogging is involved — just a step-by-step process of learning and mostly improving.

    A recent example in Wired Magazine about the Apollo 11 computer system illustrates this way of thinking very well. Here's the title:

    Her Code Got Humans on the Moon — and Invented Software Itself

    The Apollo code in general and Margaret Hamilton in particular "invented software itself." Really? Let's look at some of the article. Here's a key paragraph:

    By mid-1968, more than 400 people were working on Apollo’s software, because software was how the US was going to win the race to the moon. As it turned out, of course, software was going to help the world do so much more. As Hamilton and her colleagues were programming the Apollo spacecraft, they were also hatching what would become a $400 billion industry.

    The last clause of the above quote clearly states that the $400 billion software industry that eventually emerged was "hatched" from the Apollo effort. Like so many people, the author thinks that the 400 person effort to build Apollo software somehow made possible, laid the groundwork for, or something all the developments to follow.

    What a pile of hogwash. But typical!

    My personal initiation into programming was on an IBM 360 model 50 computer that I was able to access on Saturdays during the 1966 school year when I was in high school. I wrote FORTRAN programs, key-punched them into cards, fed them into the card reader, got direct feedback from the computer about my imperfections as a programmer, and finally got results I wanted. While doing this, I was one of many people at many facilities using this software.

    The IBM System 360 effort was started about 1960, and was announced in 1964. Gene Amdahl was the main hardware architect, tasked with the then-unprecedented task of creating a family of computers with a wide range of capabilities that could all run the same instruction set, i.e., run the same software.

    Operating system software for such a system had never been created, and was a difficult task, particularly because supporting running multiple programs at the same time, again a first, was a requirement. The software was much-delayed, and by the time Fred Brooks took over the effort, more than 1,000 people were working on the project. Brooks later went on to write a classic book inspired by the effort, the Mythical Man-Month.

    The System 360 was wildly successful, and fueled the incredible growth of IBM. Its instruction set lives on to this day in the System z mainframe servers. Its use was exploding at the time the Apollo software was still being developed.

    What about the Apollo software? It was peculiar from the beginning, and exhibited issues that had already been solved in OS/360 and its programs. While the independent software industry did indeed grow and mature on S/360 computers, in spite of IBM's efforts to keep anyone from buying any software except their own, the Apollo software led to nothing. The now-lauded Margaret Hamilton tried to commercialize the special software methods she had used on the Apollo project, but her efforts failed and led to nothing. There were a couple of later attempts to build and commercialize what we can now see are versions of her ideas, for example in model-driven development, but after a period of hype they went nowhere.

    None of this, by the way, is intended in any way to minimize the achievement of the Apollo mission, the pioneering computer system and software that was an integral part of it, or of Margaret Hamilton herself. She led a huge team that had to get software for a novel system and mission working the first time. It's a real achievement! But it's hardly "inventing software itself."

    Even if you believe that someone "invented software itself," the inventor was certainly not Ms. Hamilton; even if you believe that a big, early software effort "hatched" today's software industry, it was not the Apollo effort.

    So did IBM do it? Is IBM OS/360 the progenitor of today's software industry? Nope! A growing number of groups were designing and building computers, and even more were building software. You can see patterns, but this biological evolutionary idea of a capability being somehow created in DNA, and through inheritance and random improvement leading to subsequent efforts in a consequential way is nothing but the application of an inapplicable metaphor. While patterns can be observed, each piece of software is written from scratch, based on the ideas and experience of the authors. No software-equivalent of DNA, with the obvious exception of the use of shared code.

    I doubt this little rant will be sufficient to stamp out the use of bad logic and inapplicable metaphors by people who insist on writing about software, even though they've never seen it, can't read it, and are clueless about what it really is and how it works. But at least writing this has gotten it out of my system, kind of…

  • The Slow Spread of Linear Programming Illustrates How In-old-vation in Software Evolution Works

    There is loads of talk about “innovation.” Lots of people want to do it, lots of people think they’re doing it, consultants run courses in how to be innovative, and large organizations claim to promote innovation and be innovative. The assumption behind most of this “innovation” talk is that a wonderful bright idea that will change the world (or at least your organization or startup) can pop into anyone’s head. It’s new! It’s brilliant! We’re going to win big with this great new idea! See this for example.

    When you study software evolution, you get an entirely different picture of software-based “innovation.” Software evolution shows you that new ideas that work are extremely rare. Oh sure, there’s a flood of new ideas popping into people’s heads all the time. Mostly, they’re not new, and the new ones rarely work. The software concepts that make it big are, in the vast majority of cases, clear examples of existing patterns of software evolution, and have in most cases already been implemented in a different context.

    I first encountered the mystery of software evolution while in my first job programming software.

    I started programming in a course I took starting in 1966 in high school, taught by a math teacher who couldn’t himself program, but had convinced the school to let him teach the course, and had convinced a local company, a pioneering rocket engine company called Reaction Motors, to give us computer time on Saturdays. I had a textbook about FORTRAN, a steady stream of programming assignments and a computer on which to test my programs. It was great! I continued programming the following summer, as part of an NSF math camp I was able to attend. As I was nearing high school graduation the following year, I got lucky; Diane, a high school friend, talked about me with her father, who got me connected with a nearby company, and I landed a job there for the summer before starting at Harvard College in the fall of 1968.

    The company was EMSI, Esso Mathematics and Systems, Inc. in Florham Park NJ.

    1969 06 EMSI badge 1

    They were a service company, one of the about 100 units of the Standard Oil of NJ (Esso) companies, devoted to applying math and computers to improving every aspect of the company. I was immediately thrown into the group that was developing optimization models for oil refinery operation. Our focus was on the giant refinery in Venezuela.

    What we had running was an implementation of a classic OR (Operations Research) algorithm called LP (linear programming), solved via the simplex algorithm first devised by George Danzig in 1948. In this kind of model, there is a goal equation and a set of constraints. The goal equation calculated profits, using hundreds of contributing variables, including prices you could sell things for, and the costs of various inputs. The constraints were greater/less than equations, each essentially describing some tiny aspect of how the refinery worked. What the algorithm did was find the values of the variables that maximized the goal equation (profits) while satisfying all the constraint equations.

    The model was constantly being modified to make it more precise and applicable to actual refinery operations. I had a variety of jobs, including writing new code, fixing bugs, etc.

    Since prices were such a key part of the LP model, we had a separate program to calculate what they were likely to be in the future, a Monte Carlo model. I also made enhancements and fixed bugs in this body of code.

    I was fortunate to be able to hitch a ride to work and back with a PhD who worked there and lived in my town. During the ride I would tap his deep knowledge of things. He put me onto the various journals in which advances in various forms of OR were described, which I dove into. The math was often above my head, but I was motivated to teach it to myself on a rolling, as-needed basis.

    I thought this was a really cool way of running things. There were all sorts of controls in the refinery, controls that let you create more airline fuel and less heating oil, or any number of other trade-offs. It was amazing that you could compute the setting of all the control knobs that would produce the best mix of products that the market needed. Why would you ever run any operation any other way?? It would be simply ignorant and stupid.

    I finished school, learned more about the way the world worked, and searched high and low for implementations of LP. Anywhere! If they were out there, they were doing a great job of hiding.

    I was confused. How could this be? LP was math, doggone it. It yielded a provably optimal way of running a business. It’s proven in real-life production at Esso. Any other way of operating a business was clearly seat-of-the-pants, wet-finger-in-the-wind amateur hour; anyone whose operation was sizable enough to justify the effort would have to use this method if they weren’t plain-and-simple incompetent. But no one seemed to be using it! What’s going on here??

    This mystery was on my mind while I participated in one of the periodic AI crazes that has swept the world of with-it people. While I was still in college, Winograd published his MIT SHRDLU research, in which an “intelligent” robot would converse with a human in English about a world consisting entirely and solely of blocks. You could ask SHRDLU to “put the red block on top of the blue block” and it would do it; you could ask it questions about block world, and it would answer. Amazing! Super-practical! While this was happening, I wrote and submitted a thesis about how to structure knowledge inside of an intelligent robot. All of it useless compared to LP and the associated OR techniques.

    The craze was AI, and the mania lasted a few more years, generating a steady stream of "promising" results, none of them in the same universe of practicality and benefit as LP or any other OR optimization technique, which continued to be used only in "secret" little islands of astounding efficiency and productivity.

    Later in the 1970’s I first applied for a home mortgage. A key part of getting the mortgage was the interview with the loan officer. You had to pass muster to get the mortgage! Another 10 years passed before OR-type models started to be used for credit. When I next applied for a mortgage in 1981, I was interviewed by a loan officer. By my next mortgage in 1987 it was at least partly automated.

    When I got into venture capital in the 1990’s, I discovered that high-value repair parts were had recently been optimized in terms of inventory levels and locations. I looked in detail at the pioneering company ProfitLogic, which did inventory and sale optimization for retail stores, answering questions like when should which products be put on what kind of sale, questions that had traditionally been answered by the local marketing “expert,” just as oil refineries had previously been run exclusively by experienced experts.

    Only in the late 2010’s did exactly the same LP models start being applied to medical scheduling, to optimize the use of things like infusion centers and operating rooms. Just as oil refineries produced much more value from exactly the same crude oil inputs in 1968 as a result of LP models, so are infusion centers handling 30% more throughput using exactly the same capital and human resources using the very same LP models.

    Here’s the mystery: why did it take so blankety-blank long for LP models to be applied??? There has been no theoretical break-through. Yes, the computers are less expensive, but given the scale of the opportunity, that was never the obstacle. WHY??!! The answer is simple: there is no good answer. Except of course for the ever-relevant one of human ignorance, stupidity and sloth.

    The example of LP optimization and its agonizingly long roll-out through different applications and industries over more than 50 years – a roll-out that is far from complete! – is a prime example of the reality of computer/software evolution. Among other things, it illustrates the point that many of the most impactful "innovations" are really "in-old-vations," things that are just sitting there, proven in production and waiting for someone to apply them to one of the many domains which they would benefit.

    Here are a couple cornerstones of computer software evolution:

    • Software evolution resembles biological evolution only a little. Not-very-fit software species thrive in broad areas of application, unchallenged for years or decades, while vastly superior ones rule the roost not far away. The only reason why the superior software species don’t migrate to the attractive new place appears to be human ignorance and inertia.
    • Software evolution resembles the much-derided theory of “intelligent design” quite a bit, if you make a slight edit and call it “un-intelligent, un-educated design.” A “superior” (HA!) being does indeed do the designing of the software, in the form of highly paid software professionals.
    • When software appears in a new “land” (platform, business domain), it most often starts evolution all over again, first appearing in classic primitive forms, and then slowly re-evolving through stages already traversed in the past in other “lands.” This persistent phenomenon supports the “unintelligent design with blinders on" theory of software evolution.

    I will explain and illustrate these points in future posts and a forthcoming book.

  • Barriers to Software Innovation: Radiology 1

    There is a general impression that software innovation in one of its many forms e.g. “Digital Transformation” is marching ahead full steam. There are courses, consultants, posters hanging in common spaces and newly-created Chief Innovation Officer positions.  What’s new? What’s the latest in software?

    The reality is that there are large reservoirs of proven, tested and working software innovations ready to be rolled out, but these riches are kept behind the solid walls of dams, with armies of alert guardians ready to leap in and patch any holes through which these valuable innovations may start leaking into practice. Almost no one is aware of the treasure-trove of proven innovations kept dammed up from being piped to the many places that could benefit from them; even the guardians are rarely fully conscious of what they’re doing.

    If anyone really wanted to know what was coming in software, all they would have to do is find the dams and peer into the waters they hold back.  In spite of the mighty dams, it sometimes happens that the software finds its way into practice, normally in a flood that blankets a small neighborhood. Sometimes the flood has been held back for decades. There are cases I know of where an innovation was proven 50 years ago, and is still not close to being rolled out.

    The dams are built in many ways with many materials. The raw materials appear to include aspects of human nature: ignorance, sloth, greed — you know, the usual. The really high, solid dams have broad institutional support, in which “everyone” is fine with things as they are, and won’t so much as give the time of day to an amazing innovation that would change many things for the better – except of course for a key interest group.

    Here is one of the examples I personally know about. It was one of my introductions to what innovation is all about, and the sad fact that creating a valuable innovation is generally the easy part – the hard part is usually overcoming the human and institutional barriers to deploying it.

    Automating Medical Image Reading and Reporting

    When a radiologist gets an X-ray, there are two phases of work. The first is to “read” the X-ray and observe anything non-typical that it shows, anything from a broken bone to a tumor. The second is to generate a report of the findings. Most radiologists, then and now, dictate their findings; then someone transcribes the dictated report and sends it as needed. The details of the report can vary depending on the purpose of the X-ray and the kind of person for whom it’s intended.

    There has been technology first tested decades ago that appears to show that software is capable of “reading” an X-ray at least as accurately as a human radiologist. I will ignore that work for now, and focus on what should be the less threatening technology, which is translating the doctor's observations to an appropriate report.

    While I was in college, I worked on the early ARPAnet with an amazing group of people, one of whom was an MIT student from the San Francisco area who later went on to fame making major advances in integrated circuits, among other things. The summer after we did most of our ARPAnet work, he got involved with a new initiative to transform the way radiologist reports of X-rays were created. He knew that some of my skills in automated language parsing and generation were relevant, so he invited me out to pitch in. I went.

    GE, then and now, was a major maker of medical imaging systems. They were seriously experimenting with ways of enhancing their systems to make it easier for radiologists to produce reports of their findings. They created a set of mark sense forms, of the kind widely used at the time for recording the answers to tests, to enable a radiologist to quickly mark his observations of the part of the body in question. Here is the form for a person's guts: X form

    Here is part of the form for the hand, showing how you can mark your observations: X observe

    Here is part of the form for the spine, showing how you can customize the report output as needed: X type

    My friends had gotten most of the system together — all I had to do was build the software that would create the radiologist's report. Because of uncertainty about radiologist's accepting the results, I had to make the report generator easily customizable, so that the radiologist's typical style of writing was created.

    Leaving out the details, in a few weeks I created a domain-specific language resembling a generative grammar and rules engine to do the job — along with the necessary interpreter, all written in PDP-8 assembler language, which was new to me. My friends wrote a clear and compelling report describing our work and included an example of our working software in it. Here was the sample filled-out form: X ex pic

    And here was part of the report that was generated by the software we wrote from the input of that form: X ex rep

    The software worked! And yes, the date on the report, 1971, is the date we did the work.

    A major company, prominent in the field, had taken the initiative to design mark-sense forms, incorporating input from many radiologists. A few college kids, in contact with one of GE's leading partners, created a working prototype of customization report-generating software, along with a proposal to bring the project to production.

    Just as a side effect, this project would have done something transformative: capture the diagnostic observations of radiologists into fully coded semantic form. This is a form of electronic medical record (EMR) that still doesn't exist, even today! For all the billions of dollars that have been spent on EMR's, supposedly to capture the data that wil fuel the insights that will improve medical care, a great deal of essential medical observation is still recorded only in un-coded, narrative form — including medical imaging reports!

    The bottom line is that this project never got off the ground. Not because the software couldn't be written, but because … well, you tell me.

    See the next post for the continuation of this sad but typical story.

  • The Evolution of Software

    There is strong interest in the latest developments in software. No one wants to be left behind.

    At the same time, there is a peculiar (to me) lack of interest in whatever the latest thing grew out of or evolved from. What are the new thing's predecessors? What did that thing grow out of. Have similar things appeared in the past? Are there patterns we can observe here, or are the new software things that explode onto the scene just a meaningless sequence of random events? Are they things that we obsess about while they're here without questioning or wondering where they came from, without asking what else they may resemble in some way? Are they things we just forget about and ignore once they depart the stage of software fads?

    The questions have obvious answers. Answers that no one seems to be interested in.

    So long as we fail to ask these crucial questions, we will remain ignorant of software in deep ways, and will continue to stumble from fad to fad without improvement or understanding. History is important!

    Evolution in Science

    If you study different sciences and their history, you notice that, at some point, scientists begin to pay attention to evolution, i.e., change over time of the thing being studied.

    The most obvious thing is evolution of an individual item over time. In biology, you have the cycle from birth to death. In geology, you have the change of mountains, rivers, beaches and glaciers. In materials science you have the change of objects as they are subjected to various conditions over time, for example the way that iron rusts.

    Evolutionary studies reach a breaking point and get really interesting when you study groups and types and see the patterns of change. In biology, this was the revolution that happened with Darwin and his study of the evolution of species. Something similar happened in geology with plate tectonics.

    When you put the patterns together, things get totally amazing and transformative. While no longer fashionable to talk about and study, the parallels between the growth stages of an individual animal and the evolution of species are obvious, though not literal at a detail level.

    Studying evolution is an important stage in the evolution of most sciences!

    It's long-since past time to study software evolution, as an integral part of software science.

    Software Evolution

    Software evolves at every level.

    At a basic level, software languages evolve. I have given an introduction to this subject here. The evolution of software languages resembles biological evolution in remarkable ways, as species (languages) emerge and evolve in response to new conditions, including other species. Similarly to biology, languages evolve; sometimes many species descend from them; and sometimes a species goes extinct. See this snapshot, for example:

    11

    (Credit)

    Consistent with the determination of nearly everyone involved with software to ignore its history, this chart and anything like it are ignored, and the evolutionary principles behind it are not studied. When you start to study them, you learn amazing things. Suddenly some of the random change in the world of software languages starts to make sense.

    Software languages are the very most basic things about software. Just as interesting and vastly more important are the programs written in software languages. Everyone involved in writing or maintaining a software program (application) just thinks about that program. It definitely occurs to business people what the program can do compared to its immediate competitors, if any. Yes, there is a kind of "survival of the fittest" in software evolution.

    What about the "insides" of the program compared to other program insides? Sometimes people are aware of gross differences, like the language the program is written in. But the curiosity normally stops there.

    The program "insides" have huge practical consequences. Depending on the details of how a program is written, it can be more or less:

    • reliable/buggy
    • easy to hack (secure)
    • expensive to run/operate
    • dangerous/costly to change
    • able to respond quickly to changing loads
    • speedy or slow

    People commonly talk about structure or architecture, but those things are just the tip of the iceberg.

    When you dig in, you find the equivalent of islands and continents of software. I've treated this subject here, for example. For example, you may have spent some number of years after getting your CS degree working on website design, and interacted with a community of people similarly engaged. Not everyone uses the same tools or does things the same way, of course, and you're likely to think there's quite a variety of approaches.

    Then you go wild, dive into hospital management systems, and your mind is blown. What you thought was the wide variety of software and tools used by website designers turns out to be a whole continent apart from what you find in hospital automation. You learn that the flagship product of the company that has more than a third of the market (Epic) is written in a language you've never even heard of! It's like growing up in New Jersey among what you think is a very wide variety of people and going to Chennai — a huge city in India you've never heard of, whose population is larger than your whole state, where the main language is one you've never heard of, written in a script you've never before seen.

    Even more ignored are the obvious (to those who trouble to look) repeating patterns that can be observed in applications as they are first written and then evolve over time.

    The Study of Software Evolution

    To put it bluntly, the study of software evolution has barely begun. A few isolated souls, hardy or foolhardy as you like, have dipped their toes into the deep waters of software evolution. Those few have found worlds to explore, oceans whose depths have yet to be plumbed.

    This is particularly the case because software is a "the only reality is the present" kind of field, as it now stands. It doesn't have to be this way!

    Very few people study the history of math. They don't need to: anyone who learns math starts with math as it existed thousands of years ago. If they do well in math, they get all the way up to … the late 1600's, when Newton and/or Leibnitz discovered calculus! Each step in math is based on and often depends on the earlier steps.

    In some sense, this is also true of software. Except not really.

    First, knowing nothing about software, you can dive right into the latest language and tools and use them to make something that works. Anyone think you could do that with differential equations, I mean without already knowing how to count, add and multiply? The difference is that in software, the "accomplished" new-comer really is standing on generations of older software … of which s/he is blissfully unaware! The new programmer, of course, uses an incredible array of operating systems, drivers, networking, file and database systems, and on and on. These form the "land" on which the new programmer "stands" to work. None of the wonderful new stuff would work without it. The big difference, of course, is that using the underlying software and understanding the underlying software are two rather different things. This makes software totally different from not just math, but most of the other legitimate sciences (leaving out the fake ones like political "science" and the rest).

    Second, when you're building a program, chances are excellent that you're not the first person to try building something very much like that program. Lots of details may be different, but even the details are probably not as unique as you might think. Suppose you need a program that sets up an account for a person, and then when the person does stuff, the details are checked with the person's account, and if OK are added to the account, and if not are rejected but recorded. This high-level description applies to a sales operation with customers, a medical office with patients, a manufacturing company with customers, and a bank. Given that programs like this have been built tens of thousands of times, don't you think that the ways of building programs like this would have evolved? That there would be pre-built parts, specialized tools for using the parts, building new ones and gluing them together? Special methods that have evolved and been refined to make sure it's done in the optimal way? Did any of you with any level of degree in Computer Science learn any of that — other than, of course, some boiler-plate about the virtues of object-oriented languages and maybe some other fashionable stuff? Of course not! Anyone with any sense of fashion and/or status knows that these are not career-advancing subjects. Anything involving software history or comparative study of software projects is career-killing.

    Conclusion

    Software evolves. But it evolves differently than other things studied by science. Virtually no one studies software history in any way, much less the patterns of evolution that become apparent when you study that history. There have been a couple of attempts to learn why software doesn't get better in the way many other things do, most notably Fred Brooks with his book The Mythical Man-Month and other writings. I have dissected this flawed analysis in my work on Software Project Management and Wartime Software. See this for a start.

    I have worked for years trying to identify some notable patterns in software evolution, and will be releasing some of that work in the coming months.

  • How We Got Chatbots for Mobile Financial Apps

    Chatbots for financial and other applications aren’t just a cool new thing. They’re a necessity! They solve the worsening problem of too many options to choose from on shrinking screens, with unhelpful help screens.

    Do you access your financial accounts online? If you do, perhaps you’ll remember that the first time you tried to do something, you had the fun of poring over the menu system to find what button to click, sometimes only to reach another screen full of buttons and menus. On the plus side, online financial systems let you get a lot done. On the minus side, jumping through the hoops to actually get them to do what you want can be a long slog. Have you ever gotten frustrated and tried to click on Help? And gotten the long, unhelpful Help stuff? Helped a whole lot, didn’t it?

    Today, nice big screens with high resolution, sitting on a desk somewhere, are used less and less. Cute, portable little screens with phone and camera built in are used more and more, partly because it’s in your pocket, right there when you need it. What happens to all those giant screens packed with menu choices? I guess the designers could have reduced the typeface so much that you’d need a magnifying glass to read them, but they bowed to reality and put just a few menu choices on each screen. Nice to read, but the result is that your multi-screen journey to get to the one you want got even longer. Assuming you remember how to get there. And what if you want something more elaborate, like how close am I getting to hitting my budget for restaurants this month? Fuggedabout it.

    The designers of mobile apps for financial applications aren’t between a rock and a hard place. It’s worse. They’re stuck way back in a long, narrow cave that floods. What to do?

    Echo and Siri have been training us to just ask for things. Who won the game last night? What’s the weather forecast? Even jokes! But banks … now that’s a different matter altogether. Banks are serious things. There’s money involved. Not just any money – MY money!

    So what’s a bank app creator supposed to do?

    It’s actually pretty simple, because there’s not much choice. If you want the bank app to be, you know, USED by the people who have it, you have to make it USABLE. Period. There’s not enough room for menus, except maybe a couple super-popular buttons. Help files? Waste of time. You’re down to one choice: make it so you can TALK (or chat) to the app, and ask it to do stuff for you.

    That’s the logic. Even better, it’s really happening. In real life!

    Bank of America is advertising its chatbot Erica heavily in some parts of the country. The reason is simple: they want their customers to be able to USE the BofA app, not be frustrated by it. There’s the chatbot technology provider Kasisto (Disclosure: Oak HC/FT is an investor) used by multiple banks to power human interactions. The bar has now been raised for financial institutions with apps.

    Let’s review some history. Years ago, what financial institutions had to have was a website for customers to access their accounts. As smartphones spread, the bar was raised: OK, you’ve got a website, but do you have an app? Not having an app was a reason for customers to move to a place that did. Just as most of the financial institutions were breathing sighs of relief that they’ve caught up, the bar is raised again: you mean we have to make the app USABLE? What’s a chatbot, anyway? The pattern here is clear: technology keeps marching along, some financial institution applies it to their customers’ and their own benefit, and the others scramble to catch up. What’s next?

    To find out what’s next, all we have to do is look at the non-financial domains that use chatbots. For example, look at Amazon’s Echo. It started out pretty primitive, but it’s adding capabilities all the time – as Amazon puts it, new “skills.” The bar is already being raised by technology vendors in two specific, technically challenging areas:

    It’s one thing to answer a simple question, like what’s my balance? But real chatting as done by people requires that the chatbot take into account complex questions, context and history.

    A complex question requires doing some real “thinking.” For example, “How much did I spend on eating out last month?” has loads of complexity. The system has to know that “eating out” means spending money on the merchant category “restaurants.”  It’s got to find all those transactions that took place in the month before today’s month, add them up and give you the answer.

    Taking conversation history and context in account is even trickier. Suppose you get the answer to the eating out question. Suppose you now ask “ How about the prior month?” Easy for a human; for a bot, not easy at all! The bot has to figure out that you’re still talking about eating out (restaurants) and that you want the total for a month. It’s also got to figure out that you want not last month, but the one before that. These are the kind of amazing interactions supported by Kasisto.

    Chatbots for financial apps are just being rolled out to solve the otherwise unsolvable problem of screen real estate. Even though we’re still in the early roll-out stages, the next battle is clear. Most of today’s chatbots are in elementary school – soon they’ll need to graduate to middle school!

    A slightly different version of this post originally appeared in Forbes.

  • Here’s what we can learn from the shift to smart credit card terminals

    I’ve been involved in computer software for decades. Lots has changed over that time. One thing that hasn’t changed is the question people most like to ask me. It’s this: “What do you see that’s new and interesting?” It’s a perfectly reasonable question, though one for which I rarely have a ready-made answer.

    A question I never hear goes something like this: “What do you see that’s touted as the newest new thing, but is mostly old stuff, and was completely predictable?” Now that’s an interesting question. And the un-helpful but honest answer is “Practically everything that’s touted as a new thing is mostly old stuff, with a little bit of ‘obvious next step’ thrown in for variety.”

    Still, there are some unpredictable aspects of the fancy new things: it’s really hard to know WHEN the new thing will happen and WHO will make it happen.

    A case in point is … [I’m not sorry about the pun] … the new smart card terminal company Poynt. (Disclosure: my VC fund, Oak HC/FT, is an investor.) I can see eager marketing people at Poynt are raising their hands in the back at this … ahem … point, all anxious to point (groan…) out that Poynt is a pioneer in the market, arguably the inventor of the smart terminal, an amazing device that not only takes card payments, but also rings up items just like a POS terminal and hosts endless numbers of third-party apps. True! I happily concede the point. But I hasten to point out that there are robust competitors in the market, notably including Square and Clover.

    The smart terminal is a new thing, and the market is glad to have it, but it’s hardly a NEW new thing, or something where you’d knock your head and say “now who’d-a thought-a that!?”

    It’s natural for consumers of technology to look at the new devices and appreciate them for what they are. That’s like being a tourist, driving on a road through the country-side, appreciating the nice new views. That’s nice for the tourist, but are there patterns here, patterns that would enable an educated person to expect something like a smart terminal to appear, and womdering when it would happen?

    Yes there are. The main pattern at work here is the rate of change of the underlying hardware. Today’s hardware is something like 1,000 times faster than much larger, more expensive hardware at the turn of the last century, less than 20 years ago. That number may not seem like much, but think of this: the average human walking speed is about 3 mph. The speed of a commercial jet while flying is less than 600 mph, about 200 times faster. Now imagine a human evolving so quickly that the human could walk at the speed of a jet — and increase in computer speed is 5 times greater than that, in less than 20 years!

    What’s the point? Or Poynt? Here it is: there are underlying geology-like forces in the world of computing that make it highly likely that something very much like Poynt would be invented – though as I said, predicting who will do it and when they’ll do it is a whole other thing.

    The first step in creating a technology solution to a problem is often building problem-specific hardware. Then the technology evolves, getting faster, cheaper and more capable. Then there’s a tipping point, at which the purpose-specific hardware is replaced by general-purpose hardware, and most of the specific features of the device are implemented in sofware that runs on the general-purpose hardware. Then a new era begins. The general pattern is that special-purpose devices are supplanted by general-purpose ones.

    In the case of card processing technology, first we see imprints of cards made on paper, with the physical paper being sent to a central place for processing. Then the big jump to computer technology and networking: a series of increasingly-better charge terminals, specifically made for processing card charges. The terminals evolved from dial-up networking to the internet, and from stand-alone to connected to a point-of-sale system. Wonderful devices!

    Now think about cell phones. If you’ve been around for a little while, you remember big phones getting better and smaller and finally evolving into flip phones. Great phones, … but they’re phones. Then came the big shift, to a next generation of phones that were really small, portable, general-purpose cmputers that can run a myriad of applications … with cell phone hardware and software built in. Yes, it’s a phone. But it runs Facebook, email, and any of thousands of applications avialable in the app store. It’s a “smart phone!”

    You know this. The reason I’m reminding you of that history is that it’s exactly the transition that card-charging “terminals” are going through right now – as they become “smart terminals,” i.e., small, portable, general-purpose computers that can run a myriad of applications … with card charging hardware and software built in. Yes, it’s a terminal, but a smart one.

    How often do you see “I’m just a phone” devices? Flip phones? Yup! The old card-charge terminal will become just as rare a sight in the next couple of years.

    So are the new “smart terminals” new? Yes! But hardly unexpected, at least to those who see the clearly repeating patterns of the underlying technology.

    A less Poynted version of this post was previously published at Forbes.

  • Continents and Islands in the World of Computers

    The vast majority of people appear to think that the world of computers and software is pretty uniform. While everyone recognizes that there are differences between the systems used by consumers and ones in business, it makes sense that pretty much the same thing is going on inside.

    The reality is that vast cultural and practical differences separate the various clusters of computer and software applications. I look forward to the first anthropological studies that are devoted to this subject, illustrating and spelling out the untravelled oceans that separate the diverse lands of computer and software practice. Meanwhile, there are both obstacles and opportunities that arise from these facts.

    A Diversity of Tongues

    The Bible gives a vivid explanation of how the various languages arose. In the post-flood world, there was said to be a single people with a single tongue. They built a city with a tower that reached to the sky, to make a name for themselves. Tour_de_babel.jpeg
    As a single people with a single tongue, "the sky was the limit" for how far they could go. God didn't like this. He reached down and confounded their speech, and scattered them over the face of the earth.

    Whether it's the fault of God or humans, it's well understood that groups of people develop their own languages, which then evolve and splinter. In fact, by studying the relationship of various languages, you gain insight into how humans migrated over the earth. The Indo-European family of languages is an excellent example of this.

    A Diversity of Software Languages

    Early computer programmers in the 1950's clearly saw the advantage of having a single language for software. They tried hard to create a universal software language, FORTRAN being the most well-known and successful such early language. While FORTRAN (short for "formula translator") was great for math people, people working with business records weren't impressed. Thus COBOL ("COmmon Business-Oriented Language") was invented. Things were still pretty simple in the mid-1950's. 1957

    But of course it didn't stop there. People migrated to different "lands," confronted new issues, and created new languages suitable for the new problems. Here's a snapshot of some of the major developments in the mid-1970's.

    1976

    By now, literally thousands of general-purpose languages have been created. Not including "esoteric" languages and thousands more languages for narrow problem domains.

    Idioms and Layers

    As anyone who has learned a new human language as an adult is well aware, learning a language is one thing — but learning all the incidental aspects of the language, particularly the idioms, is a task that never seems to end. This is because there are lots of them — an estimated 25,000 of them in English, for example.

    There are equivalents of idioms in software languages, in multiple categories. I'll just give a basic example: the run-time library, whose documentation frequently exceeds that of the base language, and without which you can't write practical programs. A more complex example: a "framework," for example the RAILS and Sinatra frameworks for the Ruby language.

    Let's say you know English pretty well. Are you qualified to write or even to read and understand a legal brief? Unless you're a lawyer, probably not. Yet there's no denying that the brief is written in English, with a whole pile of idioms and other special things that are not in common use.

    It's the same with computer languages. You may know the language C pretty well, but when you first look at the C code in a driver or an operating system kernel, it's got a strange idiom — "straight" C would stand out as obviously as BBC English would in Brooklyn. The C that's in a compiler is even more rife with idioms and unfamiliar constructs, so much so that a person who is otherwise fluent in C would have trouble figuring out what was going on, just as a normal fluent English-speaker would have trouble following a discussion by two doctors of a difficult medical case.

    Different Continents, Different Cultures

    Everyone knows that human languages are part of overall human culture. Cultures differ at least as much as languages. It's not widely appreciated that this is also true in the world of computers. While of course there are commonalities, you actually think differently in different languages, whether human or software. Lots of incidental things tend to get wrapped up in the cultures as well. To take a simple example: in the US, when you walk into a store, prices tend to be marked on the goods. If you like that thing at that price, you buy it; otherwise you don't. It doesn't work that way in much of South Korea. Prices may not be marked at all; and negotiation is assumed and expected. There are a whole set of cultural norms that have to be understood in order to thrive.

    Differences in software cultures are just as strong as differences in human cultures. For example, one of the major forces in the world of hospital automation is Epic. Epic is written in a language originally called MUMPS. While few write programs from scratch in MUMPS anymore, you have to use the language to customize the Epic system, just as you have to use ABAP to customize the SAP manufacturing system. In either case, learning the peculiar language is the tip of the iceberg — your programs "live in" the Epic system, and so knowing that system inside out is the key to success, far more important than utilization of the language itself. This is perhaps comparable to the importance of knowing all the relevant prior case law in writing a legal brief in support of your position, vs. simply knowing the vocabulary and syntax of English.

    Contact Between Distant Cultures

    We know that in human life, cultures developed in isolation from each other for many thousands of years. Migrating peoples would clash, and the cultures with superior instruments and warrior culture tended to decimate the others. A culture that develops superior methods of war, usually with distinct technology, tends to expand. This was true, for example of the Commanche in North America and the Mongols, each of which developed unique competence and technology with horses, and used that to rapidly expand their sphere of influence.

    This is an area of both similarity and difference with software cultures. Within a company, there are frequently members of different software "tribes," who usually neither understand nor like each other. They are constantly at war to establish primacy, the winning culture grudgingly conceding resource-poor reservations to which the losers are confined.

    It's different with software cultures that are separated by "oceans," usually different problem domains or industries. You might like to think that everyone who does software has all the information available, and therefore is at a similar cultural level, the equivalent of, say, the different countries in Europe. They may speak different languages and like different foods, but they all have cars, telephones, and electrical appliances. In software, this is not the case! In software, there are cultural differences that are the equivalent of cars being widely available in one place, while ox carts are the standard mode of transport in another. It's that extreme.

    What's even more shocking is that the denizens of these culturally isolated software continents are comfortable and secure in what outsiders see as their "backwardness" or "ignorance," and find ways to denigrate and disparage outsiders who dare to suggest there might be a better way of doing things.

    How big are these culturally retarded continents? Just a few distant places, the equivalent of Australia? If you have the opportunity to see a wide swath of software culture, what you find is that the vast majority of software groups have cultures that are dramatically inferior to the best places. Moreover, the variance in just how primitive things are is huge.  In any given place, it is likely that practices that are unknown but would dramatically enhance results are already standard practice in other places. Judging software isn't like auditing the financial books of a company, in which you either pass or fail an audit. It's more like figuring out which part of which software continent the place is, and how many years or decades behind the best known methods the place is, and to what extent their near neighbors are slightly ahead or behind.

    Conclusion

    It's useful to compare human language and culture to software language and culture. Just as with humans, language is an important part of culture, but thriving involves a whole lot more than just grasping the basics of a language. Just like with humans, there are varying levels and there are conflicts. But what's most interesting are the differences, which are the equivalent of humans living in environments that are physically next to each other, but using tools and methods that are hundreds of years different in terms of evolution. This fact has huge implications on many levels.

Links

Recent Posts

Categories