The Black Liszt

Category: Computer history

  • Summary: Computer Software History and Evolution

    This is a summary with links to my posts of the history and evolution of computer hardware and software.

    In computers and software, history is ignored and its lessons spurned. What little software history we are taught is often wrong. Everyone who writes or uses software pays for this, and pays big. This is true at a large scale for computers.

    https://blackliszt.com/2012/03/computer-history.html

    Software history can be spectacularly bad about important things.

    https://blackliszt.com/2019/08/the-comes-before-is-not-causation-fallacy-in-software-evolution.html

    Bad software history is also rife on a smaller scale, for example cryptography.

    https://blackliszt.com/2020/10/elizebeth-smith-friedman-the-cancelled-heroine-of-cryptography.html

    Software history is largely ignored.

    https://blackliszt.com/2015/06/dont-know-much-about-history.html

    When you learn math, you are learning its history. When you get to calculus, you’re essentially up to the 1600’s. You build from the ground up. Not in computing, to its detriment.

    https://blackliszt.com/2015/05/math-and-computer-science-vs-software-development.html

    Studying history would show modern software people who think they’re solving big problems that the problem is decades old and that the box of failed solutions is overflowing.

    https://blackliszt.com/2021/08/the-crisis-in-software-is-over-50-years-old.html

    Software development is a giant mess. Studying its history and the attempts to clean up the mess is an essential part of constructive change.

    https://blackliszt.com/2022/04/how-to-fix-software-development-and-security-a-brief-history.html

    The history of other fields can have lessons that are highly relevant for software.

    https://blackliszt.com/2012/07/what-can-software-learn-from-steamboats-and-antiseptic-surgery.html

    https://blackliszt.com/2014/02/lessons-for-software-from-the-history-of-scurvy.html

    https://blackliszt.com/2019/02/what-software-experts-think-about-blood-letting.html

    Some aspects of computer history are so important and relevant that, when understood and applied, they make dramatic improvements to software development.

    https://blackliszt.com/2013/12/fundamental-concepts-of-computing-speed-of-evolution.html

    Software Evolution

    Software evolves. But it evolves differently than other things studied by science. Virtually no one studies software history in any way, much less the patterns of evolution that become apparent when you study that history.

    https://blackliszt.com/2019/06/the-evolution-of-software.html

    Why should anybody bother to study software evolution? Knowing software evolution can help you predict the future! How? Software does not evolve the way most people think it does

    https://blackliszt.com/2019/09/understanding-software-evolution-helps-you-predict-the-future-of-software.html

    When studying software evolution it’s important to understand the underlying principles of automation – the things that computers and software are capable of doing.  The fundamentals will tell you WHAT will happen, but not WHO or WHEN.

    https://blackliszt.com/2020/01/the-fundamentals-of-computer-automation.html

    The way you learn software evolution is to study both its history in general and how it has been used in different industries and domains.

    https://blackliszt.com/2014/04/continents-and-islands-in-the-world-of-computers.html

    One clear pattern of evolution is that a kind of software will be established in one domain and then slowly, sometimes over decades, appear in other domains, one by one.

    https://blackliszt.com/2019/08/the-slow-spread-of-linear-programming-illustrates-how-in-old-vation-in-software-evolution-works.html

    A similar pattern often occurs with simple kinds of software.

    https://blackliszt.com/2019/09/simple-data-entry-technology-illustrates-how-in-old-vation-in-software-evolution-works.html

    A peculiar aspect of software evolution is software fashions. They appear, grow strongly and often fade away as their awfulness becomes hard to avoid. Then they come back, usually re-named.

    https://blackliszt.com/2019/05/recurring-software-fashion-nightmares.html

    A typical result of a fashion-driven software trend is to increase costs without delivering benefits, which can happen even when it involves an intrinsically good thing like workflow automation.

    https://blackliszt.com/2019/09/laser-disks-and-workflow-illustrate-the-insane-fashion-driven-nature-of-software-evolution.html

    One of the forces of evolution is resistance to automation.

    https://blackliszt.com/2020/01/luddites.html

    Software Programming Language Evolution

    Software people love to talk about progress and innovation. Most of what they claim as progress is little but milling around in a small box.

    https://blackliszt.com/2020/09/software-programming-languages-50-years-of-progress.html

    There is an odd evolution in programming languages, in which the relationship between data definitions and programs cycles from inside to outside the program and back.

    https://blackliszt.com/2015/06/innovations-that-arent-data-definitions-inside-or-outside-the-program.html

    However, there have been a couple major advances in the evolution of programming languages.

    https://blackliszt.com/2020/09/the-giant-advances-in-software-programming-languages.html

    While some truly powerful advances have become standard, others have been thoughtlessly discarded.

    https://blackliszt.com/2021/09/software-programming-language-evolution-structures-blocks-and-macros.html

    Other powerful advances, widely used for a time in the market, ended up being ignored or abandoned by academia and industry in favor of a productivity-killing combination of tools and technologies.

    https://blackliszt.com/2020/11/software-programming-language-evolution-beyond-3gls.html

    The emergence from academia of “structured programming” and the associated effort to find all GOTO statements and burn them at the stake was a particularly shameful instance of the evolution of programming languages.

    https://blackliszt.com/2021/09/software-programming-language-evolution-the-structured-programming-goto-witch-hunt.html

    While loads of people focus on language, programming tools that make huge contributions to the productivity of programmers have evolved: libraries and frameworks.

    https://blackliszt.com/2021/10/software-programming-language-evolution-libraries-and-frameworks.html

    Some programming languages that were supposed to be so much better than older ones led to major failures. But failures are expected in the normal world of software development, so the bad new languages kept marching along.

    https://blackliszt.com/2021/01/software-programming-language-evolution-credit-card-software-examples-1.html

    The small world of functional software languages has an interesting history.

    https://blackliszt.com/2021/05/software-programming-language-evolution-functional-languages.html

    Software Applications and Systems Evolution

    One of the recurring evolution patterns is that functionality emerges on a new platform in roughly the same order as it emerged on earlier platforms. The timescale of the emergence may be compressed; the important aspect of the pattern isn’t the timing but the order. Here is an explanation of the general concept and how it worked for operating systems.

    https://blackliszt.com/2020/10/software-evolution-functionality-on-a-new-platform.html

    Here is how the same pattern of functionality on a new platform worked out in security services.

    https://blackliszt.com/2020/11/software-evolution-functionality-on-a-new-platform-security-services.html

    Here’s the pattern of existing functionality on a new platform as seen for remote access software.

    https://blackliszt.com/2020/11/software-evolution-functionality-on-a-new-platform-remote-access.html

    Just because you’re building a version of existing functionality for a new platform doesn’t mean you’ll succeed. You can screw it up in various ways, including being too early – if the market doesn’t think it has a problem, it won’t buy a solution.

    https://blackliszt.com/2020/11/software-evolution-functionality-on-a-new-platform-market-research.html

    Transaction monitors are a classic example of the pattern of functionality emerging on a technology platform, and then emerging in pretty much the same way in the same order on other platforms.

    https://blackliszt.com/2021/02/software-evolution-functionality-on-a-new-platform-transaction-monitors.html

    Once an application appears on a software platform, there is a consistent way the category of applications evolves on that platform. It goes from a custom application to a basic product, then parameters are added and finally a workbench.

    https://blackliszt.com/2020/01/the-progression-of-abstraction-in-software-applications.html

    Here are examples of the progression from prototype through increasing levels of abstraction.

    https://blackliszt.com/2021/03/the-progression-towards-abstraction-on-a-software-platform-examples.html

    Software automates human effort to varying degrees. One dimension of automation is depth, in which software evolves from recording what people do through helping them and eventually to replacing them.

    https://blackliszt.com/2021/04/the-dimension-of-software-automation-depth.html

    There is surprising pain and trouble of going from one stage of automation depth to the next. Unlike the progression to increasing levels of abstraction, customers tend to resist moving to the next stage of automation for various reasons including the fear of loss of control and power.

    https://blackliszt.com/2021/05/the-dimension-of-software-automation-depth-examples.html

    Here’s how the automation depth pattern plays out in “information access,” the set of facilities that enable people to find and use computer-based information for decision making.

    https://blackliszt.com/2021/12/the-dimension-of-automation-depth-in-information-access.html

    The patterns of automation play out in multiple dimensions. In addition to automation depth, there is automation breadth.

    https://blackliszt.com/2021/11/the-dimension-of-software-automation-breadth.html

    Here are a couple of clear examples of the evolution of software applications to increasing automation breadth.

    https://blackliszt.com/2022/07/the-dimension-of-software-automation-breadth-examples.html

    The evolution of spreadsheets is a good example of how these patterns work out in history, and clearly demonstrate the predictive power they have for anyone who cares to look.

    https://blackliszt.com/2023/09/software-evolution-spreadsheet-example.html

    As software evolves in these ways, it becomes increasingly nuts for a potential customer to build it for themselves. However, there are a surprising number of cases where a smart user can build a private version of commercially available software and win with it.

    https://blackliszt.com/2023/09/software-evolution-build-buy-then-build.html

    You would think that user interfaces would be thoroughly understood. When you look at a variety of UI’s you see that the underlying principles are clear but usually ignored.

    https://blackliszt.com/2020/12/software-evolution-user-interface.html

    https://blackliszt.com/2021/01/software-evolution-user-interface-concepts-whose-perspective.html

    https://blackliszt.com/2020/01/how-to-design-software-user-interfaces-that-take-less-time.html

    Does Software Always Evolve?

    There is a large body of core software that evolves extremely slowly.

    https://blackliszt.com/2023/07/does-software-evolve-rapidly.html

    Once a kind of software gets built, it tends to live long past the problem for which it was the solution. Data was so big you needed storage. Special SW invented to handle it. Now it will all fit in memory (for most applications). But the SW and code practices live on!

    https://blackliszt.com/2010/09/databases-and-applications.html

    Yes, they’ve been obsoleted by modern hardware – but needed by software.

    https://blackliszt.com/2010/01/paleolithic-mainframes-discovered-alive-in-data-center.html

    While there are things that change in software and management, a surprising number of common things just get new names. The “cloud” is a typical example.

    https://blackliszt.com/2011/12/the-name-game-of-moving-to-the-cloud.html

    Another example is pure software, shown with SOA and micro-services.

    https://blackliszt.com/2021/03/micro-services-the-forgotten-history-of-failures.html

    One of the explosive growth areas of the internet boom was the invention and spread of social media. Most of the people who look at it ignore its deep roots in non-digital media.

    https://blackliszt.com/2018/10/social-media-has-a-long-history.html

    If and when serious study of computer software history and evolution finally starts to take place, perhaps Computer Science will start on the path to being, you know, scientific. And normal software development will stop being dominated by fashions.

    https://blackliszt.com/2023/04/summary-computer-science.html

    https://blackliszt.com/2023/07/summary-software-fashions.html

     

  • Flowcharts and Workflow in Software

    The concept of workflow has been around in software from the beginning. It is the core of a great deal of what software does, including business process automation. Workflow is implicitly implemented in most bodies of software, usually in a hard-coded, ad-hoc way that makes it laborious and error-prone to implement, understand, modify and optimize. Expressing it instead as editable declarative metadata that is executed by a small body of generic, application-independent code yields a huge increase in productivity and responsiveness. It also enables painless integration of ML and AI. There are organizations that have done exactly this; they benefit from massive competitive advantage as a result.

    Let’s start with some basics about flowcharts and workflow.

    Flowcharts

    Flowcharts pre-date computers. The concept is simple enough, as shown by this example from Wikipedia:

    Fix lamp

    The very earliest computer programs were designed using flowcharts, illustrated for example in a document written by John von Neumann in 1947. The symbols and methods became standardized. By the 1960’s software designers used templates like this from IBM

    Flowchart

    to produce clean flowcharts in standardized ways.

    Flowcharts and Workflow

    Flowcharts as a way to express workflows have been around for at least a century. Workflows are all about repeatable processes, for example in a manufacturing plant. People would systematize a process in terms of workflow in order to understand and analyze it. They would create variations to test to see if the process could be improved. The starting motivation would often be consistency and quality. Then it would often shift to process optimization – reducing the time and cost and improving the quality of the results. Some of the early work in Operations Research was done to optimize processes.

    Workflow is a natural way to express and help understand nearly any repeatable process, from manufacturing products to taking and delivering orders in a restaurant. What else is a repeatable process? A computer program is by definition a repeatable process. Bingo! Writing the program may take considerable time and effort, just like designing and building a manufacturing plant. But once written, a computer program is a repeatable process. That’s why it made sense for the very earliest computer people like John von Neumann to create flowcharts to define the process they wanted the computer to perform repeatedly.

    What’s in a Flowchart?

    There are different representations, but the basic work steps are common sense:

    • Get data from somewhere (another program, storage, a user)
    • Do something to the data
    • Test the data, and branch to different steps depending on the results of the test
    • Put the data somewhere (another program, storage, a user)
    • Lots of these work steps are connected in a flow of control

    This sounds like a regular computer software program, right? It is! When charted at the right level of detail, the translation from a flowchart to a body of code is largely mechanical. But humans perform this largely mechanical task, and get all wrapped up in the fine details of writing the code – just like pre-industrial craftsmen did.

    Hey, that's not just a metaphor — it is literally true! The vast, vast majority of software programming is done in a way that appears from the outside to be highly structured, but in fact is designing and crafting yet another fine wood/upholstery chair (each one unique!) or, for advanced programmers, goblets and plates made out of silver for rich customers.

    Workflow

    In the software world, workflow in general has been a subject of varying interest from the beginning. It can be applied to any level of detail. It has led to all sorts of names and even what amount to fashion trends. There is business process management. Business process engineering. And re-engineering. And business process automation. A specialized version of workflow is simulation software, which led early programmers to invent what came to be called "object-oriented programming." To see more about this on-going disaster that proved to be no better for simulating systems than it has been for software in general, see this.

    When document image processing became practical in the 1980’s, the related term workflow emerged to describe the business process an organization took to process a document from its arrival through various departments and finally to resolution and archiving of the document. The company that popularized this kind of software, Filenet, was bought by IBM. I personally wrote the workflow software for a small vendor of document image processing software at that time.

    Workflow in practice

    There has been lots of noise about what amounts to workflow over the years, with books, movements and trends. A management professor in the 1980's talked about how business processes could be automated and improved using business process re-engineering. He said that each process should be re-thought from scratch — otherwise you would just be "paving the cow paths," instead of creating an optimal process. As usual, lots of talk and little action. Here's the story of my personal involvement in such a project in which the people in charge insisted they were doing great things, while in fact they were spending lots of money helping the cows move a bit faster than they had been.

    The Potential of Workflow

    The potential of workflow can be understood in terms of maps and driving from one place to another. I've explained the general idea here.

    Most software design starts with the equivalent of figuring out a map that shows where you are and where you want to get to. Then the craftsmanship begins. You end up with a hard-coded set of voluminous, low-level "directions" for driving two blocks, getting in the left lane, turning left, etc.

    When the hard-coded directions fail to work well and the complaints are loud enough, the code is "enhanced," i.e., made even more complex, voluminous and hard to figure out by adding conditions and potential directional alternations.

    Making the leap to an online, real time navigation system is way beyond the vast majority of software organizations. You know, one that takes account of changes, construction, feedback from other drivers on similar routes about congestion, whether your vehicle has a fee payment device installed, whether your vehicle is a truck, etc. Enhancements are regularly made to the metadata map and the ML/AI direction algorithms, which are independent of map details.

    When software stays at the level of craftsmanship, you're looking at a nightmare of spaghetti code. Your cow paths aren't just paved — they have foundations with top-grade rebar, concrete and curbs crafted of marble.

    Conclusion

    Metadata-driven workflow is the next step beyond schema enhancement for building automated systems to perform almost any job. It's a proven approach that many organizations have deployed — literally for decades. But all the leaders of computing, including Computer Science departments at leading universities, remain obsessed with subjects that are irrelevant to the realities of building software that works; instead they stay focused on the wonders of craftsman-level low-level software languages. It's a self-contained universe where prestige is clearly defined and has nothing to do with the eternal truths of how optimal software is built.

     

  • Cartoons and Video games evolved into Bitcoin and NFT’s

    Bitcoin and other cryptocurrencies are in the news. NFT’s (non-fungible tokens) have exploded onto the scene, with people spending large amounts of money to acquire unique rights to digital images. The explosion of invention and innovation is amazing, isn’t it?

    Except that it's all just minor variations of things that were created decades ago, grew into huge markets with the participation of a good part of the world's population, and continue to grow today. Invention? Creativity? How about minor variations of proven ideas, giving them a new name and slightly different context, and getting super-rich?

    From Drawing to Cartoons to Video Games

    Drawing, sculpting and otherwise creating artificial images of the reality we experience has a long history.

    For example, here’s a painting of a bovine from a cave created by early humans over 40,000 years ago:

    Lubang_Jeriji_Saléh_cave_painting_of_Bull

    Drawings that suggest reality but are purposely different from real things are called cartoons, and go back hundreds of years, becoming more widespread in the 1800’s in print media.

    Then there was a breakthrough: animation. Leveraging early movie technology, artists worked enormously hard to create a fast-changing sequence of images to create the illusion of motion. Along with sound, you could now go to a theater and watch and hear a whole cartoon movie, filled with characters and actions that could never happen in real life. Characters like Mickey Mouse and Bugs Bunny became part of modern culture.

    The next big step took place after computers were invented and got video screens. Of course the computers transformed the process of creating animation. But animation was always like watching a movie: the human could only watch and listen. With computers, the possibility first arose for actions of the person to directly and immediately change what happened on the screen. The video game was born.

    The video game has gone through an extensive evolution from the primitive, simple Space War to immersive MMORPG's (massively multiplayer online role-playing games), enabling players to interact with each other in evolving shared animated worlds, often with fighting but also including other activities.

    World of Warcraft (WoW) wasn't the first, but became the most popular of the MMORPG's.

    Similar to other MMORPGs, the game allows players to create a characteravatar and explore an open game world in third– or first-person view, exploring the landscape, fighting various monsters, completing quests, and interacting with non-player characters (NPCs) or other players. The game encourages players to work together to complete quests, enter dungeons and engage in player versus player (PvP) combat, however the game can also be played solo without interacting with others. The game primarily focuses on character progression, in which players earn experience points to level up their character to make them more powerful and buy and sell items using in-game currency to acquire better equipment, among other game systems.

    World of Warcraft was a major critical and commercial success upon its original release in 2004 and quickly became the most popular MMORPG of all time, reaching a peak of 12 million subscribers in 2010.[4] The game had over one hundred million registered accounts by 2014[5] and by 2017, had grossed over $9.23 billion in revenue, making it one of the highest-grossing video game franchises of all time. The game has been cited by gaming journalists as the greatest MMORPG of all time and one of the greatest video games of all time

    The industries creating hardware and software for these artificial worlds has grown to be huge. In 2020 video gaming generated over $179 billion in global revenue, having surpassed the film industry years before.

    Video games aren’t just for kids. There are an estimated 3.24 billion gamers across the globe.

    In the US the numbers are huge. “Three out of every four, or 244 million, people in the U.S. play video games, an increase of 32 million people since 2018." Gamers spend lots of time on their games: “… gamers average 14 hours per week playing video games.”

    Game World and Virtual Economies

    Huge numbers of people go to a screen or put on a headset and "enter" the world of a video game, where they often spend hours at a time. While in that world, they can move from place to place as an observer, or as the controller of their personal avatar. They can interact with others, as shown by this scene from the virtual world of Second Life in 2003.

    Second_Life_11th_Birthday_Live_Drax_Files_Radio_Hour

    Long before Bitcoin was created, video games had virtual economies with digital currencies.

    The currency used in a game world can be called different things. For example in World of Warcraft it's called — big shock coming up here — Gold. Gold can be earned by players accomplishing things in the game world, and can be spent for skills or in-game objects. Players can buy and sell items among themselves using such currencies. Many games enable players to buy in-game currencies using real money. In some cases, in-game virtual "land" is also for sale.

    Long before Bitcoin, markets arose to enable in-game currencies to be traded (exchanged) for real-world currencies. It is now a multi-billion dollar industry. "In 2001, EverQuest players Brock Pierce and Alan Debonneville founded Internet Gaming Entertainment Ltd (IGE), a company that offered not only the virtual commodities in exchange for real money but also provided professional customer service." The company was the largest such on-line exchange and accounted for hundreds of millions of dollars of transactions.

    Video Games, Bitcoin and NFT's

    The first Bitcoin was sent in 2009. It wasn't much used or valued until 2013. Ethereum first went live in 2014. By this time there were already MMORPG's with many hundreds of millions of players earning, spending and exchanging digital currencies involving virtual objects in their game worlds.

    Let's see how the things used by literally billions of gamers compares to Bitcoin (and other crypto-currencies) and NFT's.

    • Games have digital currencies with no real-world value.
      • Sounds like Bitcoin and other crypto-currencies
    • In-game virtual objects can be bought and sold using in-game currencies
      • Sounds like buying crypto-world NFT's with Bitcoin
    • New units of the digital currency are created by the game software
      • New crypto is created by Bitcoin mining software
    • Game currencies can be used and exchanged among gamers
      • Same with Bitcoin
    • Game currencies can be exchanged for and bought with real-world money
      • Same with Bitcoin
    • There are exchanges outside the game that enable buying/selling
      • Same with Bitcoin
    • The exchange price can vary greatly
      • Same with Bitcoin
    • Teams create new games with currencies and virtual objects
      • Teams create new crypto-currencies and NFT's

    Still think there's no relationship between gaming and crypto? How about, as mentioned above, the fact that Brock Pierce and a partner founded the game currency exchange IGE in 2001, and the same Mr. Pierce was active in crypto-currency by 2013 and became a "Bitcoin billionaire" by 2018.

    Of course, the new worlds of crypto and NFT's are different in some important ways from the gaming worlds. Games along with the objects and currencies are created and managed by the game company. While there's more control than is generally recognized, crypto-currencies have a large degree of self-management with their built-in miners. Similarly, NFT's are created independently

    Conclusion

    First Bitcoin came seemingly out of nowhere in 2009. A few years later, variations of Bitcoin appeared on the market. An astounding explosion of crypto followed, along with digital objects that "live" in the crypto world.

    Like many other "brand new" things, the worlds of crypto and NFT's have remarkably close relations to the world of gaming, from which they appear to have evolved. Compared to the gaming world, the number of people invested in crypto is truly tiny, hundredths of a percent. But the inflation and amount of real-world currency that has been converted to crypto dwarfs the amounts in the gaming world.

    As with many other tech trends, the history and evolution of the elements of the trend reward study.

    Note: this was originally published on Forbes.

  • How to Fix Software Development and Security: A Brief History

    As the use of computers grew rapidly in the 1960’s, the difficulty of creating quality software that met customer needs became increasingly evident. Wave after wave of methods were created, many of them becoming standard practice – without solving the problem! This is a high-level survey of the failed attempts to solve the problem of software development and security — all of which are now standard practice in spite of failing to fix the problem! There are now software auditing firms that will carefully examine a software organization to see in which ways it deviates from ineffective standard practice – so that it can be “corrected!”

    Houston, We've Had a problem

    The astronauts of the Apollo 13 mission to the Moon radioed base control about the problem that threatened the mission and their lives.

    Apollo_13_fd4_capcom_lousma_with_slayton_mattingly_brand_and_young_apr_14_1970_s70-34902

    The astronauts were saved after much effort and nail-biting time.

    Should the people in charge of software make a similar call to mission control? Yes! They have made those calls, and continue to make them, thousands of times per day!!

    Getting computers to do what you want by creating the kind of data we call software was a huge advance over the original physical method of plugs and switches. But it was still wildly difficult. Giant advances were made in the 1950’s that largely eliminated the original difficulty.

    As the years went on into the 1960’s, the time, cost and trouble of creating and modifying software became hard to ignore. The problems got worse, and bad quality surfaced as a persistent issue. There were conferences of experts and widely read books, including one from a leader of the IBM 360 Operating System project, one of the largest efforts of its kind.

    Mythical_man-month_(book_cover)

    The Apollo 13 astronauts were saved, but the disaster in software development has resisted all the treatments that have been devised for it. With the invention and spread of the internet, we are now plagued with cybercrime, to the extent that ransomware attacks now take place (as of June 2021) … get ready for it … 149,000 times per week!! I think a few more exclamation points would have been appropriate for that astounding statistic, but I'm a laid-back kind of guy, so I figured I'd stick with a mild-mannered two. Here's my description from more than four years ago on the state of ransomware and the response of the "experts."

    Following are highlights of the methods that have been devised to solve the problem of software development and security. None of which have worked, but have nonetheless become standard practice.

    Programming Languages

    The assembled experts in 1968 decided that "structured programming" would solve the problem of writing software that worked. Here's how that turned out. A group of experts recently convened to survey 50 years of progress in programming languages. They were proud of themselves, but had actually made things worse. Highlights here. One of those early efforts led to object-oriented languages, which are now faith-based dogma in the evidence-free world of Computer Science. New languages continue to be invented that claim to reduce programmer errors and increase productivity. What always happens is that the "advances" are widely publicized while the large-scale failures are concealed; here are a couple of juicy examples. Above all, the flow of new languages provides clear demonstration that programmers don't have enough useful work to do to keep them busy.

    Outsourcing

    While programmers babbled among themselves about this language or that method, company managers couldn't help but noticing that IT budgets continued to explode while the results were an avalanche of failures. Service companies emerged that promised better results than company management could achieve because they claimed to be experts in software and its management.

    One of the pioneers in computer outsourcing was Ross Perot's Electronic Data Systems (EDS), which had $100 million in revenue by 1975. By the mid-1980's it had over 40,000 employees and over $4 billion in revenue. It continued to grow rapidly. along with a growing number of competitors including services branches of the major computer vendors. In a typical deal, a company would turn over some or all of its hardware and software operations to the outsourcer, who would add layers of management and process. They succeeded by lowering expectations and hiding the failures.

    Off-shoring

    As communications and travel grew more cost effective, basing outsourced operations in another country became practical. Off-shoring had the huge advantage that while the results were no better than regular outsourcing, the fact that employees were paid much less than US wages enabled marginally lower prices and a huge infrastructure of management and reporting which further disguised the as-usual results.

    US-based outsourcers started doing this fairly early, while non-US vendors providing software services grew rapidly. Tata Computer Services first established a center in the 1980's and now has revenue over $20 billion and has been the world's most valuable IT company, employing over 500,000 people. Infosys is another India-based giant, along with hundreds of others.

    Project Management

    Deeper investment in the growing and evolving collection of project management methods has been a key part of software organizations. The larger the organization, the greater the commitment to widely accepted, mainstream project management methods. As new methods enter the mainstream, for example Agile, new managers and outsourcing organizations sell their embracing of the methods as a reason for hiring and contracting managers to feel confident in the decisions they are making. Project management has definitely succeeded in forming a thick layer of obfuscation over the reality of software development, quality and security. The only method of project management that has ever delivered "good" results for software is the classic method of wild over-estimation. See this for one of the conceptual flaws at the heart of software project management, see this for an overview and this for a comprehensive look.

    Education and Certification

    Nearly all professions have education and certification requirements, from plumbers and electricians to doctors. Software organizations grew to embrace imposing education and certification requirements on their people as a sure method to achieve better outcomes. This came to apply not to just the developers themselves but also to the QA people, security people and project managers. Unlike those other professions, computer education and certification has been distinctly divorced from results.

    Standards, Compliance and Auditing

    Standards were developed early on to assure that all implementations of a given software language were the same. As software problems grew, standards were increasingly created to solve the problem. The standards grew to include fine-grained detail of the development process itself and even standards that specified the "maturity level" of the organization.

    Just as a new house is inspected to assure that it complies with the relevant building code, including specialists to inspect the plumbing and electricity, the software industry embraced a discipline of auditing and compliance certification to assure that the organization was meeting the relevant standards. A specialized set of standards grew for various aspects of computer security, with different measures applied to financial transactions and healthcare records for example. The standards and compliance audits have succeeded in consuming huge amounts of time and money while having no positive impact on software quality and security.

    Suppress, deflect and Ignore

    Everything I've described above continues apace, intent on its mission of making software development successful and computer security effective. People who write code spend much of their time engaged in activities that are supposed to assure that the code they write meets the needs in a timely manner and has no errors, using languages that are supposed to help them avoid error, surrounded by tests created by certified people using certified methods to assure that it is correct. Meanwhile highly educated certified computer security specialists assure that security is designed into the code and that it will pass all required audits when released.

    How is this working out? In spite of widespread information blackouts imposed on failures, enough failures are so blatant that they can't be suppressed or ignored that we know that the problems aren't getting better.

    Conclusion

    We're over 50 years into this farce, which only continues because the clown show that sucks in all the money is invisible to nearly everyone. The on-the-scene reporters who tell all the spectators what's happening on the field can't see it either. What people end up hearing are a series of fantasies intended to deliver no surprises, except in those cases where the reality can't be hidden. Mostly what people do is things that make themselves feel better while not making things better. The time for a major paradigm shift to address these problems has long since passed.

  • Software Programming Language Evolution: Structures, Blocks and Macros

    In prior posts I’ve given an overview of the advances in programming languages, described in detail the major advances and defined just what is meant by “high” in the phrase high-level language. In this post I’ll dive into the additional capabilities added to 3-GL’s that brought them to a peak of productivity.

    History

    Let’s remember what high-level languages are all about: productivity! They are about the amount of work it takes to write the code and how easy code is read.

    The first major advance, from machine language to assembler, was largely about eliminating the grim scut-work of taking the statements you wanted to write and making the statements “understandable” to the machine by expressing them in binary. Ugh.

    The second major advance, to 3-GL’s like FORTRAN and COBOL, was about eliminating the work of translating from your intention to the assembler statements required to express that intention. A single line of 3-GL code can easily translate into 10 or 20 lines of assembler code. And the 3-GL line of code often comes remarkably close to what you actually want to “say” to the computer, both writing it and reading it.

    FORTRAN achieved this goal to an amazing extent.

    “with the first FORTRAN compiler delivered in April 1957.[9]:75 This was the first optimizing compiler, because customers were reluctant to use a high-level programming language unless its compiler could generate code with performance approaching that of hand-coded assembly language.[16]”

    “While the community was skeptical that this new method could possibly outperform hand-coding, it reduced the number of programming statements necessary to operate a machine by a factor of 20, and quickly gained acceptance.”

    The reduction in the amount of work was the crucial achievement, but just as important was the fact that each set of FORTRAN statements were understandable, in that they came remarkably close to expressing the programmer’s intent, what the programmer wanted to achieve. No scratching your head when you read the code thinking to yourself “I wonder what he’s trying to say here??” This meant easier to write, fewer errors and easier to read.

    Enhancing conditional branching and loops

    The very first iteration of FORTRAN was an amazing achievement, but it’s no surprise that it wasn’t perfect. At an individual statement level it was nearly perfect. When reading long groups of statements there were situations where the code was clear, but the intention of the programmer not clearly expressed in the code itself – it had to be inferred from the code.

    The first FORTRAN had a couple of the intention-expressing statements: a primitive IF statement and DO loop. Programmers soon realized that more could be done. The next major version was FORTRAN 66, which cleaned up and refined the early attempts at structuring. Along with the appropriate use of in-line comments, it was nearly as clear and intention-expressing as any practical programmer could want.

    The final milestone in the march to intention-expressing languages was C.

    It’s an amazing language. While FORTRAN was devised by people who wanted to do math/science calculations and COBOL by people who wanted to do business data processing, C was devised to enable computer people to write anything – above all “systems” software, like operating systems, compilers and other tools. In fact, C was used to re-write the first Unix operating system so that it could run on any machine without re-writing. C remains the language that is used to implement the vast majority of systems software to this day.

    I bring it up in this context because C added important intention-expressing elements to its language that have remained foundational to this day. It enhanced conditional statements, creating the full IF-THEN-ELSE-ENDIF that has been common ever since. This meant you could say IF <condition is true> THEN <a statement>. The statement would only be executed if the condition was true. You could tack on ELSE <another statement> ENDIF. This isn’t dramatic, but the only other way to express this common thought is with GOTO statement – which certainly can be understood, but takes some figuring. In addition, C added the ability to use a delimited block of statements wherever a single statement could be used. When there are a moderate number of statements in a block, the code is easy to read. When there would be a large number, a good programmer creates a subroutine instead.

    C added a couple more handy intention-expressing statements. A prime example is the SWITCH CASE BREAK statement. This is used when you have a number of conditions and something specific to do for each. The SWITCH defines the condition, and CASE <value> BREAK pairs define what to do for each possible <value> of the SWITCH.

    Lots of following languages have added more and more statements to handle special cases, but the cost of a more complex language is rarely balanced with the benefit in ease and readability.

    The great advance that's been ignored: Macros

    C did something about all those special cases that goes far beyond the power of adding new statements to a language, and vastly increases not just the power and readability of the language but more important the speed and accuracy of making changes. This is the macro pre-processor.

    I was already very familiar with macros when I first encountered the C language, because they form a key part of a good assembler language – to the extent that an assembler that has such a facility is usually called a macro-assembler. Macros enable you to define blocks of text including argument substitution. They resemble subroutine calls, but they are translated at compile time to code which is then compiled along with the hand-written code. A macro can do something simple like create a symbol for a constant that is widely used in a program, but which may have to be changed. When a change is needed, you just change the macro definition and – poof! – everywhere it’s used it has the new value. It can also do complex things that result in multiple lines of action statements and/or data definitions. It is the most powerful and extensible tool for expressing intention and enabling rapid, low-risk change in the programmer’s toolbox. While the C macro facility isn’t quite as powerful as the best macro-assemblers, it’s a zillion times better than not having one at all, like all the proud but pathetic modern languages that wouldn’t know a macro if it bit them in the shin.

    The next 50 years of software language advances

    The refrain of people who want to stay up to date with computers is, of course, "What's new?" Everyone knows that computers evolve more quickly than anything else in human existence, by a long shot. The first cell phones appeared not so long ago, and they were "just" cell phones. We all know that now they're astounding miniature computers complete with screens, finger and voice input, cameras and incredible storage and just about any app you can think of. So course we want to know what's new.

    The trouble is that all that blindingly-fast progress is in the hardware. Software is just along for the ride! The software languages and statements that you write are 99% the same as in the long-ago times before smart phones. Of course there are different drivers and things you have to call, just as in any hardware environment. But the substance of the software and the lines of code you use to write it are nearly the same. Here is a review of the last 50 years of "advances" in software. Bottom line: it hasn't advanced!

    Oh, an insider might argue: what about the huge advances of the object-oriented revolution? What about Google's new powerhouse, Go? Insiders may ooh and ahh, just like people at a fashion catwalk. The come-back is simple: what about programmer productivity? Claims to this effect are sometimes made, but more often its that the wonderful new language protects against stupid programmers making errors or something. There has not been so much as a single effort to measure increase of programmer productivity or quality. There is NO experimental data!! See this for more. Calling what programmers do "Computer Science" is a bad joke. It's anything but a science.

    What this means is simple: everyone knows the answer — claims about improvement would not withstand objective experiments — and therefore the whole subject is shut down. If you're looking for a decades-old example of "cancel culture," this is it. Don't ask, don't tell.

    Conclusion

    3-GL's brought software programming to an astounding level of productivity. Using them you could write code quickly. The code you wrote came pretty close to expressing what you wanted the computer to do with minimal wasted effort. Given suitable compilers, the code could run on any machine. Using a 3-GL was at least 10X more productive than what came before for many applications.

    A couple language features were incorporated into the very first languages, like conditional branching and controlled looping, that were good first steps. The next few years led early programmers to realize that a few more elaborations of conditional branching and controlled looping would handle the vast majority of practical cases. With those extra language features, code became highly expressive. All subsequent languages have incorporated these advances in some form. Sadly, the even more productive feature of macros has been abandoned, but as we'll see in future posts, their power can be harnessed to an even greater extent in the world of declarative metadata.

  • The Crisis in Software is over 50 years old

    Everyone knows there’s a problem in software. No one likes to talk about it. Waves of new tools and techniques are introduced, hyped and fade away. Are there waves of new tools and techniques in accounting? During interviews, are accountants asked if they practice the currently “hot” methods? Of course not! What about cars? Hah, you might say – look at the wave of new electric cars! Yes, let’s look at them – do electric cars crash more than normal ones? If cars crashed or mis-performed at the same rate as software, we would all be afraid to ride in them!

    The existence of a software crisis was first prominently identified in 1968. The results of the crisis were clear at that time, and haven’t changed a bit since then. Nothing – no wonderful new language, paradigm, project management or quality method, architectural technique or anything else has made a dent in the crisis.

    The solutions to the crisis have largely been identified and repeatedly proven in practice by small groups of programmers who desperately need to get things done. Their methods are ignored or suppressed by the ruling elites in academia, government and corporations – including Big Tech. Sadly, this is unlikely to change any time soon. Happily, effective methods for building software that are ignored and suppressed provide a way for small, ambitious groups to do new stuff and win!

    What is the software crisis?

    The term “software crisis” was coined during the 1968 NATO Software Engineering Conference. Many computer science experts attended. There was general agreement that there was a serious problem. Part of the solution was identified by creating a discipline of “computer engineering,” which would structure and formalize the techniques for building software. Papers identifying the problems and solutions were generated as a result of the conference and a similar one held the following year.

    The problems are summarized in the Wiki article:

    Software crisis

    Look familiar?

    How do we know there’s a crisis?

    When things are really bad, it’s typical for people to avoid talking about it. When there’s a whiff of a solution, though, it’s likely to receive attention. When people engage in something that’s likely to go wrong, most of them will get the best advice and be sure to follow widely accepted practices in order to avoid trouble. If something goes wrong anyway, having followed the standard accepted techniques provides a good defense – I did what people in my position are supposed to do! When hiring people, the hiring organization will often trumpet their use of these authorized and accepted methods to assure avoidance of blame. And finally, when things go wrong, great care is taken to assure that as few people know about the problem as possible.

    When you hire someone in physics, do you advertise that the position requires knowledge and acceptance of relativity theory? When you hire a doctor, do you need to make sure that the doctor adheres to the infectious theory of disease?

    We know there’s a crisis because, in spite of all the effort to keep it quiet, disasters happen that are so public, widespread and annoying that they get talked about. The trouble is that the highly publicized disasters are just the visible tip of a truly gigantic iceberg that takes up most of the space in the ocean of software.

    We know there's a crisis because the academic field devoted to the subject, Computer Science, isn't a "science." Not even close. Check out these.

    We know there's a crisis because the famous big tech companies have amazing reputations, like they're all geniuses — while the facts show that they're pretentious bumblers with astounding salaries.

    We know there's a problem because of the fashion-driven nature of the field. Here is a discussion of specific fashions, and see these for examples.

    Conclusion

    Look again at the list of problem identified over 50 years ago: software that's late, over budget, not what you wanted, poor quality, and sometimes has to be thrown out. This is one of the reasons managers take estimates from programmers and multiply the time by 3X before passing them on up the chain. Higher managers pad even more. People try to avoid publicizing the wonderful thing that's coming real soon because all too often, days before completion, it blows up.

    There have been decades of widely-touted solutions to the problem that never solve the problem. Here is a review of 50 years of "progress" in programming languages, for example.

    There are solutions to this 50 year problem. Here is one of those solutions. Small groups of programmers, pressured to produce great results in short period of time, re-invent the solutions. There are underlying errors of thought that are the primary factors behind the problem and its incredible persistence. How else can you understand the near-universal resistance to winning methods– proven in practice! — to be ignored?

     

  • Software programming language evolution: Functional Languages

    Once computers were invented and started being used, people discovered that writing programs for them was brutally hard. It didn’t take long before some smart folks figured out ways to make it easier, taking two giant steps forward in ease and productivity in quick succession.

    After that wonderful start, people kept on inventing new languages, but somehow never made things better – in fact the new languages often made programming harder and less productive, a situation that the experts and exalted Professors studiously ignored. They never even bothered to spell out what makes one language better than another, which you'd think would be at the heart of promoting a new language.

    I this post I’ll discuss a major category of new languages that keep getting created, receive passionate kudos, are rarely used but refuse to die: functional languages. In a subsequent post I'll describe the truly good idea that lurks inside the madness of the functional language world.

    The core idea: time and timeless, music and math

    I learned math as a kid. In high school I took the most accelerated and advanced courses available, finishing with a course in calculus, which I aced while mostly sitting in the back of the classroom teaching myself differential equations from my father’s grad school text book. I liked it. At the same time I got into music of multiple kinds but with a strong preference for classical. The strong connection between music and math has often been noted.

    During my junior year of high school I took a course in computer programming, with FORTRAN as the language. Our “teacher,” scrambling to keep a chapter ahead of us in the text book, had arranged machine time for the class on Saturdays at a computer at a nearby rocket firm, Reaction Motors. After lots more programming, by my first year of college, it was clear that math took a distinct second place to both music and software in my heart.

    In this post I explain where software comes from and its relation to music. The key concept is that, while you can read a page of music or software or math, math usually exists in a world “outside” of time – at most, time is a dimension like space; think for example of a proof in geometry. Music and software, by contrast, can only be understood as flowing through time. When you look at a page of music or software you naturally start at the beginning and hear/see it flow through time in your mind, just like reading a page of text.

    What this means is that there is a fundamental dis-connect between math and computing. I explore this disconnect here. The math types are always trying to turn software into math, for example plodding for decades on a fruitless pursuit for ways to “prove” the correctness of programs.

    This is the best way to understand both the failure and the refusal to admit defeat of the math types to develop a practical math-like way to write software, an effort and category of languages that is usually called “functional.”

    The history of functional languages

    Wikipedia's introduction to functional languages is accurate:

    Lambda calculus

    Turing had nothing to do with making software programming practical. That amazing job fell to others, who created the two giant advances in software language development. But since the math types were hanging around computers in the early days, particularly because doing math calculation was one of the earliest tasks to which computers were applied, finding a way to make programming more math-like, essentially eliminating the factor of time from software was a top priority.

    I had already done serious programming, including working on a giant FORTRAN project to optimize oil refinery operation, when a friend introduced me to functional programming for the first time. The language was APL. It came a bit closer to being able to handle practical math than many of the other non-functional "functional languages" because of its focus on the matrix, but it was as easy to read and understand as advanced math. I could and can imagine that it would be appropriate for representing certain math transformations in a compact way, but why anyone who wasn't being tortured would agree to use it for general programming remains beyond me — assuming of course that what you want is to … radical idea here, be warned! … get stuff done. As opposed to demonstrate and revel in your ideological and mathematical purity and exalted status in the pantheon of Computer Science greatness.

    None of this is any concern to the cultists, who march on inventing an endless stream of "new" functional languages. Among the names you might see if you look into this are  OCaml, Scheme, Haskell, Clojure, Scala. There are even a few that are object-oriented on top of being functional, like Erlang. Articles continue to pop their heads up out of the underground insisting that functional languages really are making inroads in commercial applications, better for getting a job, more effective to use to create your ground-breaking start-up, etc. I predict no end to the flow.

    The fact that functional languages are mostly of interest to a cult of fanatical core believers minimizes their widespread, on-going pernicious impact on software development, fueled by the near-universal math orientation of academic and corporate Computer Science. Functional concepts keep getting introduced by language fanatics into otherwise-sane normal languages.

    Conclusion

    Computer software is rarely driven by purely practical considerations much less objective knowledge and definition of what constitutes "good" in a language. But when an approach like functional programming is pushed that leads to results that are often 10X worse on multiple dimensions than "normal" programming, the problem is so large that people who aren't already committed members of the cult notice the difference and refuse to put up with the nonsense.

    In spite of all this, there is an amazing valuable insight at the core of the functional language movement that leads to highly productive, real-world-valuable results when embodied in the right way. I will spell this insight out and illustrate its use in a subsequent post. The heart of the insight is taking a declarative approach instead of an imperative one when and to the extent that is appropriate.

     

  • Micro-Services: the Forgotten History of Failures

    Micro-services are one of today's leading software fashions; see this for more on software fashions in general. I have treated in depth the false claims of micro-services to make applications "scalable," and separately treated in depth the false claim that micro-services enhance programmer productivity.

    Where the heck did such a crappy idea that has taken such a large mind-share in the software and management community come from?

    I don't know. Who knows why skirts are short or long this year? Or whether shirts are tucked or not tucked? What I do know is that bogus fashion trends infect the software industry, some of them delivering decades of perniciousness. Looking at the spectrum of software fashions, I've noticed similarities among some of them. Some fashions well up, spread and peter out, only to recur again later with minor changes and a new name. The new name appears to be important, so that everyone is fooled into thinking the hot fashion is truly "new," not tarnished by the abject failures of its prior incarnations. I describe a clear example of an old idea with new name here.

    The History of Micro-services

    Why do we have micro-services? How did this bad idea get to be so popular? Was it someone's recent "great idea?" There are probably people who think it's their idea.

    Do micro-services have a history? Of course they do! Nearly every major practice in software has a history of some kind. The number of software practices that are newly invented is truly tiny and going down rapidly. So why do we hear about software advances and the "new thing" so often? The whole field of software ignores its own history in truly radical ways. It even ignores what other software people in other fields and domains are doing.

    Nonetheless, anyone who knows a broad range of software practices over a period of decades can easily recognize similarities and patterns — patterns that surely constitute "history" whether or not the actors recognize the precedents of what they do.

    What are Services?

    Let's step back and understand what "micro-services" are. By the name, it's obvious that a micro-service is an itty-bitty "service." So what's a service? Once you peel back all the layers, a service is a plain old subroutine call with a bunch of fancy, time-consuming stuff stuck in between the code that calls and the code that is called. See this for my explanation of the fundamental concept of the subroutine.

    Subroutines (or functions or methods or whatever) are an essential aspect of programming. Every programming language has its syntax, statements and other things that people get all wrapped up in. Every programming language also has an associated library — as in function/subroutine library. If you look at a manual for a language and one for the associated library, the library is nearly always larger. The richness of a library is a key factor in the utility of a language. In modern terms, an associated framework serves essentially the same purpose; an example is the RAILS framework for Ruby, without which Ruby would just be one line on the ever-growing list of mostly-forgotten languages.

    When you're writing a program, knowing and making great use of the associated subroutine library is essential. But what if you see that you're doing the same kind of thing over and over in your program? It makes sense to create a subroutine to perform that common function as part of the overall application you're building. As time goes on, most well-structured applications include a large portion of nested subroutines.

    What if there's a program that can only run on another computer and you want to call it as a subroutine — give it some data to work on and get the results back? This problem was addressed and solved in many variations decades ago. It's most often called an RPC — Remote Procedure Call. To make a call to a distant subroutine you implement a version of it locally that looks and acts the same, but does some networking to send the fact of the call and the parameters to a cooperating routine on the distant computer. That routine gets the network call, retrieves the data, and makes a local call to the subroutine. When the return happens, the data is sent back by the same mechanism. It's just like writing a memo and putting in your out-box. The clerk who picks up the outgoing memos delivers the ones that are local, and puts the ones that aren't into an envelope, addresses it and puts the memo with return instructions into the mail. And so on. The calling code and the called code act like they normally do and the RPC makes it happen.

    Along comes the internet. People notice that http is supported all over the place. Why not leverage it to implement the mechanics of the RPC? Voila! Now we have SOAP — Simple Object Access Protocol, which uses http and XML to send and return the data. A subroutine with a SOAP interface was called a "service." This all got standardized roughly 20 years ago. Then along came a simpler version to use that had even more overhead called Restful, which is still in widespread use.

    If you need to call a subroutine that can't or shouldn't run on your computer but on some other computer, having some form of RPC is extremely useful. The time penalty can easily amount to a factor of thousands or even millions compared to a normal local subroutine call, but you'll gladly pay the price if there's no other way.

    Now let's recall what all this amounts to. A "service" is a subroutine that has a HUGE amount of overhead. There is no reason to ever take on the overhead unless, in the vast majority of cases, the service you want to call is on a different computer than the one doing the calling, a computer that can only be accessed by some kind of networking.

    Prior Incarnations of Service Insanity

    There have been a couple waves of service fashion mania. One of the waves took place roughly twenty years ago during the internet bubble. People were very concerned to be able to support growth in the use of their applications by unprecedented factors. Expert opinion rapidly agreed that building a "distributed application" was the way to accomplish this. The idea was that the load on the computer would greatly exceed the capacity of even the largest, most capable computer to handle it. To anticipate this, the application should be architected to be able to run on multiple computers that could be changed at will. Instead of a "central" application, the application would be "distributed." This was further elaborated by replacing the simple RPC mechanism with queues called "service buses," as in a bus to route large numbers of service calls from one place to another. The bus itself had to be robust so that messages weren't dropped, so it evolved into an "enterprise service bus," something which exists today. A huge, complex mechanism for accomplishing this became part of java, J2EE (Java 2 Enterprise Edition). See this for the complex, reincarnating history of the transaction monitor.

    It turned out that building a "distributed application" was vastly more expensive and time-consuming than just building a plain old application, the kind that today is sneeringly dismissed as being "monolithic." The computational and elapsed time running cost of such an application was also huge. Hardly any applications had loads so large that they needed to be distributed. Even worse, the few applications that ended up with huge loads that required many computers found vastly simpler, more effective ways to get the job done. It's worth noting that articles with titles like "Why were we all fooled into thinking building distributed applications was a good idea" never appeared. The whole subject simply faded away. Here's a description from another angle of the distributed computing mania.

    Another incarnation of this nutty idea took place largely inside large enterprises. Many of these places had highly diverse collections of applications running the business, accumulated by acquisition, multiple departments building applications for themselves, etc. Instead of doing the sensible thing of reducing the total number of applications by evolving the best versions to be able to handle more diverse tasks, the idea of a SOA, service-oriented architecture, became the focus. All these applications could be turned into services! Instead of building new applications, people would now build services. Tools were built to manage all these new services. There were directories. There was tracking and control. All sorts of new demands came latching onto the IT budget.

    SOA never got to the fever pitch of distributed applications, but it was big. It went the same way — fading as it turned out to be a lot of time and trouble with little benefit.

    Now we come to the present. Remember that the original motivation of the RPC in any form is that a subroutine you want to call is ONLY running on some other computer. An RPC, whatever the form or cost, is better than nothing. Sensible then and sensible now. Then came ways of building programs so that their parts COULD BE run on different computers, if the computational requirements became huge. This also turned out to be rarely needed, and when it was needed, there were always better, faster, cheaper ways than a service approach. Now, with micro-services, we have reincarnated these proven-to-be-bad ideas, claiming that not only will micro-services effortlessly yield that wonderful virtue "scalability," but it will even make programmers more productive. Wrong and wrong.

    Conclusion

    I like the phrase "oldie but goodie." Sometimes it's applicable. In computing when a hot new tech trend comes along that "everyone" decides is the smart way to go, it rarely is. It most often is an "oldie but baddie" with a new name and new image. You would think that software people were so facts-oriented that they wouldn't be subject to this kind of emotionally-driven, be-part-of-the-group, get-with-the-program-man kind of thinking. But most people want to belong and want increased status. If believing in micro-services is the ticket to membership in the ranks of the software elite, a shocking (to me) number of people are all in.

  • Software Evolution: Functionality on a New Platform

    When a new “platform” emerges (UNIX, Windows, Web, Apps), if you look at any application area and see how it evolved on prior platforms, the application’s functionality will emerge on the new platform in roughly the same order, though often on a compressed timescale. The functionality that is relevant depends on the particular application area. This concept applies both to system and application software.

    The pattern is: functionality emerges on a new platform in roughly the same order as it emerged on earlier platforms. The timescale of the emergence may be compressed; the important aspect of the pattern isn’t the timing but the order. The pattern means that functional steps are rarely skipped – what was next last time is also next this time. The pattern also means that when someone tries to introduce functionality too soon, before the functionality that preceded it on prior platforms is generally available, the market will not accept it.

    While this pattern takes a good deal of knowledge and judgment to notice and apply, I have consistently been impressed by its predictive power. By following the pattern, you can be pretty confident that you’re building proven functionality, and that you’re following a pattern of success.

    I have noticed a couple of danger points here. When the company is too aware of the pattern, it is easy for them to “get ahead of themselves” and more importantly ahead of the market, by solving problems that certainly will become important, but problems that the market doesn’t know it has yet. The “same order” part of the pattern is important; building ahead of the market appears to win few business benefits.

    On the other hand, without knowledge of the pattern, it is easy to make up all sorts of things you think people might want, and build them, only to find out later that you wasted time and money, because the functionality you built is never part of the accepted function set.

    Really great companies who have lots of creative people, the ability to execute, and listen closely to market reactions to what they’re doing, don’t “need” to know about this pattern. However, for the rest of us mortals down here, having a “cheat sheet” to what important features the market will be demanding next can prove awfully helpful.

    Basic Example: operating systems

    IBM famously created an operating system for their mainframe line of computers, OS/360. It had the capability of running multiple jobs at once. Its multi-tasking abilities grew and became more sophisticated through the 1970’s.

    450px-360image001

    Eventually a transaction monitor, CICS, was written and became a de facto part of the operating system for applications with lots of interactive users. As the operating system was used, it became evident that various access methods for storage and communications needed to be separate from the core, and so clear interfaces were created, and the notion of completely self-contained access methods (for example, a file system) as replaceable units was supported. A strong security system was not part of the early versions of the operating system, and the need for one became critical, and so strong external modules were written and support for security was added to the core. While there was a “main” operating system, alternative operating systems were written for various purposes, and a virtual operating system was written to actually run on the “bare metal.” With VM (the virtual OS), you could devote most of a machine’s resources to the production users, while letting some of the users running on a completely different operating system.

    While all this was taking place, people were studying and experimenting in university environments, deciding just what an operating system was and was not, what the best ways to build one were, and so on.

    Before very long, mini-computers were invented; these were basically mainframes on the cheap, with all sorts of features and functions missing – but they were cheap. And, since each had a unique instruction set, each minicomputer needed an operating system. Programmers were hired, and those programmers, of course, ignored the mainframe operating systems, and built simple, cheap OS’s to go along with the cheap hardware. Surprise, surprise, those cheap OS’s resembled nothing as much as – the first generation of mainframe operating systems! But people quickly discovered the limitations, just as they had before, and set about making the same set of enhancements that the previous generation of pioneering programmers had made. Within ten years, they had re-invented many of the important mainframe OS concepts, and were on the way to building the rest.

    With all this knowledge of operating systems floating around and pretty easily available, what do you suppose happened when people took the early micro-processor chips and made them into micro-computers? Naturally, they understood the state of the art of operating systems theory and practice and adapted an existing OS (which were starting to be built in high level languages) or built one that took all this knowledge and applied it with appropriate adjustments to the new environment, right? Bzzzzt! Of course not!

    What the kids who were faced with the task did was start from scratch, not only in terms of code, but also in terms of knowledge. They didn’t stand on the shoulders of giants; they didn’t learn from the experiences of the many that preceded them; they built OS’s as though they were the first programmers who ever tried to do such a thing. And the result was pretty much like early mainframe (even pre-360!) operating systems. There was no serious memory protection or address mapping; there was no real concept of multiple users, multiple levels and types of users, or any real security; no multi-tasking; the access methods were hard-wired in, and so on. The limitations and problems emerged pretty quickly, and so did add-on rubber band and baling wire patches, just like in earlier days.

    It’s a good thing that IBM came along at this point, and brought commercial order and education to the emerging microcomputer market. When they came out with the IBM PC, they not only legitimized the market, they had deep history with mainframes and minicomputers. They employed true experts who knew operating systems inside and out. They had a research division, where there were people who could tell you what the operating systems of the future would look like. So it makes sense they would get those experts together, and they would create a small but efficient micro-tasking kernel, common interfaces for installable access methods, and many other appropriate variations on all the modern operating systems concepts. The last thing such a smart, educated and astute major company like IBM would do was make an exclusive deal with a small company that had never built an operating system, who had just bought the rights on the cheap to a half-baked excuse for a primitive first-generation OS, and make that the IBM-blessed … Wait! … that’s what they did do! Arrrgggghhh!

    Explanation

    One might well ask, how can a pattern like this continue to have predictive power? Why wouldn’t the people who develop on a new platform simply take a little time to examine the relevant applications on the older platforms, and leap to the state of the art? Why wouldn’t customers demand it?

    It is hard to know for sure, but I think there are a couple main factors at work, and there is evidence for the relevance of each of the factors.

    The first factor is the developers. It is well known that most developers learn a platform and then stick with the platform they’ve learned for an extended period of time, basically as long as they can. The reason is simple: they are experts on the platform they already know, and therefore have prestige and make more money than they would as novices on a platform they’re just learning. I speculate that this is one of the many contributing factors to the rapid migration of ambitious programmers into management, where they can advance without being tied to a platform, at least as much. So who learns the new platforms? With few exceptions, new people. If you’re just entering the industry, you are counseled to learn the hot new languages; you tend to be sensitive to where the demands and rising salaries are. Still, you expect and are paid entry-level wages, along with most other people (except managers). Why should the experienced programmers jump to the new platform? They would have to compete with hard-working young people, their knowledge of the older platform will be considered a liability, and on top of everything else, they’d have to take a pay cut.

    The result is that no one working on the new platform has an in-depth, working knowledge of the applications on the older platform, and at least in part because of this, everyone considers knowledge and use of the platform to be vastly more important than knowledge of an old application on an obsolete platform. So they ignore it! As a result, they dive in and attempt to automate the application area “from scratch.” Their attempts are usually quite close to every first generation program for that application on past platforms, because it turns out that the determining factor isn’t the platform, it’s the business problem. They proceed to re-discover, step by step, the reasons why the first generation was inadequate and had to be supplanted by a second generation, etc.

    The second factor is the buyers. When a new platform emerges, most buyers simply ignore it. Why pay attention? It’s a toy, there are no good applications, etc. The few buyers who do pay attention tend to be crazy, early adopter types who just love the experience of trying new things. Like the programmers, they also tend to care about the platform more than the application – otherwise, they wouldn’t even consider buying what is typically a seriously immature application on the new platform. But they can only buy what’s being sold, and so they choose among the inadequate applications. Because they don’t care about applications as much as platforms, they don’t even ask for features they know could only be present in the mature applications for older platforms – they press the applications vendors for the “next” obvious cool feature, in the narrow universe of the new platform.

    The reason why application evolution repeats itself on the new platform, then, is that nearly everyone involved, builder and buyer, is ignorant of the past and wears what amounts to blinders. It’s as though they are building and buying the application for the first time. Therefore, they respond most strongly to the same business pressures that people involved in computer automation tend to see first, and then next, and so on. It’s as though there’s a “natural” most climb-able path up a mountain, and successive waves of climbers approach the climb from the foot of the mountain, completely ignorant of the experiences of those who came before them, but confronted with the same options they tend to make the same choices, and so everyone takes roughly the same route up the mountain.

    Why don’t smart groups who know all this leap-frog their way to the most advanced application functionality? I have seen this happen, and it’s the buyers who tend to rain on this kind of parade. The buyers tend to be familiar with the range of applications in a category, and those applications tend to address a highly overlapping set of problems in ways that vary only slightly. The “far-seeing” builder then comes along with all sorts of solutions to problems that the buyers don’t even know they have! The buyers look around in confusion – why isn’t anyone else talking about this? I kind of understand what you’re talking about, but I feel silly thinking that something’s important when no one else does. I think I’ll wait. And they do. So getting too far ahead of the buyers is just as much of a problem as being too far behind the competition. The result: the application evolution repeats itself on the new platform, in roughly the same order each time, and no “cheating” or “running ahead” is allowed.

    This all sounds incredibly common-sense when I read it written down, but I have to admit that this particular piece of common sense is not only uncommon, it took me personally decades and multiple failures to finally get this simple thought into my thick skull. The key thought is this one: you may think you know a person has a problem; the problem might be a severe one, and cost the person a great deal of money and trouble; you may even be entirely right in this judgment. However, if the person in question does not think he has a problem, why should he pay for a solution – in fact, why would he go to the trouble of implementing a solution even if it were free?  Even worse, why would he even waste time talking with you, once he got the idea that you thought he had a problem he didn’t think he had? Why would he listen to a chimney sweeper’s pitch if he lives in a house without chimneys? The air quality in Los Angeles in 1970 was terrible. But there was no market for catalytic converters on automobile exhaust systems at that time. The problem that converters solve existed, and was getting worse. It was obvious for anyone to see. It was even talked about. But in peoples’ minds at the time, having a car that contributed to air pollution was not a problem most people accepted they had (even though we know that, objectively speaking, they did have it).

  • Elizebeth Smith Friedman: The Cancelled Heroine of Cryptography

    The unheralded Elizebeth Smith Friedman is a textbook example of the vast gulf that too often separates achievement in a field from getting credit for the achievement.

    Eliz pic

    She was a true pioneer of cryptography and code-breaking, leading multiple efforts against the international criminal mob and the Axis in World War II. Unlike most people called “leaders,” she was actually the best at what she did, personally cracking “uncrackable” codes and personally pioneering new methods. She was a leader in the true sense: the manager/boss of the long distance runners AND the runner far in front of everyone else who gets there first AND helps all her fellow runners speed up.

    Technical History Issues

    Making an advance in technology is hard. Not many people try to do it, and a tiny fraction of those who try seem to succeed. Many of those apparent successes burn out – they weren’t advances after all. Sometimes a true advance, for various reasons, is never adopted. When it is adopted, there is often a race to claim credit for the advance. The race isn’t so much a race as it is a no-holds-barred war. To win the war, you usually need the support of loads of people who have no idea what the advance is about. These ignorant people create history, along with its winners and ignored achievers.

    Is this cynical? Yes. Is it an accurate description of what happens? In all too many cases, sadly yes. Here are examples of from the war for credit for inventing the computer.

    In most cases, technical invention isn’t like a giant comet streaking to earth and creating a big boom. It’s more like a sequence of parallel, overlapping efforts to solve a problem or make something better. Often an advance is made by more than one person or group without involvement with the other. What in retrospect is described as the big advance is usually a step forward, one of many, building on earlier work. Sometimes the advance isn’t an advance so much as a commercialization. Matt Ridley describes this with many examples in his mostly excellent book on Innovation. Elizebeth stands out as being a true innovator on multiple dimensions.

    Elizebeth Smith Friedman

    Getting to the truth about inventors and technology innovation is a problem in general. In the case of Ms. Friedman, the problem was made worse by the credit-taking actions of government leaders. The truth has only emerged recently with the release of previously concealed documents and the ending of secrecy periods.

    Here are some highlights of her career:

    In the 1930s, Elizebeth Smith Friedman became America’s and indeed the world’s best-known codebreaker. She inflicted severe damage on the interests of organized crime and at times needed to be protected by bodyguards. The evidence she gave in criminal trials describing how she cracked encrypted messages passing between mobsters made her a newspaper sensation.

    Later, during World War 2, she broke coded messages sent on Germany’s Enigma machines. These messages revealed a plot by the Argentinian government to help Germany replace South American governments with Nazis, giving Germany bases from which to attack America. Her discoveries allowed the western allies to thwart the Argentinian and German plans.

    Elizebeth Smith Friedman’s wartime codebreaking work was so secret that she was forbidden to mention it in public. She died many years before government archives were brought to light showing what she had done. During and after World War 2, J. Edgar Hoover and the FBI took the credit for work Elizebeth and her U.S. Coast Guard team had carried out.

    Her whole story is fascinating. Among other things she is a wonderful example of the power of bureaucracies (education, government and corporate) to control and often suppress outstanding talent, and how sometimes, when the bureaucracy is desperate for results, it will break its own rules to achieve a goal – and then claim credit. It is particularly striking in Elizebeth’s case because she was NOT trained in math or STEM of any kind; her fascination was with literature and philosophy.

    There is a book about her life which is includes previously classified and/or ignored documents about her career:

    Woman smash cover

    If you’re at all interested in people like Alan Turing and Grace Hopper and/or computing history, it’s worth reading. She was completely “unqualified” to do what she did – and she became the best at it, for example cracking the German Enigma machine without the huge staff and machines at Bletchley Park that have become famous.

    Likewise, I had never heard of her husband William Friedman, who was also accomplished as a code-breaker both by himself and working with Elizebeth. Here are a couple links that give some highlights, though I still recommend the book.

  • The giant advances in software programming languages

    There have been two gigantic advances in programming languages, each contributing huge advances in programmer productivity. Since then, thousands of times more attention has been given to essentially trivial changes than has been given to the two giant advances. I described these 50 years of non-advances here. In this post I’ll go back in history to describe the two giant advances that were made in the early days of computing, back in the 50’s, and I’ll tack on the one important corrective that’s been made since then.

    First, the pre-history of computer languages

    The famous early computer was the ENIAC.

    https://en.wikipedia.org/wiki/ENIAC

    It cost the equivalent about of $7 million to build and took about 3 years. It was big, occupying about 300 sq ft. It was roughly 1,000 times faster than electro-mechanical machines, so you can see why people were excited about it. Its purpose was to perform complex calculations on scientific/engineering data.

    Glen_Beck_and_Betty_Snyder_program_the_ENIAC_in_building_328_at_the_Ballistic_Research_Laboratory

    While the excitement was justified, the method of programming was gruesome.

    “ENIAC was just a large collection of arithmetic machines, which originally had programs set up into the machine[33] by a combination of plugboard wiring and three portable function tables (containing 1,200 ten-way switches each).[34] The task of taking a problem and mapping it onto the machine was complex, and usually took weeks. Due to the complexity of mapping programs onto the machine, programs were only changed after huge numbers of tests of the current program.[35] After the program was figured out on paper, the process of getting the program into ENIAC by manipulating its switches and cables could take days. This was followed by a period of verification and debugging, aided by the ability to execute the program step by step”

    Engineers (all men) would figure out what they needed the machine to calculate. It then fell to the “programmers,” all women, to perform the excruciatingly detailed work with the knobs and switches.

    375px-Two_women_operating_ENIAC_(full_resolution)

    From this it is extremely clear that while the machine was an astounding achievement in speed, the speed of programming the calculation steps was the huge bottleneck. How can we make this faster?

    Answer: instead of physical plugs and switches, let's invent a language for the machine — a "machine language!" That's what happened with the invention of the stored program computer, which took place in 1948. Instead of moving plugs and flipping switches, programming was done by writing it out in machine language on paper, loading it into the computer via punched paper cards, and then having the computer execute it. Figuring out the detailed steps was still hugely difficult, but at least getting it into the computer was faster.

    From machine language to assembler

    Every machine has one and only one “machine language.” This is a stream of binary data that the machine’s processor “understands” as instructions. Each instruction causes the machine to perform a single action on a single piece of data.

    I learned to read and write machine language for a couple different machines. It’s not easy. From early on, machine language was usually shown as hexadecimal instead of raw binary, so that instead of looking at “0100101000101111” you see the hexadecimal equivalent: 4C2F.  A big plus but you’re looking at a hexadecimal pile of data. A nice editor will at least break it up into lines of code, but you still have to read the hex for the instruction, registers and storage locations

    Moving from machine language to assembler language was HUGE. Night and day in terms of readability and programmer productivity. Assembler language is a body of text that closely corresponds to the machine language. Each line of text typically corresponds to exactly one machine language instruction. Assembler language transformed programming. If you understand any machine language, you can easily write and read the corresponding assembly language. For example, here's the binary of a simple instruction:

    1

    Here's the equivalent in hexadecimal:

    2

    And here it is in assembler language, including a comment:

    3

    Just as important as making instructions readable was using text labels for locations of data and addresses of other instructions. In machine language, an address is expressed as a “real” address, for example as a hexadecimal number.  If you inserted a command after a jump command and before its destination, you would have to change the address in the jump. You can see that with lots of  jumps this would quickly become a nightmare. By labeling program lines and letting the assembler generate the “real” addresses, the problem disappears.

    From assembler to high-level languages

    Assembler made writing code in the language of the machine incredibly easier. It was a huge advance. But people quickly noticed two big problems. The solution to both problems was the second giant advance in software programming, high-level languages.

    The first problem was that writing in assembler language, while worlds easier than machine language, still required lots of busy-work. For example, adding B and C together, dividing by 2 and storing the result in A should be pretty easy, right? In nearly any high-level language it looks something like this:

                    A = (B+C)/2

    Putting aside messy details like data type, this would be the following in pseudo-assembler language:

                    Load B into Register 1

                    Load C into Register 2

                    Add Register 2 to Register 1

                    Load 2 into Register 2

                    Divide Register 1 by Register 2

                    Store Register 1 into A

    Yes, you can quickly become proficient in reading and writing assembler, but wouldn’t the 1 line version be easier to write and understand than the 6 line version?

    Expressions were the core advance of high level languages. Expressions turned multi-line statement blocks that were tedious and error-prone into simple, common-sense things. Even better, the use of expressions didn’t just help calculations – it helped many things. While assembler language has the equivalent of conditional branches, many such conditionals also involve expressions, for example, the following easy-to-read IF statement with an expression

                    If ((B+C)/2 > D) THEN … ELSE …

    Would turn into many lines of assembler – tedious to write, taking effort to read.

    The second problem emerged as more kinds of computers were produced, each with its own machine and assembler language. A guy might spend months writing something in assembler and then want to share with a friend at another place using a different type of computer. But the friend’s computer was different – it had a different assembler/machine language! The whole program would have to be re-written. Total bummer!

    Wouldn’t it be nice to write a program in some language that was like the 1 line version above, and that could be translated to run on any machine??

    Easier to write by a good 5X, easy to read and can run on any machine, present or future. Hmmmm… There must be something wrong, this sounds too good to be true.

    The concept of high level languages was good and wasn’t too good to be true. Everyone agreed, and it wasn’t long before FORTRAN and similar compute-oriented languages arose. Even performance-obsessed data nerds wrote programs in FORTRAN rather than assembler because FORTRAN compiled into machine language was just as fast, and sometimes even faster because of optimizations those clever compiler guys started inventing! And the programs could run anywhere that had a FORTRAN compiler!

    The accounting and business people looked at what was happening over in heavy-compute land and grew more and more jealous. They tried FORTRAN but it just wasn’t suitable for handling records and transactions, not to mention having nice support for financial data. So business processing languages got invented, and COBOL emerged as the best of the bunch.

    If you look at a compute statement in FORTRAN and COBOL, they’re nearly identical. But in the early days, FORTRAN had no support for the money data type and no good way to representing blocks of financial data that are core to anything involving accounting.

    Once they got going, people just kept on inventing new languages for all sorts of reasons. The obvious deficiencies of FORTRAN and COBOL were fixed. Languages could handle intense calculations and business data processing about as well. But back around 1970, 50 years ago, there remained a gap. That gap was that an important category of programs could not be written in any of the high-level languages. Essential programs like operating systems, device drivers, compilers and other “systems” software continued to be written in the assembler language of the machine for which it was intended. There were attempts to fill this gap, but they failed. Then a couple super-nerds at Bell Labs, building on an earlier good shot at solving the problem, invented the language C.

    Lang 2
    They rewrote the UNIX kernel in C, and the rest is history – C became the high-level language of choice for writing systems software and still holds that place.

    Conclusion

    Two giant advances were made in programming languages. Each of these happened early in computing history, in the 1950’s. The first giant advance was the biggest: with assembler language, machine language was reasonable to write, read and change. The second giant advance was the category of high level languages. Two minor variants on the concept, COBOL and FORTRAN, established the value of having a language that enabled expressions and could be compiled to run efficiently on any machine. In spite of the on-going tumult of language invention that has taken place since, the fact that huge bodies of working code written in those two languages continue to perform important jobs makes it clear that they embodied the giant advance that was high level languages. The only substantial omission in those languages – the ability to directly reference computer memory – was filled by the invention of the language C.

    Most of what’s happened since in programming languages is little but sound and fury. For a review of those non-advances see this.

    I do appreciate some of the subtle changes that have been made in language design. Some modern languages really are better – but mostly not because of the details of the language! I’ll treat this issue in a future post. Meanwhile, it’s important to understand history and appreciate the giant steps forward that high level languages as a category represent.

  • Software Programming Languages: 50 years of Progress

    I was inspired to examine 50 years of progress in computer languages by an article in the Harvard alumni magazine written by one of the leading figures in Harvard computer activity, Harry Lewis. Lewis graduated from Harvard College the year I entered, and he stayed on for a graduate degree while I was a student there. He says:

    Thirty veterans of Harvard’s Aiken Computation Lab reunited on January 19, 2020, some 50 years after each of us had a role in creating today’s networked, information-rich, artificially intelligent world. Rip van Winkles who had never fallen asleep, we gathered to make sense of what had evolved from our experience as undergraduates and Ph.D. candidates during the decade 1966-1975. One thing was clear: we hadn’t understood how the work we were doing would change the world.

    I interacted with many of the people and projects he describes in the period he focuses on. For this post I decided to focus just on one of his major topics, programming languages.

    50 years of progress

    You might think a book would be required to adequately describe the advances that have been made in programming languages in the last 50 years. In fact, many experts seem to think so. Programming languages continue to advance and improve, even with everything that’s been achieved in the last 50 years – and more!

    There is no doubt that many programming languages have been introduced in decades past. The pace of new language introduction does not seem to have slowed. But has progress been made? Yes – if you count dead-ends, bad ideas, going in circles and trivial variations as progress.

    While describing the true progress in programming languages is easy, explaining why all the widely-touted “advances” aren’t in fact progress would indeed take a book. Which someone other than me is welcome to write. It would be boring! What could you expect from a gruesomely detailed exposition that amounts to Idea 1 is stupid because …; Idea 2 isn’t any better because …; idea 3 just another attempt with slightly varied syntax to go after the same fruitless goal as Idea 4, decades before …

    I gave a talk that included this subject to a large group of programmers in India 25 years ago. Here is the slide illustrating the mighty progress of languages as illustrated by what COBOL calls a Compute statement:

    Lang 1

    I hope that gives you a basic idea of the massive advances that had been achieved by the mid 1990’s.

    The same progress has continued unabated until the present. Google is promoting the language Go, often called Golang. Go must be one terrific language if the geniuses at Google are behind it, right? Here’s that same simple assignment statement I illustrated decades ago, updated to show how Google has made it even better:

    A:=B*C+D

    This illustrates the exquisite judgment expressed in new language design: The assignment statement has the colon before the equals sign, like PL/1, but leaves off the ending semicolon. Genius!

    Uncommon common sense about programming languages

    The inventors and promoters of new programming languages nearly always talk about how the wonderful new language is "better" than all other languages. Frequently mentioned are that the new language helps programmers be more productive and commit fewer errors.

    In my experience and subjective judgment, all such claims are false and simply not proven out in practice. More important is the fact that the inventors/promoters of new language have no real arguments beyond their own enthusiasm to back their claims: there is NO evidence-backed science to support ANY of the claims that a given language is superior on some measure than others. There isn't even any substantial agreement on the appropriate measure of goodness, much less an evidence-based hierarchy of languages against the measure of goodness!

    That should be the end of it. Once you realize how devoid of evidence, experimentation and science the whole endeavor is, it becomes clear that anyone promoting the virtues of a new language is either an ignorant buffoon or self-promoting cult leader. Or some of each.

    Following are some observations to help put this into context.

    I do understand how the enthusiasm for a new language can catch hold. Being a programmer requires a powerful set of blinders and the ability to focus on an incredibly narrow field for long periods of time. It's essential to being a good programmer! But you have to step back and look at your enthusiasm objectively. Above all you have to have objective measures to compare your wonderful new language to the bad old ones. This is almost never done in software, though it's commonplace in other fields. Would Marie Curie have been a better scientist had she used German instead of French as her primary language at work? See this for more.

    There is a hierarchy of skills in software. Knowing a programming language is an important part of the hierarchy but it's near the bottom of the stack! Think about learning a new human language as an adult, ignoring for the moment the huge advantage you have being proficient in a language. You start with basic vocabulary and grammar. How long and hard is it to get the accent a native speaker has? Now think about vocabulary, the basic subset you've learned vs. the huge set of the native speaker. Think about using the basics of the language to get things done compared to using the language as an excellent writer or speaker. Now think about idiomatic expressions, of which any native speaker has a huge set — without them, you're not fluent. Finally, think about a profession like law or medicine. Those proficient are using the base language, but with a vocabulary,, knowledge and sophistication that take years to learn. These things are like the subroutine library that is a key part of any language though not strictly speaking part of the language itself, and the professional specialty is like networking or database. Yes, the core language is a necessary foundation, but most of the real skill is in the rest. See this for more depth on this important subject.

    It turns out that the "goodness" of a software effort isn't really dependent on the language used! When you dig into what really matters, it turns out that factors outside the strict confines of language design are overwhelming important, though details that new-language enthusiasts tend to ignore can play a role. See this for details.

    One more key factor in the explosion of languages that are touted as advances but are really just a variation on the same old thing is the core human behavior of creating a language among people who work together on the same things for long periods of time. But no one has found that any of these languages are superior in some real way than others — they're just the languages that happen to have evolved, each with some unique vocabulary reflecting unique concerns, like the words that people who live in the arctic have for different kinds of snow and ice. This is a well-known behavior, and why shouldn't it also be seen when people "speak" computer languages together? See this for examples and details.

    When you lay out the spectrum of languages, you can see some patterns. The most obvious of the patterns I've observed is the extent to which data is defined as part of the language itself or outside the language proper. Data can be fully accessed and manipulated in all cases — the difference is the extent to which the data definitions are made part of the language itself, or as part of a core library attached to the language. Language mavens are passionate about each of the variations, and convinced that theirs is best! See this for details.

    What among other things enables large groups of people to hail the latest "advance" in computer languages without being laughed out of town is the shocking ignorance and lack of interest in computer and software history everyone involves exhibits. With the least bit of software history knowledge, no one would have the nerve to invent a new language, much less brag about it. See this for more.

    What does this mean?

    If you're a techie, you should get a grip, some perspective and knowledge of history and science. It will help you see, for example, that the fact that Marie Curie's native language was Polish and that she worked largely among French speakers was incidental to her great mind and achievements. It will help you see that Shakespeare wouldn't have been a much better playwright if only he had written in Latin (or whatever), or would have been much worse had he written in Norse (or whatever).

    If you're a manager, you should respect the enthusiasm of your employees but understand the fact that their enthusiasm is baseless. I saw a software re-write into a new language using the Neo4j database. Everyone was pumped. Disaster. I have seen a couple cases in which huge enthusiasm was behind a less radical choice. The results were no better in any way than using a more standard language — the engineers worked harder than they would have because of their enthusiasm, but were less productive because they were less skilled in the new environment. And of course, new hires were hampered with the same problem.

    Conclusion

    Computer Science is a profoundly unscientific field. The fashion-driven nature of it is exhibited starkly in the decades-long march of revolutionary new languages that keep getting invented — and having no impact on anything that matters.

    If you want to be the best possible programmer, it's incumbent on you to use a sensible, mainstream language and spend your effort producing excellent results — including bringing the whole notion of excellence to new levels. The language you use will not be the limiting factor — what's important is the way you use the language, just as it was for Marie Curie.

  • Computing in 1968: Gender, Cards, etc.

    I grew up in Denville, a little town in northern New Jersey. I had a friend whose father knew someone who worked at what was then called EMSI, Esso Mathematics and Systems Inc., housed in a big complex on Park Ave. in Florham Park, NJ. I don’t think I even was interviewed, but I ended up with a job there the summer after graduating from the local public high school in 1968!

    EMSI was a service company, one of over 100 companies in the Standard Oil of NJ group of companies. The purpose of the company was to apply advanced math and computer techniques to make the various Esso companies work better. The company had a large computer center on-site, a large room full of card keypunch machines, and extensive offices.

    There was a strict division of work by sex. There were secretaries, all women. The men, all dressed in white shirts and ties, were the math and software design people. Software design was done using graph paper and plastic sheets that had flowchart cut-outs in it, so you could make uniform branch triangles, etc.  Here’s a picture of what it looked like (not mine, I never had or used one): 111

    The programming was all done by women, who wrote lines of code onto coding paper, with lines and columns that matched up with Hollerith cards. This was supposed to be a mechanical affair, driven by the graphical designs. Of course it was more than that. When a sheet was ready, it would be sent for keypunching, all done by women using keypunch machines. I think we used the newer IBM 026 machines. The programmers, all women, would get the cards and sort them in a rigid box for storing and transporting to the machine room, where male machine operators would stack them into card readers for loading into the computer. Often there were errors, of course, so the machine would print the error listing on wide, 132 column paper using a mechanical printer. The paper had a string of square holes along each side that fit into gears in the printer. The printers were “line printers,” printing a single line of characters and numbers on the paper. Once the line was printed, the gears would advance the paper one line, and the next one would be printed. The pages were all joined end-to-end, fed to the printer from a big box sitting beneath it. When a print job was complete, the last page would be advanced out. An operator would tear the print jobs apart from each other and put them on a table for people to pick up, along with any related cards to the programmer. The printout could be a list of error messages from the compiler, maybe along with a printout of the program itself. You could ask for the assembler language version of the program to be printed along with the original source code in FORTRAN, which was important when debugging. If the program compiled properly, i.e., had no syntax errors, it would be run. It would either run to completion and print out the desired results, or it would crash, in which case a listing would be printed including a memory dump, which was a hexadecimal listing of the contents of the memory at and around the cause of the crash. I spent many totally absorbed hours poring over and deciphering memory dumps.

    The physical embodiment of a program was an ordered set of cards. You would carry the cards for a program around in a cardboard box, all lined up in a neat stack. The box was as wide and high as a card, and at least a foot long, long enough to carry 2,000 cards.  Every once in a while someone carrying a box of cards would stumble and drop the box, scattering the cards on the floor. A fun time would then ensue, figuring out the right order of the cards and putting them back. This is one of the reasons that the last handful of columns in many card decks contained the sequential number of the card in the program, or set of data, or whatever; without such numbers, putting the deck back together would take even longer. I dropped a deck once. It’s the kind of thing that, once you did it, you took great care never to do again, quite apart from the people around you in the hallway laughing at you.

    I don’t know how it happened, but I was the exception – a tall, skinny, male high school grad who was bright but entirely without qualifications. Everyone seemed amused by my existence. Perhaps that’s why I got away with things. It’s probably also why I kept getting asked to do things that cut through the otherwise strict divisions of labor. For example, I skipped the usual two-step of getting programs written, the design part by tie-wearing men using plastic guides to make neat-looking diagrams of program operation and flow, followed by women “translating” the design into actual lines of code. What a waste of time, I thought. Just work out what you wanted to do in your head, and then write the code. Before long I skipped the write-the-code-on-paper part, and just typed it into a vacant keypunch machine in the sea of lady keypunchers.

    Looking back on the experience, I realize how lucky I was, and how unusual the experience I had working for a large division of a giant company. I now know that there must have been projects with goals and timelines that had to be met. People were judged based on whether they hit the pre-determined goals. While I was stuck into that environment, I was there as an un-planned-for extra resource who fit into none of the pre-defined categories. People threw me bits of work to keep me busy, without ever telling me how and when it had to be done. Each task involved something I had to learn, often a lot I had to learn, so I just tore into it. When I was done, I presented it to the person who gave me the work, much like a dog who dashed off to fetch a ball someone had thrown, perhaps into a lake. I imagine I looked as eager and pleased with myself as a dog, ready to dash after another ball. I never focused on the learning; I just drove towards the ball, overcoming obstacles (mostly things I didn’t know, sometimes the strict hierarchies and divisions of labor) as quickly as I could along the way. It was a formative experience, and one that was at odds with the normal group working experience. I knew I was lucky to have the job, and my appreciation for how lucky I was has grown over the years.

    This is an edited excerpt from the draft of my Computer-related Memoir.

  • Luddites Smashed Looms then and Resist Automation today

    The few people who are familiar with the term "luddite" think that a luddite is an unfortunate but stupid person who fights against advancements that make things better, while clinging bitterly to their crappy, low-end jobs. "Luddites" in this common view, are uneducated, progress-preventing people who need to be moved to the side so that society can be improved.

    The reality is that, in most cases, luddites were highly skilled craftsmen performing difficult and challenging jobs. It's not that much different today. Luddites are often highly educated professionals and managers who are convinced they bring value to their complex jobs every day The reality is that there are luddites everywhere. Who wants to have their job disrupted? Who wants to be told that their expertise is no longer needed, and that a machine or software system can do a better job?

    Stepping back

    This notion of what software was REALLY about struck me quite early in my career. The thought was simple: the ultimate purpose of most of the software I wrote was to replace humans. The thought made me uncomfortable. But after a while, I connected the thought with all the rest of the mechanization and industrialization in society for hundreds of years. A new machine (or program) is valuable only to the extent that it somehow reduces the total amount of human labor to reach a given result, everything taken into account!

    The replacement of humans by machines was extremely clear to the luddites, the secret society of workers whose jobs were being eliminated or reduced to unskilled labor by the increased use of Jacquard Looms. The movement was named after one of the first people to smash a loom, Ned Ludd: 375px-Luddite

    The core of the movement was actions to destroy the job-killing looms that automated and de-skilled the worker's jobs: FrameBreaking-1812

    This is particularly interesting and relevant to computers eliminating human labor. No one thinks of what could be the modern equivalent of workers smashing the looms that threaten their jobs. An ironic twist is that the revolutionary Jacquard looms were mechanical computers — they executed a "program" that was encoded physically, enabling them to execute flawlessly elaborate patterns in the woven cloth.

    Luddites today

    Yes, there are luddites today. Lots of them. But they're not so crass as to pick up big sledge hammers and smash the looms that threaten their livelihoods. They're more subtle, and far more supported by elite society than the original luddites ever were. They're not even seen as resisting change — they're seen as highly trained professionals that we're so grateful to have. Very much the way that the skilled craftsmen displaced by machines were seen by themselves and most of their contemporaries!

    So who exactly are these modern luddites? They include many of the elite, highly-paid professions that ambitious young people strive for.

    • Lawyers. A growing fraction of lawyers are under attack by automation. The pain is shown by declines in law school graduate hiring, the growth of firms such as Legalzoom, and growing outsourcing to low-wage locations. The trend has been growing for at least 20 years. I have personally seen the resistance of lawyers to automation efforts.
    • Doctors. The ready-to-be-luddite feelings are strong here, though automation is still in its early stages. Doctors increasingly feel like they work in a non-stop assembly line, with patients questioning their statements because of Doctor Google. With insurance companies increasingly questioning and challenging their actions and clinical decision-tree software chomping at the bit, doctors can feel like targets. While hiring remains strong, automation efforts continue to gain strength.
    • Software engineers. It seems only fair that the people who drive automation should have their jobs automated. This isn't in the news, but in fact there have been many waves of software job elimination — masked by strong growth in newly emerging technologies. I have run into many programmers over the years who jobs were eliminated; some of them have left the field, and others slide into management positions, managing people doing jobs they barely understand. For example, during the 1970's there was an explosion of jobs building operating systems for minicomputers. Minicomputers are long gone, along with the diverse operating systems needing to be built. A long trend is skills degrading — today you can build a functioning web site without programming, while years ago building software with a sophisticated user interface required serious software effort.
    • Actors and musicians. We have more visual and musical entertainment available than at any time in human history. But the vast, vast majority of it is recorded and replayed! The vast majority of musicians, for example, were employed making adequate, best-available performances at local places. The number of such jobs is a tiny fraction of what it once was because of recordings. All the supporting jobs are greatly reduced as well. Even jobs in movie theaters have cratered, since people watch on personal and home devices.

    What will become of all those people and jobs? The answer has been the same for hundreds of years: there will be great pain for the individuals involved, but overall, the wealth of the population will grow. A clear example is agriculture. At the time of the American revolution, over 90% of the population was involved in agriculture. One step at a time, those jobs were automated, so that in 2019, only about 1% of the US population was employed in agriculture. Yet there's food enough for everyone, and jobs that didn't exist back then.

    Conclusion

    As a person who has been directly involved in automation for over 50 years, I have seen the jobs I've had and the skills I've acquired become obsolete over and over again. I've also seen or been part of eliminating through automation many jobs, a surprising number of them highly skilled jobs. An early one was the capable engineers who ran the oil refinery in Venezuela in the late 1960's whose jobs were eliminated by the software I worked on at the time — the software just did a better job than the engineers could possible do! I describe this here.

    Luddites can and often do postpone their replacement by automation. But they lose in the end. I don't see an end to this process any time soon.

  • Laser Disks and Workflow Illustrate the Insane, Fashion-Driven Nature of Software Evolution

    Computers are so exact and numbers-based that we tend to think of the people in charge of them as white-jacketed scientists with esoteric knowledge driving towards calculated optimal results, much the same way we imagine that the scientists and engineers calculated the path of the Apollo capsule to the Moon. If the Apollo program were run the way most computer projects are, it would have been a miracle if the capsule made it off the launch pad, much less traced an incredibly exact path from Cape Canaveral to the chosen landing spot on the Moon, a journey of over a quarter million miles.

    The reality is that the application of computers and software to automation is painfully slow, usually failing to achieve optimal results by large margins, and often lagging behind what is achievable by decades. The reasons appear to be a semi-random mixture of commercial interests, technology fashion trends, ignorant and risk-averse managers, and technical specialists rooted in the past and wearing blinders. That’s all!

    Here’s the story of a typical example.

    Microfilm, document imaging and laser disks

    Long before computers, microfilm was the preferred medium of librarians and archivists to preserve paper documents for future use. Microfilm was more compact than paper, and lasted much longer. What happened when computers came along is a curious and educational story, a prime example of how computer use and software evolve.

    Disclosure: I lived and worked as a participant in this history during the late 1980’s and early 1990’s. The story I will tell is not systematic or comprehensive, but like a traveler’s tale of what he did and saw.

    A side show in the long evolution of computer storage is the laser disk. A laser disk has data recorded onto it either as a whole, during manufacturing, or incrementally, as part of a computer system. The laser disk had quite a run in the consumer market in the form of CD’s, and then DVD’s. They were an improvement on existing methods for distributing digital content of all kinds.

    In the computer industry, they took the form of WORM (write-once-read-many) drives, and were used to record archival data that should not be updated, but could be read as often as needed. The technology, as often happens, went searching for problems it could solve. In an effort to reduce human labor more than microfilm could, jukeboxes were invented. Each jukebox had a disk reader and many shelves for disks, with a mechanism to load a selected disk into the reader. FileNet, now part of IBM, built one of the first of these systems in the mid-1980’s.

    At around the same time that WORM drives became practical, paper document scanners were evolved from systems to capture images of the paper onto microfilm. In each case, light was shown onto paper and focused onto a target; a scanner replaced the film with an optical sensing device.

    How could we get people to buy WORM drives and jukeboxes, the creative marketing people wondered? The departments that put documents onto microfilm weren’t going for it – the new systems were much more expensive, microfilm wasn’t accessed often enough to make it worth connecting it to computers, and the records retention people were understandably skeptical that any computer disk would be readable 100 years from now, as microfilm will be.

    Adding workflow to document imaging

    I have no idea who made the audacious leap, but some creative person/group came up with the idea of doing computer document scanning at the start of the process, when paper arrived at a location for processing, instead of microfilmed at the end for archival purposes. But WHY??? The creative types scratched their heads until they were losing hair rapidly. Why would anyone make responding to loan requests or whatever even longer than it is today by introducing the step of scanning?

    GENIUS TIME: LET’S INVENT “WORKFLOW”!!!

    Everyone has an image for what “workflow” is. For documents, it’s the path of processing steps from arrival to filing. In larger organizations, there are often multiple departments involved in processing a document, so there are in-boxes and out-boxes and clerks transporting documents from one to the other.

    Rubbing their hands and cackling, the scammers worked out their strategy: We’ll scan the documents and convert them to images so that we can move images from place to place electronically instead of piles of paper! We’ll call it “document imaging workflow.” It will deliver massive benefits to the whole organization, to everyone who touches the paper, not just to the tiny group who files and archives at the end – and storing the scanned documents onto WORM drives will do their job too! Hooray! We will leap-frog the old-fashioned, paper-flooded back office into the modern electronic age. What’s not to like?

    It was audacious and brilliant. It was a scam that would wither if confronted with just a little common sense or cost accounting. Which, true to the pattern of such fashion-driven trends, it never had to confront!

    Implementing workflow

    A typical target environment for implementing workflow was large offices with large numbers of documents, often forms of some kind, arriving every day. The forms would go from the mail room to the relevant people’s desks for processing. The person handling the form would often have an IBM 3270 terminal for interacting with a mainframe computer. When the person was done with the form, they would place it in one or more outboxes for further processing at other desks, or for filing. Sometimes documents would go into file cabinets for a period of time, but all would end up being archived, and sometimes microfilmed for permanent storage.

    Implementing workflow first of all required scanning the documents as a first step, and storing them immediately on normal computer disks, and ultimately on WORM drives, thought to be the equivalent of microfilm. Once scanned, the documents would be managed by workflow software, which would route them to the appropriate desk in the appropriate department for processing. Every desk needed to have the old computer terminal replaced or augmented with a large, high-resolution image terminal, capable of handling at least the display of the document, and sometimes also the computer screen. The old, slow connections between terminals and mainframe needed to be replaced with a fast local area network, and there needed to be substantial servers for handling the local image handling and workflow routing.

    Of course, lots of analysis and programming was involved in addition to the equipment. Workflows had to be analyzed and programmed, and all the management, reporting and exception handling taken care of.

    The benefits of workflow

    When a movement like document imaging workflow has a head of steam up, no one seems to ask whether it’s a good idea, and if so, how good of a good idea it is – quantitatively. When I was involved, everyone threw around there would be at least a 30% productivity improvement due to workflow, and that would make all the expense and disruption worth it. I never encountered a single study that measured this.

    Think about it for a minute. Would looking at an image of a document make you work faster than you would looking at the original paper? Would picking the next paper from the in-box be slower than getting the system to show you the next image? What about the labor of the clerks moving stacks of paper from one person’s outbox to the next one’s inbox? It would certainly be saved, but chances are a single clerk could handle the paper movement for many dozens of people. And remember, there’s the scanning and indexing that’s been added and the massive computer and software infrastructure that has to be justified.

    It’s obvious why no one EVER did a cost-benefit analysis of workflow – the back-of-the-envelope version easily shows that it’s a couple zeros away from making sense.

    I remember a couple of out-of-it, tech-fashion-disaster know-nothings mildly thinking out loud about how workflow could possible be worth it financially. The immediate response from workflow fashion leaders was upping the ante – don’t you know, the wonderful workflow tool from (for example) FileNet lets you program all sorts of efficiencies for each stage of work; it’s something we really need to dive into once we get the basic thing installed. End of subject!

    So who fell for this stuff? Just a who’s-who list of major organizations. Each one that fell for it increased the pressure for the rest to go for it. None of them did the numbers – they just knew that they couldn’t fall too far behind their peers.

    Conclusion

    It’s hard to think of a field that’s more precise and numbers-oriented than computers. Who has a clue what actually goes on inside those darn things? And the software? There are millions of lines of code; if a single character in any of those lines is wrong, the whole thing can break The impression nearly everyone has, understandably, is of a field that is impossibly precise and unforgiving of error. When the software experts in an organization say that some new technology should be adopted, sensible people just agree, particularly when there’s lots of noise and other major organizations are doing it. It can be even more gratifying to get known as pioneering, as one of the first organizations to jump on an emerging tech trend!

    Somehow, the smoke and noise from the software battlefield is so intense that the reality is rarely seen or understood. The reality, as I’ve illustrated here, should result in everyone involved being sent to the blackboard by a stern teacher, and being made to write, over and over: “I pledge to try harder to rise above the level of rank stupidity next time.”

  • The Evolution of Software

    There is strong interest in the latest developments in software. No one wants to be left behind.

    At the same time, there is a peculiar (to me) lack of interest in whatever the latest thing grew out of or evolved from. What are the new thing's predecessors? What did that thing grow out of. Have similar things appeared in the past? Are there patterns we can observe here, or are the new software things that explode onto the scene just a meaningless sequence of random events? Are they things that we obsess about while they're here without questioning or wondering where they came from, without asking what else they may resemble in some way? Are they things we just forget about and ignore once they depart the stage of software fads?

    The questions have obvious answers. Answers that no one seems to be interested in.

    So long as we fail to ask these crucial questions, we will remain ignorant of software in deep ways, and will continue to stumble from fad to fad without improvement or understanding. History is important!

    Evolution in Science

    If you study different sciences and their history, you notice that, at some point, scientists begin to pay attention to evolution, i.e., change over time of the thing being studied.

    The most obvious thing is evolution of an individual item over time. In biology, you have the cycle from birth to death. In geology, you have the change of mountains, rivers, beaches and glaciers. In materials science you have the change of objects as they are subjected to various conditions over time, for example the way that iron rusts.

    Evolutionary studies reach a breaking point and get really interesting when you study groups and types and see the patterns of change. In biology, this was the revolution that happened with Darwin and his study of the evolution of species. Something similar happened in geology with plate tectonics.

    When you put the patterns together, things get totally amazing and transformative. While no longer fashionable to talk about and study, the parallels between the growth stages of an individual animal and the evolution of species are obvious, though not literal at a detail level.

    Studying evolution is an important stage in the evolution of most sciences!

    It's long-since past time to study software evolution, as an integral part of software science.

    Software Evolution

    Software evolves at every level.

    At a basic level, software languages evolve. I have given an introduction to this subject here. The evolution of software languages resembles biological evolution in remarkable ways, as species (languages) emerge and evolve in response to new conditions, including other species. Similarly to biology, languages evolve; sometimes many species descend from them; and sometimes a species goes extinct. See this snapshot, for example:

    11

    (Credit)

    Consistent with the determination of nearly everyone involved with software to ignore its history, this chart and anything like it are ignored, and the evolutionary principles behind it are not studied. When you start to study them, you learn amazing things. Suddenly some of the random change in the world of software languages starts to make sense.

    Software languages are the very most basic things about software. Just as interesting and vastly more important are the programs written in software languages. Everyone involved in writing or maintaining a software program (application) just thinks about that program. It definitely occurs to business people what the program can do compared to its immediate competitors, if any. Yes, there is a kind of "survival of the fittest" in software evolution.

    What about the "insides" of the program compared to other program insides? Sometimes people are aware of gross differences, like the language the program is written in. But the curiosity normally stops there.

    The program "insides" have huge practical consequences. Depending on the details of how a program is written, it can be more or less:

    • reliable/buggy
    • easy to hack (secure)
    • expensive to run/operate
    • dangerous/costly to change
    • able to respond quickly to changing loads
    • speedy or slow

    People commonly talk about structure or architecture, but those things are just the tip of the iceberg.

    When you dig in, you find the equivalent of islands and continents of software. I've treated this subject here, for example. For example, you may have spent some number of years after getting your CS degree working on website design, and interacted with a community of people similarly engaged. Not everyone uses the same tools or does things the same way, of course, and you're likely to think there's quite a variety of approaches.

    Then you go wild, dive into hospital management systems, and your mind is blown. What you thought was the wide variety of software and tools used by website designers turns out to be a whole continent apart from what you find in hospital automation. You learn that the flagship product of the company that has more than a third of the market (Epic) is written in a language you've never even heard of! It's like growing up in New Jersey among what you think is a very wide variety of people and going to Chennai — a huge city in India you've never heard of, whose population is larger than your whole state, where the main language is one you've never heard of, written in a script you've never before seen.

    Even more ignored are the obvious (to those who trouble to look) repeating patterns that can be observed in applications as they are first written and then evolve over time.

    The Study of Software Evolution

    To put it bluntly, the study of software evolution has barely begun. A few isolated souls, hardy or foolhardy as you like, have dipped their toes into the deep waters of software evolution. Those few have found worlds to explore, oceans whose depths have yet to be plumbed.

    This is particularly the case because software is a "the only reality is the present" kind of field, as it now stands. It doesn't have to be this way!

    Very few people study the history of math. They don't need to: anyone who learns math starts with math as it existed thousands of years ago. If they do well in math, they get all the way up to … the late 1600's, when Newton and/or Leibnitz discovered calculus! Each step in math is based on and often depends on the earlier steps.

    In some sense, this is also true of software. Except not really.

    First, knowing nothing about software, you can dive right into the latest language and tools and use them to make something that works. Anyone think you could do that with differential equations, I mean without already knowing how to count, add and multiply? The difference is that in software, the "accomplished" new-comer really is standing on generations of older software … of which s/he is blissfully unaware! The new programmer, of course, uses an incredible array of operating systems, drivers, networking, file and database systems, and on and on. These form the "land" on which the new programmer "stands" to work. None of the wonderful new stuff would work without it. The big difference, of course, is that using the underlying software and understanding the underlying software are two rather different things. This makes software totally different from not just math, but most of the other legitimate sciences (leaving out the fake ones like political "science" and the rest).

    Second, when you're building a program, chances are excellent that you're not the first person to try building something very much like that program. Lots of details may be different, but even the details are probably not as unique as you might think. Suppose you need a program that sets up an account for a person, and then when the person does stuff, the details are checked with the person's account, and if OK are added to the account, and if not are rejected but recorded. This high-level description applies to a sales operation with customers, a medical office with patients, a manufacturing company with customers, and a bank. Given that programs like this have been built tens of thousands of times, don't you think that the ways of building programs like this would have evolved? That there would be pre-built parts, specialized tools for using the parts, building new ones and gluing them together? Special methods that have evolved and been refined to make sure it's done in the optimal way? Did any of you with any level of degree in Computer Science learn any of that — other than, of course, some boiler-plate about the virtues of object-oriented languages and maybe some other fashionable stuff? Of course not! Anyone with any sense of fashion and/or status knows that these are not career-advancing subjects. Anything involving software history or comparative study of software projects is career-killing.

    Conclusion

    Software evolves. But it evolves differently than other things studied by science. Virtually no one studies software history in any way, much less the patterns of evolution that become apparent when you study that history. There have been a couple of attempts to learn why software doesn't get better in the way many other things do, most notably Fred Brooks with his book The Mythical Man-Month and other writings. I have dissected this flawed analysis in my work on Software Project Management and Wartime Software. See this for a start.

    I have worked for years trying to identify some notable patterns in software evolution, and will be releasing some of that work in the coming months.

  • Recurring Software Fashion Nightmares

    Computer software is plagued by nightmares. The nightmares vary.

    Sometimes they are fundamentally sound ideas that are pursued in the wrong way, in the wrong circumstances or at the wrong time. Therefore they fail – but usually come back, sometimes with a different name or emphasis.

    Sometimes they’re just plain bad ideas, but sound good and are cleverly promoted, and sound like they may be relevant to solving widely acknowledged problems. Except they just don’t work. Sometimes these fundamentally bad ideas resurface, sometimes with a new name.

    Sometimes they’re a good idea for an extremely narrow problem that is wildly applied beyond its area of applicability.

    The worst nightmares are the ones that slowly evolve, take hold, and become widely accepted as part of what “good programmers” believe and do. Some of these even become part of what is ludicrously called “computer science.” Foundation stones such as these, accepted and taught without proof, evidence or even serious analysis, make it clear to any objective observer that computer “science” is anything but scientific, and that computer “engineering” is a joke. If the bridges and buildings designed by mechanical engineers collapsed with anything close to the frequency that software systems malfunction, the bums and frauds in charge would have long since been thrown out. But since bad software that breaks is so widespread, and since after all it “merely” causes endless delays and trouble, but doesn’t dump cars and trucks into the river, people just accept it as how things are. Sad.

    Following is a brief review of sample nightmares in each of these categories. Some of what I describe as nightmares are fervently believed by many people. Some have staked their professional lives on them. I doubt any of the true adherents will be moved by my descriptions – why should they be? It was never about evidence or proof for the faithful to start with, so why should things change now?

    Sound Ideas – but not here and now

    • Machine learning, AI, OR methods, most analytics.
      • These things are wonderful. I love them. When properly applied in the right way by competent people against the right problem at the right time, amazing things can be accomplished. But those simple-sounding conditions are rarely met, and as a result, most efforts to apply these amazing techniques fail. See this.

    Bad ideas cleverly promoted

    • Microservices.
      • Microservices are currently what modern, transformative CTO’s shove down the throats of their innocent organizations, promising great things, including wonderful productivity and an end to that horror of horrors, the “monolithic code base.” While rarely admitted, this is little but a re-incarnation of the joke of Extreme Programming from a couple decades ago, with slightly modified rhetoric. A wide variety of virtues are supposed to come from micro-services, above all scalability and productivity, but it’s all little but arm-waving. If software valued evidence even a little, micro-services would be widely accepted as the bad joke it is.
    • Big Data
      • There's nothing wrong in principle with collecting data and analyzing it. But in practice, the whole big data effort is little but an attempt to apply inapplicable methods to random collections of data, hoping to achieving generic but unspecified benefits. See here.

    Great ideas for a narrow problem

    • Blockchain.
      • My favorite example of an excellent solution to a problem that has been applied way beyond its area of legitimate application is blockchain. Bitcoin was a brilliant solution to the problem of having a currency that works with no one in charge. How do you get people to “guard the vault” and not turn into crooks? The way the mining is designed with its incentives and distributed consensus scheme brilliantly solves the problem. However, the second you start having loads of crypto-currencies, things get weaker. Once you introduce “smart contracts,” there’s a fly in the ointment. Take away the currency and make it blockchain and you’ve got a real disaster. Then “improve” it and make it private and you’ve got yourself a full-scale contradiction in terms that is spectacularly useless. See here.

    Bad ideas that are part of the Computer Science and Engineering canon

    • Object-orientation.
      • Languages can have varying ways of expressed data structures and relating to them. A particular variant on this issue called “object-orientation” emerged decades ago. After enjoying a heyday of evangelism during the first internet bubble, O-O languages are taught nearly universally in academia to the exclusion of all others, and adherence to the O-O faith is widely taken to be a sign of how serious a CS person you are. For some strange reason, no one seems to mention the abject failure of O-O databases. And no one talks about serious opposing views, such as that of Linus Torvalds, the creator/leader of the linux open source movement, which powers 90% of the world's web servers. Programmers who value results have long since moved on, but academia and industry as a whole remains committed to the false god of O-O-ism.
    • Most project management methods.
      • I’ve written a great deal about this, including a book. Whether it’s the modern fashion of Agile with its various add-ons such as SCRUM, project management methods, ripped from other disciplines and applied mindlessly to software programming, have been an unmitigated disaster. The response? It ranges from “you’re not doing it right,” to “you need to use cool new method X.” Yet, project management is a profession, with certifications and course on it taught in otherwise respectable schools. A true nightmare. See these.
    • Most software QA methods.
      • All software professionals accept what they’ve been taught, which is that some kind of software automation is an essential part of building software that works and keeping it working. Except the methods are horrible, wasteful, full of holes and rarely make things much better. See this.

    Conclusion

    I make no claim that the list of items I’ve discussed here is comprehensive. There are things I could have included that I have left out. But this is a start. Furthermore, if just half the items I’ve listed were tossed and replaced with things that actually, you know, worked, much better software would be produced more quickly than it is today. I don’t call for perfection – but some progress would sure be nice!

  • Social Media Has a Long History

    It seems like the whole world is in an uproar about social media, with frequent revelations of awfulness and malfeasance. The uproar is about social media such as Facebook, Twitter and the rest. The trouble is these issues have existed in one form or another in social media going back … hundreds of years. What we are seeing is ignorance of the present combined with ignorance of the past. In other words, business as usual.

    The Core Drivers of Social Media

    Most people care about their place in society — their status. They care about how they're perceived, who knows who they are and how others relate to them. While this drive seems to come to a kind of perverse peak in middle and high school, it persists for most peoples' lives.

    Intimately related to caring what others think is the drive to express what you think and what you've done. While this is related to influencing others' thoughts, it seems to be a kind of innate drive as well.

    In short, you want to tell people what you've done, what you think, and you want to hear about other people, particularly those you somehow are involved with or even just similar to in some way.

    Closely related to this are the core concepts of status and fashion. It's a basic urge to want to see your status reflected publicly if it's high, and many people have strong interest in what high-status people do and how they do it. Particularly as fashions of various kinds wax and wane, from clothing to activity to speech, people who want to increase their status have an intense interest in learning what the new things are.

    Key Characteristics of Social Media

    What makes something Social media vs. some random other kind of media? It's pretty simple: social media mostly consists of media (words and pictures) that is about and either written/created by the person or sourced from that person. It's about what a person says, thinks or does.

    Now let's get to the other key characteristic: money. Who pays for social media? After all, it costs quite a bit to produce it, and that money has to come from somewhere. Historically, the people who consume the media pay a little, while … get ready … advertisers pay a lot. Today, the incremental cost of delivering social media to the person who consumes it is so little that no one bothers to charge for it — the whole cost is borne by advertisers.

    Social media is an amazing phenomenon, deeply rooted in human drives and emotions. People produce the content for it for free — they are glad to have things about themselves distributed at little effort of their own to those they may like to know about it. And they read about themselves and people to whom they are socially connected with, paying to do so if necessary. They can't help but knowing that the ads that are intermixed with the "content" are going a long way (in the electronic world, all the way) to paying for their reading pleasure, but it rarely bothers them. They also know that ads are targeted to particular groups of readers. It makes common sense, after all. No big deal — if I were an advertiser, of course I'd want to show my ads to people who are likely to buy what I'm selling!

    Earlier versions of Social Media

    People like to imagine that social media are strictly electronics-age things. Mark Zuckerberg invented it, didn't he? No, sorry! Social media have been around for a looooong time. I could go all the way to ancient Sumer and Egypt, but I think the point will be clear enough with more recent examples.

    Here is a notice in the Pittston Pa Gazette from 1928 about a function attended by my grandmother, Agnes Black:

    1928 social notice

    Here is an ad that helped pay for that information to be printed:

    1928 ad

    More recently, here is a notice in the same paper from 1955 about a visit made by my parents:

    1955 news

    Here is one of the ads that helped pay for the notice.

    1955 ad 2

    The notice was actually fake news! My parents visited with their two children, David Bruce Black and Douglas John Black — not David and Bruce. The advertisers don't care a whit — they just want the eyeballs to persuade them to buy some hot new technology:

    1955 ad

    I could give examples from many other places and centuries. Things have evolved, but since the principles are rooted in human in human nature, not as much has changed in principle as you might think.

    Conclusion

    Everything is about people. A great deal about people is relationships and status. The experiences we had in middle school and high school didn't disappear into nothingness. They just evolved as each of us entered new groups of people, each with its own pecking order and rules for engagement. One of the most ironic things about modern social media is that certain groups of people are really upset about what gets published, and want to make sure that only the "truth" is published. They, of course, want to be in charge of defining what "truth" is. Sorry, guys, in the world of social relations and much else, "truth" is nothing but a pretty veneer on top of raw power. Yes, your grace, your honor. Why should it be different now that we're staring into little screens and swiping while we walk?

     

  • Uber’s Evolution from the inside

    Uber is a great success; you can tell because competition has sprung from all corners, and scads of new companies say they’re going to be “the Uber of X.” Along with great success comes great myths, which inform a set of assumptions about how Uber got to be Uber. The myths are all derivatives of the broadly accepted narrative of how tech companies start and grow. In every case I’ve examined, fact-based history contradicts these myths and explodes what “everyone knows” about Uber and every other tech company. I’ve explored this theme in a couple of cases, and I’ve just had the opportunity to question an early Uber driver, whose story provides fascinating insight on just a small part of what really happened.

    Recruiting the first drivers

    My driver was one of the first handful of drivers in the New York City area. Uber is a network-effect company, i.e., customers need access to a large pool of drivers, while drivers need business from a large pool of customers. How do you get that started? Of course I don’t know the whole story, but I did hear how my driver informant was recruited.

    He was sitting in his car near Madison Square Park in Manhattan with a half hour to go before his ride would appear. Someone knocked on his window. He rolled it down, and it was two people who wanted to talk with him about a new way to get rides. It started to rain, and he invited the people to sit in his car, for which they were very grateful. He ended up signing up for their service, encouraged by a $1,000 signing bonus, and further encouraged by being paid $35 an hour just for keeping the app on his phone. He was then paid $1,000 for each additional driver he referred who signed up, and a further $1,000 when that driver took their first ride. That’s how a critical mass of drivers was recruited even when there was little work for them to do.

    Here’s the part of the story I found most impressive: the two people who recruited him were … the head of Uber NYC and Travis Kalanich, the co-founder and CEO of Uber. Travis Kalanick
    Yes, the big boss himself, personally walking the streets of NYC and hitting on strangers. Wow. For every 1,000 bosses who puts up one of those stupid pyramid-on-its-head charts Inverted pyramid
    during a company talk to say how he’s just there to serve the important people on the front lines, there might be a couple like Mr. Kalanich who act on it and get into the trenches themselves.

    Recruiting the first customers

    Uber today is an easy way to use an app to call for an on-demand car service. It’s all “get me a ride now. As in, now.” At the start, it wasn’t that way at all! It was mostly for really long rides, like from NYC to Boston or Philadelphia, and the rides were often round trips with a big wait in the middle. This is a tiny minority of all rides, but one that was typically arranged way in advance by phone. With Uber, you could get someone to pick you up in half an hour or less, which was great compared to the alternatives. It also meant that having just dozens of drivers available was OK.

    It was only as the population of drivers and customers grew that shorter trips with shorter lead times made sense – and the long-trip buyers were an ideal starting population as customers for that, so it evolved naturally.

    Evolution of the service

    Here are some tidbits about how the service has evolved.

    At the beginning, drivers were screened thoroughly. For example, every candidate was shown a map of Manhattan with no street names. A pin was placed on the map, and the driver had to name the streets. If you didn’t get them, you were out. The screening has gone way down over time.

    The cut taken by Uber has ratcheted up sharply. It started at just 10%, moved in a few steps and is now up to 38%. A huge difference.

    As the number of drivers and customers grew, Uber provided a “heat map” in the driver’s app to show where drivers were needed and where there was an excess, which helped drivers position their cars to best effect. A subsequent version of the app dropped this feature and replaced it with congestion pricing. My driver misses it, and thinks a new version is coming out with the heat map restored.

    The policies and standards for drivers have evolved quite a bit, something which is also visible to customers with the addition of Uber-X and evaluations of drivers – I learned that drivers can also evaluate customers! My informant says the evaluations are on the whole helpful – one incident reminded him he always has to inspect the passenger seat when someone leaves, for example, something he had been neglecting.

    In spite of all the changes, my driver/informant loves Uber. The main reason is he has control. He can start and stop any time. He can work more and make more money, or take time off any time he needs to, without asking anyone’s permission.

    Observations

    It appears that Uber did a very smart thing I have seen other companies do: pick a sub-market in which they could get traction, and then evolve into the actual, larger market they probably had in mind all along. The sub-market enabled them to get to a critical mass of drivers and riders with less friction.

    It's easy to focus on Uber as enabled by an app, or Uber as a marketplace. Those things are true. I think of them as being good choices (the app) and reasonable thoughts (marketplace), but somehow not essential.

    For the consumer, Uber provides the convenience of hailing a cab (no advance planning, scheduling, committing) with cars that you can't see. It would work just as well if you used your phone to dial a number, gave your location, and then were told how long you'd have to wait for the driver. An on-demand car service.

    For the driver, Uber provides control over time and money, as my informant made clear. When you drive a cab, you sign up for a shift. With a car service, someone schedules you. With Uber, the driver can opt in and opt out whenever it meets his needs, whatever those may be. Whether he got rides from an app or from someone calling him, the result would be the same.

    Conclusion

    Every company has a different story. Uber is no different. Even a narrow and limited glimpse into the factual history of Uber reminds us that every company ends up having a myth-based story that explains its success, and that if you’re going to learn something useful from their success, you have to find and pay attention to the facts. AND, the facts of similar companies that failed.

  • Innovations that aren’t: data definitions inside or outside the program

    There are decades-long trends and cycles in software that explain a great deal of what goes on in software innovation – including repeated “innovations” that are actually more like “in-old-vations,” and repeated eruptions of re-cycled “innovations” that are actually retreating to bad old ways of doing things. Of the many examples illustrating these trends and cycles, I’ll focus on one of the most persistent and amusing: the cycle of the relationship between data storage definitions and program definitions. This particular cycle started in the 1950’s, and is still going strong in 2015!

    The relationship between data in programs and data in storage

    Data is found in two basic places in the world of computing: in “persistent storage” (in a database and/or on disk), where it’s mostly “at rest;” and in “memory” (generally in RAM, the same place where the program is, in labelled variables for use in program statements), where it’s “in use.” The world of programming has gone around and around about what is the best relationship between these two things.

    The data-program relationship spectrum

    At one end of the spectrum, there is a single definition that serves both purposes. The programming language defines the data generically, and has commands to manipulate the data, and other commands to get it from storage and put it back. IMG_0002
    At the other end of the spectrum, there is a set of software used to define and manipulate persistent storage (for example a DBMS with its DDL and DML), and a portion of the programming language used for defining “local” variables and collections of variables (called various things, including “objects” and “structures.” At this far end of the spectrum, there are a variety of means used to define the relationship between the two types of data (in-storage and in-program) and how data is moved from one place to the other. IMG_0001

    For convenience, I’m going to call the end of the spectrum in which persistent data and data-in-program are as far as possible from each other the “left” end of the spectrum. The “right” end of the spectrum is the one in which data for both purposes is defined in a unified way.

    People at the left end of the spectrum tend to be passionate about their choice and driven by a sense of ideological purity. People at the right end of the spectrum tend to be practical and results-oriented, focused on delivering end-user results.

    Recent example: Ruby and RAILS

    The object-oriented movement tends, for the most part, to be at the left end of the spectrum. Object-oriented people focus on classes and objects, which are entirely in-program representations of data. The problem of “persisting” those objects is an annoying detail that the imperfections of current computing environments make necessary. They solve this problem by various means; one of the most popular solutions is using an ORM (object-relational mapper), which does the work of moving objects between a DBMS and where they “should” be nearly automatic.  It also can automate the process of creating effective but really ugly schemas for the data in the DBMS.

    A nice new object-oriented language, Ruby, appeared in 1995. It was invented to bring a level of O-O purity to shell scripting that alternatives like PERL lacked. About 10 years later, a guy was working in Ruby for Basecamp and realized that the framework he'd created for it was really valuable: it enabled him to build and modify things his company needed really quickly. He released it to open source and became known as the RAILS framework for Ruby, the result being Ruby on RAILS. Ruby on RAILS was quickly adopted by results-oriented developers, and is the tool used by more than half a million web sites. While "pure" Ruby resides firmly at the left end of the spectrum, the main characteristic of the RAILS framework is that you define variables that serve for both program access and storage, putting it near the right end of the spectrum.

    Rhetoric of the ends of the spectrum

    Left-end believers tend to think of people at the other end as rank amateurs. Left-enders tend to claim the cloak of computer science for their style of work, and claim that they're serious professionals. They stress the importance of having separation of concerns and layers in software.

    Right-end practitioners tend to think of people at the other end as purists, ivory-tower idealists who put theory ahead of practice, and who put abstract concerns ahead of practical results. They stress the importance of achieving business results with high productivity and quick turn-around.

    It goes in cycles!

    There have always been examples of programming at both ends of the spectrum; neither end disappears. But what happens is that the pendulum tends to swing from one end to the other, without end up to now.

    In 1958, Algol was invented by a committee of computer scientists. Algol 60 was purely algorithmic — it only dealt with data in memory, and didn't concern itself with getting data or putting it back. Its backers liked its "syntactic and semantic purity." It was clearly left-end oriented. But then practical business people, in part building on earlier efforts and in part inspired by the need to deliver results, agreed on COBOL 60, which fully incorporated data definitions that were used both for interacting with storage and for performing operations. It was clearly right-end oriented. All the big computer manufacturers, anxious to sell machines to business users, built COBOL compilers. Meanwhile, Algol became the choice for expressing algorithms in academic journals and text books.

    There was a explosion of interest in right-end languages during the heyday of minicomputers, with languages like PICK growing in use, and niche languages like MUMPS having their day. The rise of the DBMS presented new problems and opportunities. After a period of purism, in which programmers struggled with using left-end languages that were supposed to be "easy" like BASIC to get jobs done, along came Powersoft's Powerbuilder, which dominated client-server computing because of the productivity win of integrating the DBMS schema with the language. The pendulum swung back with the rise of the internet and the emergence of "pure" java, with its arms-length relationship to persistence, putting it firmly in the left-end camp. Since Java wasn't pure enough for some people, "pure" Ruby got out there and grows — and some Ruby user is under pressure to deliver results and invents RAILS, which then spreads to similarly-minded programmers. But "amateurs" who like Java but also like productivity invented and are spreading Grails, a derivative of Java and RAILS that combines advantages of each.

    So is RAILS an innovation? Java? I don't think so. They're just minor variants of a recurring pattern in programming language design, fueled by the intellectual and emotional habits of groups of programmers, some of whom push towards the left end of the spectrum (with disdain for the amateurs) and others of whom push towards the right end of the spectrum (with disdain for the purists).

    Conclusion

    This spectrum of the relation between data and program has persisted for more than 50 years, and shows no signs of weakening. Every person or group who, finding themselves in an environment which emphasizes the end of the spectrum they find distasteful, tries to get to the end they like. Sometimes they write a bunch of code, and everyone thinks they're innovative. Except it would be more accurate to say they're "inno-old-ive." This is one of the many patterns to be found in software history, patterns that account for so much of the detail that is otherwise hard to explain.

Links

Recent Posts

Categories