Category: Software Myth-Conceptions

Software is invisible. The vast majority of people who talk about it have little or no practical experience writing it. But software is incredibly important and people need to be able to talk about it: creating it, modifying it, installing and using it and making it work as intended. How do you talk about something that's invisible? Answer: you use real-world metaphors that everyone knows and talk in those terms. Given the importance of the process of creating and modifying software, you also use real-world metaphors for the processes, methods and techniques that people use for those things on software.

Given the high cost, long time periods and risky outcomes of software development, real-world metaphors are used as models for how to create what seem like the equivalent things in the invisible world. Software professionals use the same metaphors and think in the same terms as their non-programming counterparts, the more so as advancement in software normally correlates with doing less hands-on programming.

Before long, the metaphors turn into highly detailed, structured things embodied in software tools for software management, and regulations for how software must be built in order to be acceptable for acquisition by governments and major enterprises. The software in medical devices, for example, is highly regulated in this way, everything from requirements to testing and modifications.

These metaphors are great for communication among people involved in software in some way, but disastrous for building fast, efficient software, quickly and effectively. The metaphors are largely to blame for the rolling crisis/disaster of software that was first recognized over 50 years ago.

Small groups of smart, hard-working software nerds sometimes break out of metaphor prison and use techniques that naturally grow out of the experience of programmers who live in the land of software, which, though invisible to the vast majority of people, is vividly visible to them. They are usually simple ideas that come out of the writing real-world software that meets needs and evolves with experience. Top programmers often think of themselves as "lazy," meaning that they are aware of how they spend their time and find ways to avoid doing things that don't advance the cause. They visualize the world of software they're working on, always looking for ways to spend less time writing less code to get more stuff done without errors.

The methods that they use rarely conform to any of the established software methodologies and typically make non-software people uncomfortable. I've observed and described many of those techniques, which as a whole I term "wartime software."

In this post I will identify and summarize some of the widely accepted metaphors that continue to wreak havoc on software development

The factory metaphor

The metaphor of the software factory tends to float around. Wouldn’t it be nice if we could churn out software as reliably and predictably as a factory? We’d put the requirements and other goals and constraints into the factory, and after a series of careful, orderly processing steps, out would come a wonderful piece of software! Unfortunately, factories make copies of things that are already designed and tested. All the effort goes in to making the design of a new car or drug; sometimes the design is no good and has to be discarded. Once you’ve made test versions and made sure they operate the way you planned, you turn the design over to the factory for making lots of identical copies of it. See this for more.

The building metaphor

Buildings and roads are widely used as metaphors for software. Lots of the words are the same: we have requirements, architects, standards that must be met, common design patterns. We have project plans with resource requirements and target dates. We might put the project out to bid. We use specialists for each aspect of the work. We build in phases and have regular reviews by the owners and quality inspectors. At the end we have walk-throughs, which may result in punch lists of defects to be corrected. Then we have the closing and we’re off and running!

It’s a powerful metaphor. It’s better than the factory metaphor because each house is uniquely built by a team, just like each piece of software is uniquely built by a team. The trouble is that a house just sits there. It’s a passive physical object. Software is NOT a physical object; it’s mostly is a set of actions to be taken by a computer, along with some built-in data. See this for more detail.

There's another issue with the metaphor. Buildings are built and then mostly used. Modifications are sometimes made, but they are normally small compared to the effort put into the original building. Software, on the other hand, is mostly changed. Vastly more effort is typically put into changing a body of software over time than was spent with the original building. Most of what most employees of software departments do is make changes. See this for more detail.

The Architect metaphor

The architect metaphor is closely related to the building metaphor. It's just as bad for the same reasons. Here is a detailed description that uses architecture software to illustrate the difference.

Waterfall and Agile

The waterfall method is the universally derided classic method of applying project management to building software. After years of failure with increasingly rigorous and detailed project planning to improve software development outcomes, it was supposed that waterfall just wasn't flexible enough. It was too rigid. More agility to respond to emerging knowledge and conditions would solve the problems. Thus, Agile software development method was invented, purposely built on the metaphor of a nimble-footed engineering team going through the process with lots of re-calibration and obstacle avoidance.

Hah. Nothing has changed except the rhetoric. Waterfall is a method that fixes the requirements and figures out the time to build them. Agile is a method that fixes the time periods and figures out what can be built in each. The difference in practice is that Agile has more overhead, and no better results. See this for more.

Objects in software

Could there be a more inappropriate metaphor? A bundle of data definitions and code is an "object?" A thing? Sorry, programs are NOT things. They're not passive; programs are actions!

Object-orientation started as an obscure design pattern for creating simulation programs, grew into a fad and has evolved into a standard part of software orthodoxy. I remember long ago hearing the sales pitch for the early O-O languages like Smalltalk. The business sales pitch was that it would greatly enhance the speed and quality of software development, while enabling changes to be more easily accomplished. The tech pitch was that by creating "objects" consisting of a definition of a related set of data and by including in that definition all the code that would read and write the data, concerns would be isolated and mistakes avoided by having data manipulated only by "its own" code.

The common metaphor was that objects were like Lego blocks, lending themselves to easily build elaborate structures and making changes to them without collateral damage. By throwing in things like inheritance — which was always illustrated by physical things like the hierarchy of species, families, etc. — the Lego effect was even stronger.

As it turned out, O-O has been a giant step backwards for software. While appropriate in some situations, as a rigid structure with no exceptions it imposes difficult and unnatural constraints on how code and data can be related and arranged, making everything more complicated than it needs to be. Lego blocks are great as physical objects, but their software version is a failure.

The walls and defenses metaphor

This one is huge in cybersecurity. Everyone talks about putting up walls to protect our computers, establishing and maintaining an effective periphery. We talk about attackers who want to "penetrate" our security defenses.

A great deal of software security is built according to this metaphor. The fact that software is invisible to everyone who makes important decisions about computer security leads directly to the string of monstrous security disasters that continue to this day. Here is a good way using baseball to understand the impact of the invisibility. Here is a detailed description of the impact of invisibility.

Conclusion

We all think using metaphors. It's not inherently a bad thing. But when the metaphor has a strong real-world grounding that everyone understands, it can lead to disaster when applied to software in ways that don't match the correct understanding of the software itself. This is unfortunately the case for a great deal of software methods and processes, and helps explain the ongoing software disaster that was first recognized over 50 years ago.

 

An amazing amount of discussion about software is governed by metaphors. This is not surprising, since software is so abstract, and invisible to most people. So a shared physical metaphor helps everyone communicate.

The trouble comes when the metaphor doesn't match with the reality of software, as it often doesn't. In those cases, what usually seems to happen is that what everyone understands about the metaphor is assumed to be true of the software. Bad things then happen.

One of the most common metaphors for software is a physical building. Buildings normally have architectural plans that are determined before ground is broken, with the architect being in charge. Then some digging happens and a foundation is created. From then on the framing is built, the electrical and mechanicals are installed, and the finishes are applied inside and out. Now the building is ready for use. Something everyone can understand, right? It makes sense that in software we also have architects, carefully planned and laid foundations, workers who fill things out according to plan, and finally finishes applied.

Software is usually built according to the physical building metaphor. Things like Agile methodology don't change things very much — they just mean that instead of a complete plan of everything from the beginning, things happen in phases. The physical building metaphor for software is unacknowledged, unchallenged … and pernicious. Ignoring the metaphor is one of the main things that smart software people who follow Wartime Software practices do to beat the competition and win.

Temporary physical buildings

When we think of software in terms of the building metaphor, we nearly always think of "permanent" buildings, ones with poured foundations and all the rest. Of course, you darned well better have it all planned out right — altering a foundation is no easy job!

What about the technology used to build temporary buildings? These aren't just tents for camping; some of them are extensive. A great example is the Frieze Art Fair, an international art fair held in NYC every year. It's huge; over 200 galleries exhibited last year. Here is a view of the extensive building from the air:

Here's a view of part of the exhibition area on the inside:

There were even a couple of restaurants:

All this was assembled in under a week, used for the time of the fair, and then torn down and carted away. No poured foundations. No big deal!

Architectural programs

Architects use software to do their work today. There are varying levels of programs they can use. I used a semi-professional one to design a house. The program was amazing. It helped me with everything. Here's part of a drawing from my project:

See the steps in the lower left? The software made it easy. I didn't draw every step. There was a staircase module which did most of the work for me. Same thing with roofs, walls, windows, doors, appliances and pretty much everything. At any time, it could generate a list of the building materials required.

In the physical world, I would need those building materials. But what if I were designing a building that existed only in a virtual reality world? I wouldn't have to go farther than the design — the design itself is all that would be needed! Now let's turn to software. Suppose I'm designing a UI and I want a footer on every page, and a logo on the top right corner. If I'm reasonably competent with software, I don't have to count the pages and place the logo and footer as many times as there are pages — I do it once, in a page master!

In the plans for the physical house, while it's easy for me to pop 10 identical windows into the drawing, when it gets built, someone has to build and install 10 physical windows. In the software world, the realization is done by more software. In the physical world, if I want a change to all 10 windows after they're installed … ugh! Go back to all 10 and re-do them. In software world, I go to the footer or logo definition, change what I want, and boom! It's done!

Yes, you can design software that doesn't work this way. Yes, all too many people do it the wrong way. They're stupid. What's important is that software can be designed the smart way, and it's not hard, so often enough, it is.

The building metaphor

So what does this mean? A couple things.

Using the building metaphor encourages software to be built the wrong way. It encourages people to think about it inappropriately. The smarter you are in building software, the farther away from the physical building metaphor you get. The metaphor even gets you to think about roles badly — the "architect" is someone who does lots of planning but no real work. Lots of workers are needed to build, workers who follow exact instructions. Then checkers who walk around with the plans making sure the workers did their job. All of this is counter-productive! It leads to bad process and bad software.

Let's knock the building metaphor down, bury it and move on.

 

Elephants are hard to figure out for people who are blind, but at least blind people can touch the elephant. Software is invisible, can't be touched and isn't anything like things we're familiar with. It's much harder to figure out, which is why what passes for understanding of software is a collection of unrelated-to-reality myths.

Elephants and Invisible Software

What would we think of elephants if the vast majority of people had never seen one in any way, even a picture? What if the few people who had encountered an elephant were blind and their experience limited to touching one part, for example a leg or the trunk?
And what if elephants were nonetheless a very important part of daily life, both personal and commercial? This is hardly thinkable, but that's what software is.

Myth-conceptions

Software is important and ubiquitous, but also invisible. The information we have about it is mixed, inconsistent and mutually incompatible. The vast majority of people, including programmers, resolve this by accepting a set of myths about what this unseen world is like. The myths tend to make sense and explain what software is about in a satisfying way. Most programmers, with their real but typically skewed experience with software, accept the same myths, but add details. The few programmers who can't or won't accept the common software myths find it difficult to communicate with the majority who hold firm to the myths.

Software "myth-conceptions" of this kind aren't just stories, or ways of talking about software. They are widely shared concepts and metaphors that guide how software is designed, built and maintained. Since many of them have been imposed on software from the outside, they tend not to fit with the reality of software — like things designed for chipmunks, but imposed on elephants. The result is that the process of building software is really expensive and hard, with crappy results.

I have, through hard personal experience over many years, grasped the mythical nature of many widespread thoughts about software. I hope someone will systematically lay out this world of myths that weighs down and perverts software. I have already described some of the myths in prior posts and hope to go into more depth in future posts.

The Software Factory

The job of a factory is to make high quality things. There are inputs, processes, stages, and work in process. At the end, you get what you expected, when you expected it. Let's make software work like a factory — a software factory! Hah! Here is more detail.

Software is like a Building

First you have to have a plan. You figure out what you need and work with an architect to lay out the details.  Then you put the job out to bid. You pick the best bidder by some criteria. The software builder gives you a plan with dates, and often expects partial payments based on progress. The builder gets permits and approvals.The site is prepared, the foundation is dug, footings are poured, the basement created, and the structure is built and the outside put on. The inside work is done — first the plumbing and heating, then walls, floors, appliances and finishes. Final inspections take place, and the building is ready for use.

Sounds practically like software, right? Yup! This is one of the prime myths; its fit with reality is marginal. Acting on this myth is a prime cause of software development disasters.

Software is just another thing to Manage

Lots of activities involving different skills happen in an organization. They all need to be managed. Management is a skill that can be taught — that's what getting an MBA is all about, after all. Some organizations need to build software, just like they need to set up a new office in Chicago. It's just another activity that needs a skilled, experienced person to manage it.

Not! Here's a post that goes into it.

Advances in Software

There's a widespread myth that software is one of those things that advances at a fierce rate. Maybe you were trained in a certain kind of software and were good at it, but after a couple of years you become hopelessly out of date and effectively unemployable. 

For certain kinds of low-level software roles like Novell Sysadmin, this is an appropriate thought. But in general, it's dead wrong. Software advances incredibly slowly; often it moves sideways, and sometimes goes backwards. When software is created for a new hardware environment, it is often extremely primitive, with all the "advances" that have been thoroughly proven by widespread use simply ignored. Much of what is presented as software "advances" is little but a fashion.

Software and Security

Securing software is much like securing a building or country. You have to have a wall. The wall should be strong and high. It should have lots of defenses against being breached. It must be constantly monitored. The places where people can enter must have extra protection, and everyone who enters must have ID and submit to a search. There are well-understood methods for accomplishing all this, and conforming to accepted procedure is essential.

That's the myth. The myth guides the vast majority of cyber-security thinking. It is pernicious and wrong. I've written about this already, and will write more.

Modern Software Methods

It's true (according to the myth) that software has been a chancy thing to build. There have been lots of delays, overruns and out-right failures. But now, with method X, we've put all that behind us. We finally understand how to create a flexible, dynamic process that puts all that behind us. We are all (for example) Agile with certified Scrum Masters.

This myth has persisted for decades. All that changes is the value of X. This myth is a perennial favorite. I've written about Agile specifically here, and will write more.

Software Innovation

Software is an exciting field, with innovation constantly frothing about. Software engineers are highly creative people — after all, every line of software they write has never been written before! It's no wonder that the cup of software is just bubbling over with innovation.

Nice thought. But the fact is, after all these years, software projects fail left and right. Managers grasp at all sorts of things in vain attempts to increase the odds of success. Innovation is all about making something better; the challenge in software is the struggle to rise to the level of adequacy.

Conclusion

The myth-conceptions go on and on. There are a myriad of "solutions" floating around for any given software problem. The vast majority of the solutions make things worse.

There are in fact, fast, effective ways to build software. But the methods are flagrantly at odds with most of the myths that dominate the industry. So the methods remain marginal, employed mostly by small groups of outsiders who are desperate, and really want to win.