The Black Liszt

Category: Occams Razor Occamality

  • How to Improve Software Productivity and Quality: Schema Enhancements

    Most efforts to improve programmer productivity and software quality fail to generate lasting gains. New languages, new project management and the rest are decades-long disappointments – not that anyone admits failure, of course.

    The general approach of software abstraction, i.e., moving program definition from imperative code to declarative metadata, has decades of success to prove its viability. It’s a peculiar fact of software history and Computer Science that the approach is not mainstream. So much the more competitive advantage for hungry teams that want to fight the entrenched software armies and win!

    The first step – and it’s a big one! – on the journey to building better software more quickly is to migrate application functionality from lines of code to attributes in central schema (data) definitions.

    Data Definitions and Schemas

    Every software language has two kinds of statements: statements that define and name data and statements that do things that are related to getting, processing and storing data. Definitions are like a map of what exists. Action statements are like sets of directions for going between places on a map. The map/directions metaphor is key here.

    In practice, programmers tend to first create the data definitions and then proceed to spend the vast majority of their time and effort creating and evolving the action statements. If you look at most programs, the vast majority of the lines are “action” lines.

    The action lines are endlessly complex, needing books to describe all the kinds of statements, the grammar, the available libraries and frameworks, etc. The data definitions are extremely simple. They first and foremost name a piece of data, and then (usually) give its type, which is one of a small selection of things like integer, character, and floating point (a number that has decimal digits). There are often some grouping and array options that allow you to put data items into a block (like address with street, town and state) and sets (like an array for days in a year).

    One of the peculiar elements of software language evolution is whether the data used in a program is defined in a single place or multiple places. You would think – correctly! – that the sensible choice is a single definition. That was the case for the early batch-oriented languages like COBOL, which has a shared copybook library of data definitions. A single definition was a key aspect of the 4-GL languages that fueled their high productivity.

    Then the DBMS grew as a standard part of the software toolkit; each DBMS has its own set of data definitions, called a “schema.” Schemas enable each piece of data to have a name, a data type and be part of a grouping (table). That’s pretty much it! Then software began to be developed in layers, like UI, server and database, each with its own data/schema definitions and language. Next came services and distributed applications, each with its own data definitions and often written in different languages. Each of these things need to “talk” with each other, passing and getting back data, with further definitions for the interfaces.

    The result of all this was an explosion of data definitions, with what amounts to the same data being defined multiple times in multiple languages and locations in a program.

    In terms of maps and directions, this is very much like having many different collections of directions, each of which has exactly and only the parts of the map those directions traverse. Insane!

    The BIG First Step towards Productivity and Quality

    The first big step towards sanity, with the nice side effect of productivity and quality, is to centralize all of a program’s data definitions in a single place. Eliminate the redundancy!

    Yes, it may take a bit of work. The central schema would be stored in a multi-part file in a standardized format, with selectors and generators for each program that shared the schema. Each sub-program (like a UI or service) would generally only use some of the program’s data, and would name the part it used in a header. A translator/generator would then grab the relevant subset of definitions and generate them in the format required for the language of the program – generally not a hard task, and one that in the future should be provided as a widely-available toolset.

    Why bother? Make your change in ONE place, and with no further work it’s deployed in ALL relevant places. Quality (no errors, no missing a place to change) and productivity (less work). You just have to bend your head around the "radical" thought that data can be defined outside of a program.

    If you're scratching your head and thinking that this approach doesn't fit into the object-oriented paradigm in which data definitions are an integral part of the code that works with them, i.e. a Class, you're right. Only by breaking this death-grip can we eliminate the horrible cancer of redundant data definitions that make bodies of O-O code so hard to write and change. That is the single biggest reason why O-O is bad — but there are more!

    The BIG Next Step towards Productivity and Quality

    Depending on your situation, this can be your first step.

    Data definitions, as you may know, are pretty sparse. There is a huge amount of information we know about data that we normally express in various languages, often in many places. When we put a field on a screen, we may:

    • Set permissions to make it not visible, read-only or editable.
    • If the field can be entered, it may be required or optional
    • Display a label for the field
    • Control the size and format of the field to handle things like selecting from a list of choices or entering a date
    • Check the input to make sure it’s valid, and display an error message if it isn’t
    • Fields may be grouped for display and be given a label, like an address

    Here's the core move: each one of the above bullet items — and more! — should be defined as attributes of the data/schema definition. In other words, these things shouldn't be arguments of functions or otherwise part of procedural code. They should be just like the Type attribute of a data definition is, an attribute of the data definition.

    This is just in the UI layer. Why not take what’s defined there and apply it as required at the server and database layers – surely you want the same error checking there as well, right?

    Another GIANT step forward

    Now we get to some fun stuff. You know all that rhetoric about “inheritance” you hear about in the object-oriented world? The stuff that sounds good but never much pans out? In schemas and data definitions, inheritance is simple and … it’s effective! It’s been implemented for a long time in the DBMS concept of domains, but it makes sense to greatly extend it and make it multi-level and multi-parent.

    You’ve gone to the trouble of defining the multi-field group of address. There may be variations that have lots in common, like billing and shipping address. Why define each kind of address from scratch? Why not define the common parts once and then say what’s unique about shipping and billing?

    Once you’re in the world of inheritance, you start getting some killer quality and productivity. Suppose it’s decades ago and the USPS has decided to add another 4 digits to the zip code. Bummer. If you’re in the enhanced schema world, you just go into the master definition, make the change, and voila! Every use of zip code is now updated.

    Schema updating with databases

    Every step you take down the road of centralized schema takes some work but delivers serious benefits. So let’s turn to database schema updates.

    Everyone who works with a database knows that updating the database schema is a process. Generally you try to make updates backwards compatible. It’s nearly always the case that the database schema change has to be applied to the test version of the database first. Then you update the programs that depend on the new or changed schema elements and test with the database. When it’s OK, you do the same to the production system, updating the production database first before releasing the code that uses it.

    Having a centralized schema that encompasses all programs and databases doesn’t change this, but makes it easier – fewer steps with fewer mistakes. First you make the change in the centralized schema. Then it’s a process of generating the data definitions first for the test systems (database and programs) and then to the production system. You may have made just a couple changes to the centralized schema, but because of inheritance and all the data definitions that are generated, you might end up with dozens of changes in your overall system – UI pages, back end services, API calls and definitions and the database schema. Making an omission or mistake on just one of the dozens of changes means a bug that has to be found and fixed.

    Conclusion

    I’ve only scratched the surface of a huge subject in this post. But in practice, it’s a hill you can climb. Each step yields benefits, and successive steps deliver increasingly large results in terms of productivity and quality. The overall picture should be clear: you are taking a wide variety of data definitions expressed in code in different languages and parts of a system and step by step, collapsing them into a small number of declarative, meta-data attributes of a centralized schema. A simple generator (compile-time or run-time) can turn the centralized information into what’s needed to make the system work.

    In doing this, you have removed a great deal of redundancy from your system. You’ve made it easier to change. While rarely looked on as a key thing to strive for, the fact that the vast majority of what we do to software is change it makes non-redundancy the most important measure of goodness that software can have.

    What I've described here are just the first steps up the mountain. Near the mountain's top, most of a program's functionality is defined by metadata!

    FWIW, the concept I'm explaining here is an OLD one. It's been around and been implemented to varying extents in many successful production systems. It's the core of climbing the tree of abstraction. When and to the extent it's been implemented, the productivity and quality gains have in fact been achieved. Ever hear of the RAILS framework in Ruby, implementing the DRY (Don't Repeat Yourself) concept? A limited version of the same idea. Apple's credit card runs on a system built on these principles today. This approach is practical and proven. But it's orthogonal to the general thoughts about software that are generally taught in Computer Science and practiced in mainstream organizations.

    This means that it's a super-power that software ninjas can use to program circles around the lumbering armies of mainstream software development organizations.

  • Software programming languages: the Declarative Core of Functional Languages

    What is most interesting about functional languages is that they strive to be declarative, instead of the imperative orientation of programming languages.

    In a prior post I described the long-standing impulse towards creating functional languages in software. Functional languages have been near the leading edge of software fashion since the beginning, while perpetually failing to enter the mainstream. There is nonetheless an insight at the core of functional languages which is highly valuable and probably has played a role in their continuing attractiveness to leading thinkers. When that insight is applied in the right situations in a good way it leads to tremendous practical and business value, and in fact defines a path of advancement for bodies of software written in traditional ways.

    This is one of those highly abstract, abstruse issues that seems far removed from practical values. While the subject is indeed abstract and abstruse, the practical implications are far-reaching and result in huge software and business value when intelligently applied.

    Declarative and Imperative

    A computer program is imperative. It consists of a series of machine language instructions that are executed, one after the other, by the computer's CPU (central processing unit). Each Instruction performs some little action. It may move a piece of data, perform a calculation, compare two pieces of data, jump to another instruction, etc. You can easily imagine yourself doing it. Pick up what's in front of you, take a step forward, if the number over there is greater than zero, jump to this other location, etc. It's tedious! But every program you write in any language ends up being implemented by imperative instructions of this kind. It's how computers work. Period.

    Computers operate on data.  Data is passive. Data can be created by a program, after which it's put someplace. The locations where data is held/stored are themselves passive; those locations are declared (named and described) as part of creating the imperative part of a computer program.

    Data is WHAT you've got; instructions are HOW you act. WHAT is declarative; HOW is imperative. WHAT is like a map; HOW is like an ordered set of directions for getting from point A to point B on a map.

    The push towards declarative

    From early in the use of computers, some people saw the incredible detail involved in spelling out the set of exacting instructions required to get the computer to do something. A single bit in the wrong position can cause a computer program to fail, or worse, arrive at the wrong results. This detailed directions approach to programming is wired into how computers work. Is there a better way?

    There is in fact no avoiding the imperative nature of the computer's CPU. As high level languages began to be invented that freed programmers from the tedious, error-prone  detail of programming in machine language, some people began to wonder if there were a way to write a low-level program, a program that of necessity would be imperative, that somehow enabled programs to be created in some higher level language that were declarative in nature.

    Some people who were involved with early computers, nearly all with a strong background in math, proceeded to create the declarative class of programming languages, the most distinctive members of which are functional programming languages as I described in an earlier post.

    The ongoing attempt to create functional languages and use them to solve the same problems for which imperative languages are used has proven to be a fruitless effort. But there are specific problem areas for which a declarative approach is well-suited and yields terrific, practical results — so long as the declarative approach is implemented by a program written in an imperative language, creating a workbench style of system for the declarations. The current fashion of "low code" and "no code" environments is an attempt to move in that direction. But I'd like to note that there's nothing new in those movements; they're just new names for things that have been done for decades.

    The Declarative approach wins: SQL

    DBMS's are ubiquitous. By far the dominant language for DBMS is SQL. Data is defined in a relational DBMS by a declarative schema, defined in DDL (data definition language). Data is operated on by a few different kinds of statements such as SELECT and INSERT.

    SELECT certainly sounds like a normal imperative keyword in any language, like COBOL's COMPUTE statement. But it's not. It's declarative. A SELECT statement defines WHAT data you want to select from a particular database, but says nothing about HOW to get it.

    This is one of the cornerstones of value in a relational DBMS system. A SELECT statement can be complicated, referencing columns in multiple rows of various tables joined in various ways. The process of getting the selected data from the database can be tricky. Without query optimization, a key aspect of a DBMS, a query could take thousands of times longer than a modern DBMS that implements query optimization will take. Furthermore, table definitions can be altered and augmented, and so long as the data referenced in the query still exists, the SELECT statement will continue to do its job.

    If all you're doing is grabbing a row from a table (like a record from an indexed file), SQL is nothing but a bunch of overhead, and you'd be better off with simple 3-GL Read statements. But the second things get complicated, your program will require many fewer lines of complex code while being easier to write and maintain if you have SQL at your disposal. A win for the declarative approach to data access, which is why, decades after it was created, SQL is in the mainstream.

    The Declarative approach wins: Excel

    I don't know many programmers who use Excel. Too bad for them; it's a really useful tool for many purposes, as its widespread continuing use makes clear.

    Excel is a normal executable program written in an imperative language, but it implements a declarative approach to working with data. Studying Excel is a good way to understand and appreciate the paradigm.

    An Excel worksheet is two dimensional matrix of values (cells). A cell can be empty or have a value of any kind (text, number, currency, etc.) entered. What's important in this context is that you can put a formula into a cell that defines its value. The formula can reference other cells, individually or by range, absolutely or relatively. A simple formula could be the sum of the values in the cells above the cell with the formula. It could be arbitrarily complex. It can have conditionals (if-then-else). Going beyond formulas, you can turn ranges of cells into a wide variety of graphs.

    If you're not familiar with them, you should look at Pivot tables, and when you've absorbed them, move on to the optimization libraries that are built in, with even better ones available from third parties. Pivot tables enable you define complex summaries and extractions of ranges of cells. For example, I have an Excel worksheet in which I list each day of the year and the place where I am that day. A simple Pivot table gives me the total of days spent in each location for the year, something that simple formulas could not compute.

    The key thing here is that even though there are computations, tests and complex operations taking place, it's all done declaratively. There is no ordering or flow of control. If your spreadsheet is simple, Excel updates sums (for example) when you enter or change a value in a cell. For more complex ones, you just click re-calc, and Excel figures out all the formula dependencies and evaluates them in the right order. This makes Excel a quicker way to get results than programming in any imperative language, assuming the problem you have fits the Excel paradigm.

    The Declarative approach wins: React.js

    One of the most widely-used frameworks for building UI's is React.js. Last time I looked, the header page of react.js included this:

    1

    It says right out that it's declarative! And that makes it easy! I have found a number of places (example, example) that have nice explanations of how it works and why it's good.

    The Declarative approach wins: Compilers

    The best computer-related course I took in college by far was a graduate course on the theory and construction of compilers. The approach I was taught all those decades ago remains the main method for compiler construction. First you have a lexical parser to turn the text of the program into a string of tokens. Then you have a grammar parser to turn the tokens into a structured graph of semantic objects. Then you have a code generator to turn the objects into an executable program.

    The key insight is that each stage in this process is driven by a rich collection of meta-data. The meta-data contains all the details of the input language and its grammar and the output target. A good compiler is really a compiler-compiler, i.e., the imperative code that reads and acts on the lexical definitions, the grammar and the code generation tables. The beauty is that you write the compiler-compiler just once. If you want it to work on a new language, you give it the grammar of the language without changing any of the imperative code in the compiler-compiler! If you want to generate code for a new computer for a language you've already got, all you do is update the code generation tables! Once you've got such a compiler, you can write the compiler in its own language, at the start "boot-strapping" it.

    While they didn't exist at the time I took the course, tools to perform these functions, YACC (Yet Another Compiler Compiler) and LEX, were built at Bell Labs by one of the groups of pioneers who created Unix and the C language. I didn't have those tools but used the concepts to build the FORTRAN compiler I wrote in 1973 while at my first post-college job. Since the only tool I had to use was assembler language, taking the meta-data, compiler-compiler approach saved me huge amounts and time and effort compared to hard-coding the compiler without meta-data.

    This is meta-data at its best.

    The Declarative spectrum

    Excel and SQL are 100% all-in on the declarative approach. But it turns out that it doesn't have to be all one way or the other. If you're attacking an appropriate problem domain, you can start with just programming it imperatively, and then as you understand the domain, introduce an increasing amount of declaration into your program, by defining and using increasing amounts of meta-data. This is exactly what I have described as climbing up the tree of abstraction. Each piece of declarative meta-data you introduce reduces the size of the imperative program, and moves information that is likely to be changed or customized into bug-free, easily-changed meta-data.

    Conclusion

    Functional languages as a total alternative to imperative languages will perpetually be attractive to some programmers, particularly those of a math theory bent. Except in highly limited situations, it won't happen. No one is going to write an operating system in a functional language.

    Nonetheless, the declarative approach with its emphasis on declaring facts, attributes and relationships is the powerful core of the functional approach, and can bring simplicity and power to otherwise impossibly complicated imperative programs.

  • The Map for Building Optimal Software

    So you want to build optimal software, do you? What’s “optimal” in software, anyway? Is it building the software that’s needed quickly and well, with no bugs? Is it building software that’s easy to enhance, adding new features – including ones you never thought of – with minimal fuss and bother? Is it building software that’s easy to scale endlessly with little trouble? How about all of the above, all at once? Yup, that would be optimal, sure enough.

    I’ve described in general terms about optimal software. Here’s a map, a specific example, of how to get there.

    Going to new places

    Let’s think about going places we haven’t been before, or taking routes that are new to us. What are the options for figuring out how to do it?

    If you’re talking with someone who is familiar with the place to which you want to go, you might ask them how to get there. If there are more than a couple of turns, you might try to simplify the directions or write them down. If you’re on your own, you might consult a map. If you’re computer-oriented you might use one of the on-line mapping programs. If you’re driving in a car, you might use a navigation system.

    No matter what you do, you’ll end up (let’s say) getting in a car and driving. When you drive, one way or another, you’ll be following directions and making choices of turns. Nothing happens unless you drive and to drive you need directions.

    It’s obvious that while you need directions, a fixed set of directions by itself is adequate only if nothing goes wrong – there are no accidents, no construction detours, no recent changes to the roads, etc. The minute there is any problem, you need a map or a live, map-driven navigation system.

    What you do when you look at a map, or what a navigation system does, is execute a generic direction-creating algorithm. Either your brain or a computer looks at the map, sees where you are and where you want to go, and figures out the sequence of roads and turns to get you there. Is there an obstruction? The nav system doesn't care — it just computes a fresh set of directions for you, based on where you are now.

    Here's an example. It shows two different routes for me to drive from my home to the wonderfully close source of some of the world's best home-made ice cream: Denville Dairy.

    163 Cedar Lake E to Denville Dairy Broadway Denville NJ Google Maps

    A direction-creating algorithm is a relatively small amount of code that is independent of the map that it uses. You may need to enhance it when different kinds of things are introduced into the map (for example, you decide to support avoiding toll roads or taking ferries), but once you’ve added the capability, it should automatically apply to all the maps you have.

    So “where” is the application functionality here? The ability to create directions is in a small amount of abstract code. This code accepts a reference to a map, a starting point, an ending point and perhaps some parameters, like whether to minimize distance or time or avoid toll roads. The actual directions themselves are drawn from the substance of the map. In this case,

    • what you’ve got is the map;
    • what you do to what you’ve got is create directions.

    Isn’t it interesting that most of the time and effort is creating the set of maps, without which there are no directions? And isn’t it interesting that the maps are data?

    Maps are:

    • Extensive
    • Subject to frequent updating and change

    An error in a map is bad, but should not “crash the program” – it should just lead to bad results, and should be easily fixed, without going into debuggers, single-stepping and other geeky things.

    The direction-creating program is:

    • Short
    • Infrequently updated
    • Unaffected by additional maps
    • Unaffected by changes to maps

    Best of all, the direction-creating program is unaffected by the driver not following the directions – the program just builds new directions to the destination from wherever the car happens to be.

    Errors in this program are crippling, but it’s likely that errors will be found early in the process, after which it won’t change much.

    Maps and writing optimal code

    So what does this have to do with writing code? Lots.

    A map (meta-data) is an efficient, compact, re-purpose-able representation of what the user cares about. It is truly Occamal, without redundancy. Each piece of knowledge you have about the world is represented in the map exactly once.

    A reasonably well-written direction-generating program is an efficient, compact representation of strategies for generating routes through the entire universe of possible maps. Each algorithm or strategy for generating directions should be represented in the program exactly once.

    Combined, the map and the direction-generating program are a pretty good template/metaphor for an Occamal program. Here’s the pattern:

    • You should make everything you possibly can into a “map” (metadata).
    • Abstract actions that cannot be reduced to metadata should be coded once, and should be “driven” to the largest extent possible by the metadata.

    The map to building optimal software is a simple idea: build the smallest amount of code you possibly can, and put as much information as possible into passive, declarative meta-data. That will get you to your destination safely and in the shortest amount of time without breaking laws.

  • Lessons for Better Software from Washing Machine Design

    The power of Occam-optimality, a.k.a. occamality, can be difficult to appreciate if your head is awash in myriad software buzzwords, and explaining it in software terms can make understanding it challenging to anyone unfamiliar with those terms. So it makes sense at this point to explain the core concept in a different context.

    Let’s think about the design of physical things, the kind of things that require discrete manufacturing, have bills of material, and so on, for example a washing machine.

    Capture

    If every section of the washing machine were designed by someone different, each designer may call for a particular screw here, or a particular composition of aluminum there, making his best judgment on what is the best to use. When you broke down the bill of material into a consolidated parts list, you may see two of this kind of screw, one of that kind, and a couple of yet another kind. It may be that the various screws need to be sourced from multiple suppliers. As far as the designer is concerned, the various screws are completely legitimate requirements. Each was chosen to have everything required to get the job done, but only to get the job done. The designer is probably proud, with good justification, of the care and effort he expended in specifying exactly the right screw for the job. When the design is complete, the washing machine will have a list of unique parts and designs and/or instructions for how to construct and/or assemble them.

    While the designers may be proud, everyone else’s jobs have been made more difficult and the overall expense of building washing machines has been increased. If instead the designers had been motivated to get together and find a way to use a single screw type that could be sourced from a single supplier, more screws would be bought from a single supplier in greater quantities, reducing both unit and shipping costs, and enabling all assembly points to be supplied from a single pool of screws. Only one type of screw driver would be needed, which would also help keep all the screw drivers in working order during manufacturing. Even the repair manual would be slightly easier to write and would be slightly shorter, and repair people would need only a single type of screw and a single type of driver, which would increase the odds that they would have what they needed to do a repair on a single visit. If the design were done in this way, the overall design would be virtually unchanged, but the parts list would be shorter – there would be a smaller number of unique parts, with each used more frequently.

    How about the designers? Instead of starting from scratch when they needed a screw, picking the “right” screw from the universe of available screws, which is large, they would agree up front on a screw that would meet all their anticipated needs closely enough, and then, every time they needed a screw, they would just check to make sure the pre-chosen screw was adequate to the job. This is a simpler job than selecting the “best” screw from a large selection, because you’re just applying a short list of disqualifying characteristics to the pre-selected screw. Moreover, you’re arranging your design as you go along (for example, the thickness of the plates) to match up with the qualities of the screw, making the selected screw very likely to be OK in each case. So the “shortest” (without going too crazy) parts list, the one with the smallest number of unique parts, clearly is the best design.

    Notice that the original design, the one with lots of types of screw types, would be defended by its designers as the “best” design, because they understood that the important thing for them to do was to create the “best” solution for each design problem they faced, considered in isolation. Let’s assume they did. But once the designers look at the whole problem, and try to pick the best design for the entire lifecycle of the washing machine, including sourcing, manufacturing and repair, it’s obvious that the different screw selections were incidental aspects of the washing machine design. The design could just as well have been embodied in one that had fewer screw types, and as we now understand, having fewer parts is better.

    Suppose there are a couple of established washing machine manufacturers. Suppose you wanted to enter the market with a superior product. If things worked the way they normally do, the earlier products are probably far from occamal. That means it's possible to design a washing machine with just as high quality as the ones on the market, but less expensive to build and easier to service. Everyone would like your product better than the existing ones. Cheaper to build, cheaper and easier to fix in a single visit; what's not to like? This means that, in addition to being a core design principle, occamality helps us understand and predict the evolution of products in a market, all other things being equal.

    In a virtually identical way, a program that has the smallest (within reason) number of unique “parts” is the best, for reasons that are very similar to the way that the best physical design has the shortest list of parts. It definitely helps the designers, and it helps everyone else even more. Just like with a washing machine, a program can be built, from the requirements all the way through, to have lots of unique “parts” and custom things, each of which may make sense when considered in isolation. But when you look at the whole software lifecycle, taking a view over the entire program and everything that will eventually be done to it, including later maintenance, change and enhancement, it becomes clear that designing it to contain the smallest number of “unique parts,” so that every “program concept” is defined in exactly one place, yields the best design.

    The parable of the screw can help you, your organization and your users avoid being screwed during software design. You’ll get there if you keep this principle in mind: always use the smallest number of unique “parts” that you can, and be willing to adjust your design to make the number even smaller. This may sound easy to you, or it may sound like an essentially trivial exercise in neatness. It is neither. It can be challenging in practice, and the implications of it are actually profound and far-reaching.

  • William of Occam is the Inventor of the Method for Building Optimal Software

    Lots of people would like the credit for establishing, once and for all, the core, immutable principle of optimal software design/architecture – the method for measuring which, among an infinite number of embodiments of software requirements, is provably the best among them. I am one of the crowd that would love to take credit! Alas, I and my competitors have been beaten to the punch, by a mere 700 years or so, by a guy named William of Occam.

    As it turns out, the software crowd is, as usual, late to the party: loads of luminaries in other fields have recognized William’s achievement, viz., Occam’s Razor, for the indispensable concept that it is. Various software folks have been nipping around the edges of it, for example with the DRY (don’t repeat yourself) principle that’s gotten some attention. But they haven’t grasped the fundamental, touches-nearly-everything, deep nature of the Razor. Too bad! Crappy software continues to be churned out by the ton by people who are enthralled by false idols, convinced they’re on the righteous path toward software excellence, when in fact they’re doing something else, words for which should not be used in polite company.

    William of Occam (ca. 1285 to 1349) was an English logician and Franciscan friar.

    330px-William_of_Ockham_-_Logica_1341

    He is credited with formulating a principle that is already widely applied to various aspects of computer systems. In those areas of computing to which it has been applied, it reigns supreme – it unquestionably supplies the most fundamental, relevant and optimal understanding and solutions to the relevant problems, so much so that the people involved don’t think about it or question it.

    However, there are large areas of computing to which Occam’s razor has not been applied. Worse, it is not even one of the candidates under consideration. As a result, those aspects of computing are fractured, inefficient, unpredictable, and driven by fashion and politics. Even while the practitioners call themselves "computer scientists," which is something that none of them are.

    Everyone involved knows that the whole process of specifying, designing, building, testing and supporting software is hopelessly inefficient, unpredictable and error-prone. The leading methodology that concentrates on the process of building software, “project management,” is theoretically bankrupt and in any case has an empirical track record of failure. In terms of the content of good software, there are fierce battles among competing approaches, none of which is anything but a collection of unsupported and unfounded assertions, and which in practice don’t contribute to building good software.

    Occam’s razor leads to the principles on which good software may be built, and supplies a single simple, widely applicable theme that, when applied, cuts away (as a razor should) all the inefficiency, inapplicability and generally what we don’t like about software. Once the principle is understood and widely applied, I expect it will become the un-discussed and undisputed standard for how software is built, just as it has in the other areas of computing to which it has been applied.

    What is this amazing standard? It’s simple. I mean literally simple. That’s what Occam’s Razor is all about.

    Occam’s razor is:

        Entia non sunt multiplicanda praeter necessitatem.

        No more things should be presumed to exist than are absolutely necessary.

    Occam’s razor applied to software would be:

        No more software entities should be created than are absolutely necessary.

    Another way of expressing this is that all redundancy, of any kind and in any way, should be eliminated from a piece of software. This often requires writing imperative code to implement any and all abstract actions, and to create largely declarative meta-data to specify everything that is specific to an application. The meta-data, of course, should also have no redundancy; this is normally achieved by use of multiple inheritance with override. See this for more.

    To be extremely brief, the reason why eliminating redundancy is good is that practically everything that happens to a piece of software is that it is enhanced, altered, bug-fixed or otherwise modified. Creation happens just once; everything that follows is growth and change. Implementing change of any kind to a program takes the least effort with the least risk if everything is defined in exactly one place – once you’ve found the place, you make the change and you’re done. Among the infinite universe of programs solving the same problem that work, the speed, cost and risk of change is the overriding virtue, both in technical and business terms.

    Thanks to William of Occam, I can’t claim to have invented the core principle of software, and the way to express the measure of goodness for software. William has me beat by over 700 years. Thanks a lot, Bill! Lots of people have applied Occam’s Razor to lots of tough things with optimal success, even in things closely related to software, such as information theory. Way too late again. About all I can do is mess with his name and make up a term that expresses goodness in software. Here it is: I propose that a piece of software can be measured by its “Occamality.” The more “Occamal” it is, the better it is. And I propose that Occamality is strictly correlated with the extent to which there is no redundancy of any kind in a program, which is almost always strictly correlated with the program being at the high end of the hierarchy of abstraction.

  • How to Pay Down Technical Debt

    When we look at a body of software, how do we judge it? Clearly, there are strong opinions on this subject – but there is no general consensus on the standards by which a body of code should be judged! This is shocking and unacceptable!

    For most normal people, i.e., people who aren’t software developers, software is “good” if is doesn’t crash and pretty much does what you want most of the time. What more could anyone want? Well, how about changing the software. This is one of the BIG things that makes software different from pretty much anything else in our experience. How often do you try to make changes to your car in the same sense that you change software? The answer is never.

    There are standards for nearly everything in life, certainly for things that are built by engineers and other technical people. How is it that, not only are there NO standards for what constitutes “good” software, but the fact that there are no such standards is not decried, and no one appears to be sheepish or unapologetic about admitting it – because they’re never asked to admit it. The subject never comes up – not only in “polite company” but even in “impolite company,” i.e., among programmers talking quietly among themselves.

    “Ya know,” someone might say, “we tell them there’s technical debt. Sounds fancy, right? Anyone ever been asked what it means? They seem to think that WE know what it means, and that’s good enough for them. I wonder how much longer we’re going to be able to get away with it?” Someone else says, “Forever, man. This has been going on since before we were BORN, ya know? Unless word somehow gets out, and how could it, we’ll die without anyone being the wiser. Good thing.”

    I wish programmers did have quiet dialog among themselves like the one I made up above. But they don’t, at least that I’ve ever heard. There’s nearly always someone who has strong opinions about what software should look like internally, and of course the current body of software doesn’t measure up. So there is agitation and there are threats about the awful future consequences of failing to rein in the spreading disease – soon! RIGHT AWAY!! – unless the transformation is authorized.

    Then the re-writing death march starts off, with various sections of the code slated for repair, renewal or even flat-out re-write. What is the target? It varies. Favorite criticisms include that the code is spaghetti, it’s a monolith, it’s not structured according to some set of object-oriented principles, it’s written in some obviously inferior language, there aren’t components, there are no services, micro or otherwise, and on and on. The solution is obvious to the converted, and value to be achieved by the solution is so obvious that it’s hardly worth stating.

    Is there any value to be gained by the clean-up of “technical debt?” Maybe. But often, nothing really changes except a lot of time has been spent with no improvement to the business. No one can point to a proof or a study or a clear historic pattern demonstrating the effort is worth making.

    This is a shame, because there is a clear pattern of success for a method of organizing and/or re-organizing code for business benefit.

    Let’s start from the business benefit. Of all the ways a piece of software could be written that performs the same function, which is the best in business terms? I’m assuming here roughly the same level of performance, etc.

    Tick, tick, tick… Think about it!

    When you come right down to it, no one cares about today’s software doing today’s things for today’s customers – so long as it works and performs reasonably well. What matters is simple, and everyone knows it: it’s the next request by an important existing customer, and more important, the next potential customer who is balking about things they need that aren’t there, the time, cost and risk to implement the software, etc. In other words, what matters about today’s software is TOMORROW’s changes, whatever they may be – we can suspect, but we won’t know until we know them!

    Software people have talked forever about things like “architecting for change” and various fantasies that mostly go poof when the harsh wind of reality rushes at them.

    There are exactly two rarely-discussed aspects of code that position it optimally for the widest possible range of unanticipated changes and the needs of installation:

    • A minimum of redundancy in the code – anything you want to change can be changed by going to one place
    • To the largest extent possible, particular things the application does are defined in meta-data instead of code, with the result that the code is smaller and more abstract with minimal redundancy — and that any change can more likely be made by changing meta-data, which is quicker and safer than changing code.

    That’s it! See this for more.

  • How to Build Applications that can be Changed Quickly

    Nearly everyone who talks about fast and reliable application building seems to talk about the process (Agile etc.) and/or a couple architectural things (object-oriented, micro-services, etc.). From what I can tell, these haven’t led to any break-throughs in application velocity or predictability. However, it’s clear that there are methods for high-velocity application growth. A wide variety of small, motivated groups have re-invented them over the years. I have had the privilege of observing these groups and their methods, and seen the impact when they add or drop velocity-aiding methods.

    Here is an overview of some of the methods I’ve seen used. The improvements can range from cutting time in half to whole number factors, things like turning weeks into days or in some cases hours.

    Forget requirements, focus on change

    The normal requirements process starts with the often non-technical people who have a need. The need can be expressed directly, or through refusal to buy/use a product, which gets the attention of sales and marketing people. There then ensues a process of requirements gathering and refinement. Wire-frame UI’s may be involved. Finally, the requirements are ready for engineering, and away we go on hopefully not too many “sprints.” This is an “outside-in” approach.

    Speed is achieved by taking an “inside-out” approach – instead of focusing almost exclusively on requirements, center your efforts on the code and what it can do or easily be changed to do today to go in the direction of the requirements. Take steps to change and add to the existing code to meet the unmet needs. Do this in small steps, getting feedback at each step. When you’re done, go back and “clean up” the code to eliminate redundancies as much as possible, to make the next change as easy as possible.

    What this does is ground the changes in the existing code, resulting in the changes and additions being smaller than they otherwise would be. As a additional benefit, it usually results in a greater uniformity of both the code and the interfaces (UI and API) to the code.

    Minimize the customization and optimization of the code and UI

    When product people think about a new process that has UI, they usually try to do a good job, understanding all the details and nuances involved. The result of this sophistication is often requirements that are finely adapted to the particular function being performed. What’s wrong with this, you may wonder? One problem is that this approach ends up taking longer to built and results in more code. More important is the users – the less someone has to learn to use a new feature, the more quickly and easily they’ll be able to use it. Familiarity is incredibly important. Which means, among other things, that literally using rarely changing templates to generate interfaces will both make any new interface element recognizable and easy to learn, but also lead to the least amount of code changes to get the job done.

    This means that the fineness and sophistication of product people can be a problem. The more time they spend on a subject and the more they think and reflect on it, the worse the temptation is likely to get to do something other than just crank out another instance of a familiar pattern.  Resist!

    Avoid planning and designing for change and future requirements

    When starting a project, accomplished software professionals often insist on a period devoted to design or architecture, in which the overall technical approach to the project is determined. When things go wrong later, inadequate or insufficient up-front design is often cited as the reason. “We’re gong to insist on being able to do it right next time, so this won’t happen again.” But all too often, projects slip and slide regardless of any up-front design work.

    Generally, up-front design time is actually counter-productive. The reasons are simple: you spend time “designing” when you could have been building; the result of the design is often building for anticipated change, which takes time; when the anticipated change doesn’t happen and unanticipated change arrives, the code added because of the design almost always makes things harder.

    None of this happens when you build incrementally and strive toward uniformity, eliminating code redundancies and moving functionality from code into meta-data.

    Wartime software

    The methods I have described as wartime software maximize speed and efficiency of building new software that meets business needs. See here for details. Generally, the methods include:

    • Eliminate overhead
    • Avoid things resembling “project management”
    • Optimize for speed instead of expectations
    • Use production environments for testing instead of sand boxes
    • Use comparison QA; avoid test-driven development, unit testing and test scripts
    • Minimize the use of documents, meetings and other internal overhead
    • The main architectural value should be speed and scale
    • Avoid technical fashions

    Move up the mountain of abstraction

    When you think about code and methods, the main values should be the elimination of redundancy in both data definitions and code, and migrating functionality from code into meta-data. See this and this.

    The other methods described here are important for moving quickly, and are complementary to this one. But moving up Abstraction Mountain is the most powerful of the methods, often yielding qualitative improvements.

    Conclusion

    This is a short post, and the methods are described briefly. In a different world, each topic here would be at least a course in the Computer Science curriculum, and in a couple cases multiple books. I don't claim anything I've said here is fully described. But every single item has been proven in practice by many groups over many years, in different ways, always resulting in better quality software that meets business needs being built dramatically more quickly than other methods could possibly have achieved.

    My goal here is simply to provide a short list of the main methods for fast development in a single place.

  • The Progression of Abstraction in Software Applications

    This post describes a little-known concept for understanding and creating software architecture that small groups use to defeat large, powerful incumbents and nimble competitors. It is one of a small number of powerful, repeating patterns that help us understand and predict the evolution of software. Understanding these patterns can help entrepreneurs direct their efforts; if they do it well, they greatly enhance their chances of success. Understanding the patterns can also help investors choose to invest in groups that are walking a path to success.

    Evolution of Applications Towards Abstraction on a Platform

    One of these patterns is the stages that applications naturally evolve through on a technology platform. Each step or stage brings a big increase in the power of the software, decreasing the effort and increasing the speed and effectiveness of being deployed to meet customer needs.

    A category of software applications may well get “stuck” at a particular stage for a long time, sometimes even decades. During this time, the software may appear to move forward, and of course the marketing people and managers will always put things in the best possible light. But it’s always vulnerable to being supplanted by a next-stage version of the functionality.

    While there aren’t clear lines of delineation between the stages, it’s nonetheless useful to understand them roughly as:

    • Prototype. A hard-coded body of code.
    • Custom Application. Does a job reliably, but most changes require changing source code.
    • Basic Product. The code now has parameters, maybe user exits and API’s. Real-life implementations tend to require extensive professional services, and the cost of upgrading to new versions tends to be high.
    • Parameterized Product. The level of parameterization is high, with interface layers to many things outside the core code, so that many implementations can be done without changing source code. There may be some meta-data or editable rules.
    • Workbench Product. A large portion of the product’s functionality has migrated from code to editable meta-data, so that extensive UI, workflow, interface and functionality changes can be accomplished via some form of workbench, which could be just a text editor. The key is that the details of application functionality are expressed as editable data, meta-data, instead of code. Nonetheless, all fundamental capabilities are expressed in highly abstract code.

    As a body of code goes through this sequence of abstraction, it is increasingly able to meet the needs of new customers and changing needs of existing customers, with decreasing amounts of effort, risk and changes to source code. At the same time, the more abstract a program, the more functionality is expressed as data that is not part of the software itself, a.k.a. meta-data, and the more the software implements generic capabilities, as directed by the meta-data.

    The pattern applies both to individual bodies of code and to collections of them. It applies to code built internally for an organization and to code that is sold as a product in any way.

    I defined the stages above as a convenience; in reality, the categories are rarely hard-and-fast. A body of code could be given a big transformation and leap along the spectrum, or it could take a long series of small steps. One body of code could remain stuck with little change in abstraction, while other bodies of code doing similar things could be ahead, or progress rapidly towards abstraction.

    The Driver of Abstraction Evolution

    In biological nature, competitive pressures and external change appear to drive evolutionary changes. Similarly, when we look at categories of software, if little changes to make the software change, it doesn’t change – why take the trouble and expense to change software that meets your needs or the needs of your customers?

    In reality, someone always seems to want changes to an application. A prospective user would gladly use the software if it did this, that or the other thing. A current user complains about something – it’s too slow, too hard to use, too complicated, or it just doesn’t work for X, Y or Z. How often does a piece of software not have a “roadmap?” If it doesn’t, it’s probably slated for retirement before long. Brand-new software is rare. The vast majority of software effort goes into making changes to an existing piece of software.

    How much time and effort is needed to change a particular body of software? That is the key touch-point between techies and business people. This is the point at which the level of abstraction of the application comes into play. Regardless of the level of abstraction of the application, the “change” required either has been anticipated and provided for or it has not.

    • If the change has been anticipated, the code can already do the kind of thing the user wants – but not the particular thing. Doing the particular thing requires that a parameter be defined, a configuration file changed, a template created or altered, workflow or rule changes made, or something similar that is NOT part of the program’s source code. This means that the requirement can be met quickly, with little chance of error.
    • If the change has not been anticipated, then source code has to be changed in some way to make the change. The changes required may be simple and localized, or complex and extensive – but the source code is changed and a new version of the program is created.

    This is what the level of abstraction of a software application is all about: the more abstracted the application, the more ways it can be changed without altering the source code and making a new version of the program.

    This is the fundamental driver of applications towards increasing abstraction: as changes have to be made to the application, at some point the technical people may decide to make the change easier to make in the future, and create the appropriate abstraction for the kind of change. This may happen repeatedly. As more complex changes are required, the program may just get more complicated and more expensive to make further changes, or more sophisticated abstractions may be introduced.

    While historically it appears that outside forces drive applications towards increasing abstraction, smart programmers can understand the techniques of abstraction and build an application that is appropriately abstract from the beginning. Similarly, the abstracting methods can be applied by smart programmers to existing bodies of code to transform them, just because it's a way to build better code and meet ever-evolving business requirements.

    Conclusion

    The hierarchy of abstraction in software is one of the most important concepts for understanding a specific piece of software, or a group of related software. Over time, software tends to become more abstract because of competitive and business pressures, or because of smart programmers working to make things better. The more abstract a piece of software is, the more likely that it can respond to business and user demands without modifications to the source code itself, i..e., quickly and with low risk.

    The hierarchy of abstraction is certainly a valuable way of understanding the history of software. But it is more valuable as a framework for understanding a given piece of software, and the way to evolve that software to becoming increasingly valuable. It is most valuable to software developers as a framework for understanding software, and helping them to direct their efforts to get the greatest possible business impact with the smallest amount of time and effort.

  • How to Evaluate Programming Languages

    Programmers say language A is "better" than language B. Or, to avoid giving offense, they'll say they "like" language A. Sometimes they get passionate, and get their colleagues to program in their new favorite language.

    This doesn't happen often; inertia usually wins. When a change is made, it's usually passion and energy that win the day. If one person cares a WHOLE LOT, and everyone else is, "whatever," then the new language happens.

    Sometimes a know-nothing manager (sorry, I'm repeating myself) comes along and asks the justification for the change. The leading argument is normally "the new language is better." In response to the obvious "how is it better?" people try to get away with "It's just better!" If the manager hangs tough and demands rationality, the passionate programmer may lose his cool and insist that the new language is "more productive." This of course is dangerous, because the rational manager (or have I just defined the empty set?) should reply, "OK, I'll measure the results and we'll see." Mr. Passion has now gotten his way, but has screwed over everyone. But it usually doesn't matter — who measures "programmer productivity" anyway?

    But seriously, how should we measure degrees of goodness in programming languages? If there's a common set of yardsticks, I haven't encountered them yet.

    The Ability of the Programmer

    First, let's handle the most obvious issue: the skill of the programmer. Edgar Allan Poe had a really primitive writing tool: pen (not even ball-point!) and paper. But he still managed to write circles around millions of would-be writers equipped with the best word processing programs that technology has to offer. I've dealt with this issue in detail before, so let's accept the qualification "assuming the programmer is equally skilled and experienced in all cases."

    Dimensions of Goodness

    Programmers confined to one region of programming (i.e., most programmers) don't often encounter this, but there are multiple dimensions of goodness, and they apply quite differently to different programming demands.

    Suppose you're a big fan of object-orientation. Now it's time to write a device driver. Will you judge the goodness of the driver based on the extent to which it's object-oriented? Only if you're totally blindered and stupid. In a driver you want high performance and effective handling of exception conditions. Period. That is the most important dimension of goodness. Of all the programs that meet that condition, you could then rank them on other dimensions, for example readability of the code.

    Given that, what's the best language for writing the driver? Our fan of object orientation may love the fact that his favorite language can't do pointer arithmetic and is grudgingly willing to admit that, yes, garbage collection does happen, but with today's fast processors, who cares?

    Sorry, dude, you're importing application-centric thinking into the world of systems software. Doesn't work, terrible idea.

    It isn't just drivers. There is a universe of programs for which the main dimensions of goodness are the efficiency of resource utilization. For example, sort algorithms are valued on both performance and space utilization (less is better). There is a whole wonderful universe of work devoted to this subject, and Knuth is near the center of that universe.ArtOfComputerProgramming

    Consistent with that thinking, Knuth made up his own assembler language, and wrote programs in it to illustrate his algorithms. Knuth clearly felt that minimum space utilization and maximum performance were the primary dimensions of goodness.

    The largest Dimension of Goodness

    While there are other important special cases, the dimension of goodness most frequently relevant is hard to state simply, but is simple common sense:

    The easier it is to create and modify programs, quickly and accurately, the more goodness there is.

    • Create. Doesn't happen much, but still important.
    • Modify. The most frequent and important act by far.
    • Quickly. All other things being equal, less effort and fewer steps is better.
    • Accurately. Anything that tempts us to error is bad, anything that helps us to accuracy is good.

    That, I propose, is (in most cases) the most important measure of goodness we should use.

    Theoretical Answer

    Someday I'll get around to publishing my book on Occamality, which asks and answers the question "how good is a computer program?" Until then, here is a super-short summary: among all equally accurate expressions of a given operational requirement, the best one has exactly one place where any given semanic entity is expressed, so that for any given thing you want to change, you need only go to one place to accomplish the change. In other words, the least redundancy. What Shannon's Law is for communications channels, Occamality is for programs. Given the same logic expressed in different languages, the language that enables the least redundancy is the best, the most Occamal.

    Historical Answer

    By studying a wide variety of programs and programming languages in many fields over many years, it's possible to discern a trend. There is a slow but clear trend towards Occamality, which demonstrates what it is and how it's best expressed. The trend is the result of simple pressures of time and money.

    You write a program. Sombody comes along and wants something different, so you change the program. Other people come along and want changes. You get tired of modifying the program, you see that the changes they want aren't to different, so you create parameters that people can change to their heart's content. The parameters grow and multiply, until you've got loads of them, but at least people feel like they're in control and aren't bugging you for changes. Parameters rule.

    Then someone wants some real action-like thing just their way. You throw up your hands, give them the source code, and tell them to have fun. Maybe they succeed, maybe not. But eventually, they want your enhancements in their special crappy version of your nice program, and it's a pain. It happens again. You get sick of it, analyze the places they wanted the changes, and make official "user exits." Now they can kill themselves customizing, and it all happens outside your code. Phew.

    Things keep evolving, and the use of user exits explode. Idiots keep writing them so that the whole darn system crashes, or screws up the data. At least with parameters, nothing really awful can happen. The light bulb comes on. What if I could have something like parameters (it's data, it can't crash) that could do anything anyone's wanted to do with a user exit? In other words, what if everything my application could do was expressed in really powerful, declarative parameters? Hmmm. The users would be out of my hair for-like-ever.

    What I've just described is how a new programming "level" emerges historically. This is what led to operating systems, except that applications are one giant user exit. UNIX is chock full of things like this. This history is the history of SQL in a nutshell — a powerful, does-everything system at the level of declarative, user-exit-like parameters!

    The Answer

    In general, the more Occamal the language (and its use), the better it is. More specifically, given a set of languages, the best one has

    • semantics that are close to the problem domain
    • features that let you eliminate redundancy
    • a declarative approach (rather than an imperative one)

    Let's go through each of these.

    Problem Domain Semantics

    A great example of a such a language is the unix utility AWK. It's a language whose purpose is to parse and process strings. Period. You want an accounting system, don't use AWK. You want to generate cute-looking web pages, don't use AWK. But if you've got a stream of text that needs processing, AWK is your friend.

    From the enterprise space, ABAP is an interesting example. While it's now the prime language for writing SAP applications, ABAP was originally Allgemeiner Berichts-Aufbereitungs-Prozessor, German for "general report creation processor." In other words, they saw the endless varieties of reporting and created a language for it; then having seen the vast utility of putting customization power in the hands of users, generalized it.

    Features that let you eliminate redundancy

    This is what subroutines, classes and inheritance are supposed to be all about. And they can help. But more often, creating another program "level" is the most compelling solution, i.e., writing everything that the program might want to do in common, efficient ways, and having the application-specific "language" just select and arrange the base capabilities. This is old news. Most primitively, it's a subroutine library, something that's been around since FORTRAN days. But there's an important, incredibly powerful trick here. In FORTRAN (and in pretty much all classically-organized subroutine libraries), the library sits around passively waiting to be called by the statements in the language. In a domain-specific language, it's the other way round!

    A related approach, which has been implemented over and over, is based on noticing that languages are usually designed independent of databases, but frequently used together. The result is monster redundancy! Wouldn't it be a hoot if database access were somehow an integral part of the language! Well, that explains how and why ABAP evolved from a reporting lanaguage (necessarily intimate with the database) into "Advanced Business Application Programming." And it explains why Ruby, when combined with the database-leveraging RAILS framework, is so popular and highly productive.

    A declarative approach

    In the world of programming, as in life, the world divides between imperative (commands you, tells you how to do something, gives you directions for getting to point B) and declarative (tells you what should be done, identifies point B as the goal). In short, "what" is declarative and "how" is imperative. A core reason for the incredible success of SQL is that it is declarative, as Chris Date has described in minute detail. Declarative also tends to be less redundant and more concise. As well as doesn't crash.

    Conclusion

    I'm sorry if you're disappointed I didn't name Erlang or whatever your favorite is as the "best programming language," but I insist that it's far more novel and useful to decide on what basis we are to judge "goodness" in programming languages, and in programs. In general and in most domains (with important exceptions, like inside operating systems), non-redundancy is the reigning virtue, and languages that enable it are superior to ones that are not. Non-redundancy is nearly always best achieved with a declarative, problem-domain-centric approach. And it further achieves the common-sense goal of fast, accurate creation and modification of programs.

    Occamality is rarely explicitly valued by programmers, but the trend to it is easy to see. There are widespread examples of building domain-specific languages, meta-data and other aspects of Occamlity. Many programmers already act as though Occamality were the primary dimension of goodness — may they increase in numbers and influence! 

Links

Recent Posts

Categories