A rapidly-growing website for helping readers find books they want to read reached out to me. The deal was that I would write about one of my books and review several books that I liked. It seemed like a win-win. I dove into thinking about the books that most inspired me as a programmer and wrote them up. Here is my page on the site. These are the books that, when I look back on my decades of programming, were the most inspiring.

C Programming Language by Kernighan and Ritchie

This is the defining book on the C language, written by the authors of the language and published in 1978. By the early 1980’s it became the standard for writing systems software, helped by the fact that the UNIX operating system was written in it. UNIX grew to become the world’s dominant operating system, running over 90% of the world’s web servers.

By the time I encountered the language I already had programmed extensively in assembler, COBOL, FORTRAN and other languages. I had written a commercial FORTRAN compiler in assembler language. The C language changed my world, and this book was unique then and now for being written by the creators of the language that changed the world of programming forever. Nothing about all the languages since C makes things better – yes, including C++. All they do is add complexity and put unnecessary constraints on language, along with continuing the focus on language when your goal as a programmer should be to write as little code as possible, putting everything you can into easily editable metadata.

The Art of Computer Programming by Donald E. Knuth

This is a multi-book series. It is the definitive source on algorithms and the core of computer programming. Unlike most such book authors, Knuth is a real programmer, deep into the details of the craft, to the point of creating his own assembler language and typesetting generation system. It’s partly the substance of the algorithms and their analysis, but even more is the way he models a way of thinking about and solving complexity that makes this a must-read series.

Here is a post about software, art and music that features Knuth, shedding light on his depth.

Principles of Compiler Design by Aho and Ullman

Attending Harvard College gave me the opportunity to collaborate with great programmers in creating the early ARPA-net. But the best course I took was on compiler theory and construction, using an early draft of the material in this book. Of course I learned how to build a compiler, which I did as my first job after graduating. But more important, I learned that a well-built compiler is a small amount of language-independent code with two major parts. First the input part that realizes the content of the lexical and grammatical metadata, like today’s LEX and YACC, to turn the program being compiled into a language-independent semantic model. Second the code generator that reads the semantic model and, based on generative model metadata, turns the semantics of the program being compiled to whatever form you want, whether executable code, assembler language, byte code or whatever. This approach, while indispensable for compilers and interpreters, is also the model for taking a meta-data-driven approach to building software applications of any kind.

Godel’s Proof by Ernest Nagel

Nagel’s book is the most understandable explanation I’ve found about one of the most cosmically seminal math proofs: Godel’s incompleteness theorem. It takes the idea of recursion and self-reference to the ultimate conclusion about truth, understanding and the boundaries of existence. This may sound hootie-tootie, but think of the sequence of abstraction in math: arithmetic, algebra, calculus, etc. Once you add in recursion (self-reference), you’ve got the most important concepts underlying true understanding and productivity in software – something never discussed in computer science, and certainly not in practical programming. Think about the so-called Von Neumann computer architecture, which underlies all modern computers – instructions are data, stored in the same place as ordinary data, and some instructions write data that are instructions. This is recursion taken to the next level.

Godel, Escher, Bach: an Eternal Golden Braid by Douglas Hofstadter

This book explains fundamental concepts of recursion, symmetry, intelligence, focusing on math, art and music. It’s full of wonderful tales, amazing facts, little-talked-about relationships among math, art and music (and by reference computing), not to mention puns and amusing language. If you want to become a better person, this is a good book to read. If you want to understand and succeed in life, ditto. If you let this book lift your mind to its space and height, you will find yourself vastly more capable of creating and understanding software. It’s a wonderful companion of the Ernest Nagel book. Hofstadter also helped edit the second edition of the Nagel book. In the foreword, Hofstadter explains that the book (originally published in 1958) exerted a profound influence on him when he was young.

 

Outstanding software development groups are truly outstanding. In many sporting events, the winner is often determined by a margin of a fraction of a percent. In software, the best teams are at least 10X more productive than the competition, and sometimes 100X. Here's the most amazing thing of all: the way the super teams do it is not secret!

Discovering the secrets

For most of my software development career I was only vaguely aware of productivity differences. My job was rarely to judge and evaluate — it was to get stuff done, quickly and well. I figured out some basic stuff to the extent it helped me get stuff done, and hired people who would contribute. I quickly realized that education was irrelevant, as was experience in a particular technology or tool set. What mattered most was raw horsepower, ability to concentrate, willingness to work and the drive to learn.

I think my first wake-up call was a consulting project I did for Sallie Mae during the early 1990's, when I was one of three groups advising the company on a massive re-engineering project using document imaging and workflow technologies. One of the groups was me; the second was a local group with 5 people on the project full time; the third was a national consulting firm that had at least 15 people on the project full time with various senior people and "experts" floating around. I was the only person involved with any of the groups who had not only used the technologies, but created them.

To make a long story short, the other groups performed standard analysis and vendor evaluation, and proposed an expensive process that would result in a 30% productivity gain, if all went well. I dove into the actual work Sallie Mae was doing, noticed some things about the work, and proposed an inexpensive process that would result in a 5X productivity gain. I called the method I used "zero-based re-engineering," combining two fashionable-at-the-time concepts, "zero-based budgeting" and "process re-engineering." They put it to the test and demonstrated that it worked! I tell the whole story here.  Did it catch on? Did the dramatic advantages of the approach lead to widespread use? Nope.

It took years for me to figure out that what I had experienced with Sallie was the tip of the iceberg, both with giant bureaucracies like Sallie Mae and with the "technology expert" consulting firms.

It's developer's gold — but you can't give it away

About ten years later (yes, I know, I'm slow), I started noticing similar stark contrasts among groups of technology companies and the way they built software. The things I noticed were at heart simple — but compared to existing practice, radical. And I now finally realize, revolutionary. Man, I'm really getting the formula for the secret sauce here! I've got to be really careful, and make sure I only whisper these secrets to our portfolio companies. I can't let them get out; if one of our company's competitors got wind of the methods and rolled them out before we did, we'd get killed.

So for the first couple of years, I was really careful with these "secrets." It was only spoken, never written, and only on a need-to-know basis. Some of the companies I would visit wouldn't say they had "secrets" — but they did things differently from most groups, and I eagerly learned from then. I'd visit other companies which could really benefit by employing one or more of these "secrets." I whispered to the smart, motivated people who are leaders in companies in which we've invested where the gold is — and it's really close, right over there!! — and they're ho-hum, thanks a lot, so glad you let me know, please tell me when you plan to visit again, good bye. In other cases, there was apparent enthusiasm, followed by nothing. I thought the problem might be understanding. So I followed up the verbal with e-mails to solidify the ideas.

Learning more and revealing the secrets

Meanwhile, I'm getting a broader knowledge of how the super-developers get it done. It appears there's a spectrum of techniques. Some people use lots of one and little of another, but they all optimize for speed and quality instead of expectations.

I did notice that putting the ideas in writing helped. So I graduated from over-long emails to PDF papers. I told people the ideas, then gave them the papers. Or the other way round. Having the papers seemed to increase the chances that our companies would take the ideas seriously. I kept augmenting and editing the papers as I learned things, and peppered them with real-life examples. Some of the papers went through a couple dozen editions. By about 5 years ago, I had over ten papers comprising over 800 pages.

Why won't people grab the gold?

As I saw more examples of great results, I became ever-more mystified about the laggards who just don't get it.

Now, to be clear, these "laggards" are so only in relative terms. Most of them are way smarter and more accomplished than their compatriots are in the giant legacy companies. In the eyes of the world, the vast majority are leading edge. I started noticing that what I saw as laggards were nearly universally practicing methods that the software industry as a whole has labelled "leading edge," things like unit testing, Extreme programming or Agile. So on a bell curve of established practices, they were definitely leading edge.

My dim light bulb finally started flickering on. Almost no one decides important things like software methods on their own! They pick whatever brand of recognized external authority they're most comfortable with and which best matches their self-image, and try to follow the established method.

Understanding the resistance

But these are programmers, darn it! The methods aren't exotic, requiring a thorough grounding in quantum string theory — in most cases, they're simpler than the standard stuff. So where does the reluctance come from?

This is a BIG subject. There are lots of aspects of it, and I'll return to the subject in future posts. But the bottom line is simple: if programmers want to advance in their careers, for the most part they have to accede to the demands of non-technical managers. The non-technical managers have no clue what programming is about, except that it's full of incomprehensible gobbledygook. Understandably, they want to feel confident of what's going on in the department they're in charge of, most of whose people are engaged in activities that are opaque to them. So they absolutely require the use of standard methods, measures and goals that are entirely consistent with other disciplines in which things get built, like buildings and cars. Anything else feels like a "lack of discipline."

Most of the "secret" methods did not follow the "makes managers comfortable" formula. So the vast majority of organizations would reject their use, regardless of their provable power and effectiveness. For me, keeping things secret was not the problem — it was getting the ideas acted on. There are indeed amazing methods used by software super-developers to get things done 10X or more better than anyone else. But because the methods are not among those talked about by the public "experts" in the industry and do not resemble how houses or cars are built, they aren't a candidate for being used by the vast majority of developers, who do not make decisions for themselves based on substance.

Going fully public

That's when I decided there was no harm and possible benefit in going fully public with the things I'd been noticing over the years. That's when I decided to fix up the papers, turn them into books, and publish them in paperback and on Kindle. Here is a little more background, some illustrative blog posts and links to the books that I've already published.

I am working on book versions of the remaining papers, some of which are more revolutionary than what I've already published. Going through the process of crystallizing what I've learned over the years has taught me a great deal about knowledge itself and innovation, since part of what I've done is observe dramatic advances and seen how the knowledge is acquired and spread.

 I've been releasing the Building Better Software Better series of books for the last 3 years. Four are currently available. Another will be released in early 2016, and two more will follow. I'd like to briefly describe the origins and content of the books.

Writing lots of software

My first job, at 16, was pressing sweaters in a knitting mill during the summer for minimum wage. It was a motivating experience. Later in high school, I started programming computers. Before graduating with honors from Harvard College, I wrote code for oil refinery optimization and the ARPA-net. I then wrote code for compilers, composition systems, operating systems, DBMS internals and applications, large scale financial transaction processing, document processing, workflow and more. You can find details in my LinkedIn profile.

I was hands-on at each job, writing lots of code. I was often dragged into management, but struggled to rise to the level of mediocrity. I did fine interacting with customers and creating marketing stuff. And of course writing code.

Being on the Investor Side

I started working in the technology side of venture capital in the early 1990's. After helping to generate good returns for the fund, I became a GP at Oak Investment Partners in 2000, where I worked to improve the outcomes of our computer-based investments, from infrastructure and tools to the consumer internet. I saw lots of companies we didn't invest in, and of course spent lots of time with the ones we did. I concentrated on health care and financial technology investments with the Oak HC/FT fund, where I was a GP.

Like everyone else, I wanted our investments to succeed. But unlike most everyone else, I dove into the products and services, the code behind them, and the people who built the code. Sometimes the code made little difference. But other times, it really mattered. Regardless of its impact, I got to see how things went with the software and the business, sometimes over a period of many years.

Noticing the patterns

As I matched my own experience with that of the companies we invested in, I started to notice patterns. I first discussed the things I noticed with people in the companies. I found that writing them down was useful, so I put them into long e-mails. As the e-mails grew, they evolved into private-circulation papers. I made many additions and corrections based on feedback and new experience. I started posting about the ideas at my www.blackliszt.com blog. I'm finally releasing the material as the Building Better Software Better series of books.

The book series

Building Better Software Better describes how winning software people actually do what they do. It is often contrary to widespread practice, what is believed by many professionals, and what is taught in computer science and business school. In all too many organizations, failure to build the right software, quickly and well, is a major impediment to business success. Using even some of the methods described in these books can make software into a major driver of business success.

Wartime Software

Building bridges for civilian use takes years and thousands of people. Building bridges in wartime takes hours – and it’s done under enemy fire, and the load and quality requirements are higher. Most software is built the way bridges are in peacetime. Some groups screw up building bridges in peacetime, and their bridges fail, typically because they don't follow the established methods. Some groups  — those who are “under fire” or just don’t have time or money to do it the “right way” — build software using wartime rules – software that is better than peacetime software. They do this not by cutting corners, but by following a different set of rules and methods.

Best to start: https://blackliszt.com/2012/03/bridges-and-software-in-peace-and-war.html

               https://blackliszt.com/2014/02/lessons-for-software-from-the-history-of-scurvy.html

General posts: https://blackliszt.com/warfare/

Book: http://www.amazon.com/Wartime-Software-Building-Matters-Better-ebook/dp/B00CUGHDT8/

Project management

The title is the Disease of Software Project Management, because it has in fact been a disaster for software. The idea that software development should be managed by project management techniques is accepted nearly universally — the only disputes are about which of the many minor variations to choose. This book is an extensive deconstruction of project management, its origins and its application to software development. If phrases like "that will probably take us three sprints to get done" spring from your lips without sarcasm, you need to read this book. The core idea is that all forms of project management optimize for expectations, while in wartime you optimize for speed.

Best to start: https://blackliszt.com/2010/10/software-project-management-dates-are-evil.html

               https://blackliszt.com/2013/03/software-comparing-waterfall-and-agile.html

               https://blackliszt.com/2012/10/software-project-management-book.html

General posts: https://blackliszt.com/project-management/

Book: http://www.amazon.com/Disease-Software-Project-Management-Disaster-ebook/dp/B009RQ6PUC/

Software QA

Things are done faster in wartime, but that's only possible if they're done differently. QA is one of the key areas of difference. There are core ideas to understand, and detailed methods that are different, involving different skill sets than are normally involved in QA.

Best to start: https://blackliszt.com/2012/04/a-simple-framework-for-software-quality-assurance.html

               https://blackliszt.com/2012/10/software-quality-assurance-book.html

General posts: https://blackliszt.com/software-quality/

Book: http://www.amazon.com/Software-Quality-Assurance-Building-Better-ebook/dp/B009TAMNHA/

People

Managers tend to apply the same templates to managing software people that they apply to everyone else. They foolishly think they can manage people who are doing something they don't understand and can't even see. This book covers some of the things that are unique to software people.

Best to start: https://blackliszt.com/2014/10/joe-torre-and-software-development.html

               https://blackliszt.com/2015/03/software-people-book-just-published.html

               https://blackliszt.com/2014/01/who-makes-the-software-decisions.html

               https://blackliszt.com/2013/11/people-and-substance-in-software-management.html

General posts: https://blackliszt.com/people/

Book: http://www.amazon.com/Software-People-Human-Building-Better-ebook/dp/B00VDIFP3U/

Future books

Three more books are in the works. The next one, Software Business and Project Strategy, will be published in 2016.