The Black Liszt

Category: People

  • Process and Substance in Software Development

    High among the concerns of software management are questions of organization and process. While these are reasonable concerns to have, I generally find that paying attention to substance is more productive. If you think of your organization as being like a software factory (a line of thought I generally discourage), this means you should pay more attention to the widgets that come out than the organization of the shop floor.

    Process

    It is easy to be totally consumed by process, organization and people. Everyone wants to know who's their boss. When there are disputes, who has the deciding vote? Many people want to know their "next step" in the organization, the path to greater responsibility, power and pay. Such concerns tend to be greater in the minds of the people on the upper part of the ladder, not to mention the top, since they usually had to work at getting where they are.

    Process and organizational structure are tightly tied. Is QA a separate group with its own head? Or are there QA people as part of each small group of developers? If QA is distributed, what is the reporting structure? This is complicated by the myriad of process fashions that sweep through the industry — there are literally dozens of them in play at any given time, things like Agile and Extreme, with Lean coming up fast.

    Substance

    Substance is embodied in the code that is produced. Given a set of general requirements, the substance of what is produced can differ wildly. Suppose you're extending your application to mobile. Do you use HTML 5? How do you bridge to the details of the local device? Do you write in Objective C (the native language for the Apple devices)? How much do you store locally, and how do you communicate with the servers? What about all the Android devices?

    And I'm just talking about the simplest questions here. Real substance is contained in the details of how the code is written in the chosen environment. For example, the code can be pretty "straight," it can have loads of parameters, it can be layered to varying extents, it can be driven to varying extents by meta-data, etc. These choices have a huge impact on the outcome.

    Process vs. Substance

    Dilbert illustrates the point nicely, as he often does. In the cartoon below, the pointy-haired boss focusses, as you would expect, on process. He is concerned about dates and whether Wally has met expectations that have been set, completely ignorant of the substance.

    Wally, crafty as ever, claims to have created a disastrous substance. The pointy-haired boss, unable to determine whether Wally's claims about substance are true, and unwilling to risk that they may be true, gives in.


    Dilbert Mar 10 2013    Conclusion

    Don't be Wally — but also don't be the pointy-haired boss. Pay attention to substance. Make it your business to understand it. Your attention will provide an example to your group, telling them what's important to you. Your attention to substance will be like a chef who cares that the diners love the food that comes out of the kitchen, and does so by — what an idea — paying attention to the food itself.

  • CTO + CFO = CFBCO

    The CTO and the CFO aren't natural best friends in any organization. They are typically separated by a huge gulf of perspective; neither understands or appreciates what the other thinks or does. The best thing for any organization is when the two of them can truly take the other's perspective, and change what they do as a result.

    CTO

    What's the Chief Technology Officer (CTO) about? He or she had better be the best technical person in your organization. The one who understands the details, the big picture and everything in between. The one who actually understands all those computer acronyms, can sling them with the best, can pick the best and harness them for the good of your organization. At best, the CTO can rally the tech nerd employees to the cause and also convince the suits that everything is good.

    CFO

    What's the Chief Financial Officer (CFO) about? He or she had better be the best financial person in your organization.The one who understands every line of every statement and report, what's behind it, what led to it and where it's going. The one who understands how all that mass of detail relates to company tactics and strategy, and plays a key role it making them align. At best, the CFO can handle the big picture and the details, issues and people inside and outside the company, all to advance the company towards its goals.

    CTO vs. CFO

    When not at their absolute best, the CFO and CTO can have a chilly relationship.

    The last thing the CTO wants is for some nosy bean-counter to mess with his stuff. Even simple questions are suspect: why is he asking? what's he looking to cut? Go away! Most CFO's seem like clueless idiots to even average CTO's. All they can possibly do is waste time and, through crass stupidity, make things even harder than they already are.

    CFO's often see the whole tech group and the CTO in particular as being a bunch of whiney, spoiled, self-absorbed brats. They're anti-social, talk among themselves in their private language, get huffy or all-too-patient in response to even the simplest of common-sense questions, and seem perversely intent at avoiding anything that increases revenues or profits. They're perpetually late, reluctant to commit to anything, always complaining about lack of support, but still manage to have an attitude about pretty much everything.

    CTO Fundamentals

    Most of the thoughts I've had about computing that are worth anything took me years, often decades, longer than they should have to penetrate my thick skull. But one of the earliest realizations I had remains both true and rarely discussed: to the extent that computers are applied well, they cut jobs, and therefore (usually) costs.

    This isn't the pretty way to put it. Most people prefer to think about how computers enhance people's efforts. And they do — meaning you can get the same job done with fewer people. Or you can deliver more with the same people — to deliver more without computers, you would have had to hire more people. Any way you cut it, the more widely and effectively computers are used, the fewer people you need to get a given job done.

    Put all the gobble-de-gook aside, and what computers come down to is simple: cut costs, do more with less, get it done faster, etc.

    Now what does this sound like? Could it, perhaps, sound like the kind of thing CFO's are supposed to worry about? Hmmmm….

    The CFBCO

    Maybe there's some middle ground here. At the heart of the matter, there is no skill set in an organization better suited to helping a CFO meet his goals than the CTO's. And when a CTO who is really good in nerd terms wakes up and realizes what his job is really about, there is no person better suited to be a company-maker than a bottom-line-oriented CTO.

    Put in the most basic terms, a good CTO can make things happen:

    • Faster
    • Better, and
    • Cheaper.

    Faster, better, cheaper. That's computing in a nutshell, and when computing is applied to an organization to greatest effect, that's what happens to the organization. It does what it does faster, it does it better, and it does it cheaper. FBC. Wouldn't it be nice if we could have a Chief Faster-Better-Cheaper Officer?

    Conclusion

    In most organizations, there is a spectrum of leadership. At one end of the spectrum is the nerdy CTO. At the other end is the what's-your-handicap CFO. If you're very lucky and very deserving, maybe you have someone at the center of that spectrum who combines the best of both extremes in a single individual. The CFBCO. The CFBCO combines ultimate nerd-power with dollars-driven vision and insight, and makes the relevant numbers better in a way that even the dullest and most distracted board of directors can understand and appreciate.


  • Interviewing Software People

    The methods in widespread use for interviewing and selecting software engineers are appalling. It is only because they are so bad that ridiculous methods like those often used at Google in which applicants are given trick puzzles to solve can seem like an improvement. The sad thing is, asking candidates to solve mental puzzles is better than what's usually done, which is not much at all. Come on, people — we can do better!

    Typical Selection Practices

    Managers need more programmers. They get a job requisition from finance, then go to HR to get candidates. Since HR knows nothing about the substance of the work that needs to get done, what ends up going into the job requirements are a bunch of motherhood-and-apple-pie blah-blah (self-starter, etc.) and a list of keywords of the technologies in which experience is required.

    HR screens candidates based on whatever is in the resumes and may interview candidates, basically to see whether they mouth the expected platitudes when prompted by the HR people. Those who play the game are then passed to the programming department.

    Typical Interview Practices

    The candidate is typically scheduled for a round of interviews with programmers and managers. Since the managers rarely know much and have usually forgotten what little they used to know, they don't ask questions of substance; they basically find out if they like the candidate and if they believe the candidate will "fit in" and follow orders. The fellow programmers who interview remember their own interviews, in which questions of substance were few and far between, so they basically chat up the candidate and decide whether they like them. At the end of this "rigorous" process, if everyone agrees, the candidate is accepted.

    What a joke! When you're hiring a musician, an audition (in which the musician performs) is standard practice. When you're hiring a writer, reading things previously written by the candidate is standard practice. So when you're hiring a writer of software programs, naturally you'd expect that reading programs previously written by the programmer would be standard practice — but it's not!

    Leading Edge Interview Practices

    Instead, the leading edge at places like Google is to hit the candidate with trick questions. For example: "Suppose you were suddenly shrunk to the size of a nickel and found yourself at the bottom of a blender. The blender is going to start in a minute. What would you do?"

    If I were the size of a nickel, most of my neurons would be gone, so I wouldn't be me anymore. But more seriously, how relevant is the kind of skill questions like this test to writing programs?

    I could make an argument that this kind of thinking is relevant to a kind of programming that is important, but very rarely needed: algorithmic design. No one has ever (to my knowledge) measured it, but I would be surprised if algorithmic programming amounted to as much as 1% of all the code in a typical application. The vast majority of code that's written needs different kinds of skills: visualizing user interactions, understanding data structures and data flows, understanding and effectively using complex subsystems, and many other activities. These activities benefit little from the kind of skills and instincts required for solving trick puzzles.

    Are there people who can do the puzzles and be great "regular" programmers? Of course. The problem is the reverse: there are many people who would be perfectly adequate programmers who are flummoxed and generally disconcerted by questions of this kind. I've been in groups of trick question masters. They're great for finding what's complicated in basically simple things and other arcane but fundamentally counter-productive skills. Other than that — you can have 'em! Take 'em all — please!

    What can be Done?

    There are lots of simple steps that can be taken to improve the outcomes of sourcing and selecting software people. Any step you take is likely to make things better. I hate to do it, but I have to admit that even trick questions are better than the "Hi, how ya doin'" method of interviewing. But surely we can do better.

    Reading code. When hiring writers, we read their past works. Why are we so reluctant to do so for people who write code? I suspect it's because very few people would actually be able to read the code and make a reasonable judgment of the author — for all too many typically mediocre programmers, that would amount to a tour de force way beyond their limits. That's OK — find out who can read code with meaning and judgment, and they become your main filtering agent. Maybe, just maybe, you'll end up with .. people who write better code! What a concept!

    Auditions. What's wrong with an audition? If someone is supposed to know database design really well, show them one of your current ER diagrams and ask for comments. Tell them a change that is proposed and ask how they'd make it. Pose a recent tough problem you had to solve (which you've already solved) and ask them to solve it.

    Detailed archeology. A candidate programmer may not know much about your stuff, but she'd darn well be an expert on stuff she's coded in the past. Find a subject where your experience overlaps hers, and ask for a detailed rendition on what she did, why and how — and what she learned and would do differently today.

    Subject Matter Testing. Yes, testing. Like an audition, only more objective. If someone really is the expert php programmer they claim to be, they'll ace the test. No problem.

    Conclusion

    Software hiring is an embarassing mess. In a field that is over-the-top exacting with fairly objective pass/fail criteria (the program works or it crashes), the methods we use to ask new people to join us are random, ad hoc, and almost completely unrelated to finding out whether the candidate can actually perform as required. Asking trick questions can actually be an improvement, but that's not saying much. We can and should do better, and even a little better can make a huge improvement in the quality of the people who build our software.

     

  • Status in Software: Silliness and Stupidity

    In all too many software groups, you get higher status by being more removed from actual customers, their needs and concerns. This is bass-ackwards. It's silly. It's perverse. It is profoundly stupid and counter-productive. If this is how your software group works, change it or leave. Now.

    The Inward Flow: Support

    In most organizations, here is the perverse flow:

    • Customer has problem. Contacts Customer Service.
    • L1 customer service takes the call or e-mail. Eventually. They try to do something, but don't have much knowledge or power. So after wasting some time, it's off to…
    • L2 customer service, which is backed up failing to handle other things L1 already kicked up to them. After wasting some of their own time, and often some of the customer's as well, it's off to…
    • L3 customer service, which is the place where the really experienced L2 agents are promoted. Life is messy in L3. All the nasty problems end up there, often with the customer already being (understandably) mad, but too frequently lacking the skills and resources to even reproduce the problem, much less fix it. After spending some time here, the worst problems of the most upset customers migrate to…
    • Sustaining engineering. This is the death-watch group in engineering. Two types of characters are typically confined here: ignorant entry-level people who hope to move up and out; and experienced engineers who missed the cut for working on the new stuff. If it's an easy bug, they may be able to fix it. Otherwise…
    • …it may actually be necessary to interrupt an exalted person who wrote the code that caused the problem, taking him away from the important business of writing code that has brand-new problems! But this drastic measure is avoided if at all possible.

    There are actually more layers to march through, but the pattern should be pretty clear by now: the "most important" people are protected from the consequences of their past mistakes by layers and layers of carefully arranged bureaucracy designed to deflect and defuse any contact with real customers and the problems those customers may be having. The more you know, the more distant you are kept from having your august presence sullied by the trivial annoyances of mere customers. It doesn't need to be this way.

    The Outward Flow: Development

    When new products are created, it is all too often the case that the higher your status, the more removed you are from contact with the people who will ultimately use the product you create.

    In very large organizations, the remote peak of the status hierarchy is occupied by research groups or labs. These are truly hilarious. Why do they have ultimate status? It is a given that they see no customers, hear no customers and talk with no customers; but even better. they produce nothing tangible at all — unless you count academic papers and research reports. Those people are sure important! Their ground-breaking work will (pick your favorite) "lay the foundation for," "create the basis of," or "make the discovery on which" generations of future products will be built. Sure.

    Smaller organizations would love to have such a group — it's prestigious! — but instead make do with a few exalted individuals who think deep thoughts and create "architectures" that "solve" a wide range of present and future problems.

    High level design people then take over to create a "design" within the "architecture." This is not easy! It's important to fend off the constant pressure to produce something practical that works for today's customers, in favor of doing the design "the right way," i.e., spending lots of time thinking about problems some customers may have in some unspecified future, and "creating a framework" that will supposedly make them easy to solve.

    At this point, software development splits into an alphabet soup of competing creeds, each of them certain of their unique virtue and access to software heaven. There is the much-maligned waterfall, agile, SCRUM, extreme, and on and on. The details of what happens next vary. The status relationships and ultimate outcomes are pretty much the same: the more important you are, the less likely you are to have meaningful contact with customers. This remains true as the software staggers through phases that may various include integration, testing, staging, documentation and roll-out.

    Finally — finally! — the software is inflicted on the customers for whose benefit it was built. All I can say is that the chorus of complaints, however loud it may be, is rarely loud enough to penetrate the excellent sound insulation of the rooms in which the company's "brain trust" festers.

    Conclusion

    If you want to run a charity organization for egotistical, self-absorbed and self-important programmers (OMG! Did I just use the demeaning term "programmers," implying these people might actually lower themselves to doing actual, like, work!? I meant to use a more elevated term like "intergalactic systems architect" or "chief scientist.") — like I was saying, if it's your goal to provide welfare to high-minded computer scientists, by all means employ a staff of "elite" techies and help them avoid being interrupted by the hoi polloi. Their deep pondering is way too valuable to be sullied in any way by the mundane concerns of the common people. If, on the other hand, you have real work to do and want your best people to lead, then make sure that the closer people are to customers the more status they have. Building a product or service that real people value and want to use requires — gasp — contact and interaction with those same real people.

     

  • How Great Software Teams Can Go Wrong

    Every once in a while, a miracle happens: a group of smart, hard-working software people latch onto a group of customers with unmet needs, and wonderful things take place.

    Once things get up a head of steam and it becomes obvious that good things are happening, the vultures start circling. "That's an inappropriate metaphor!" you might exclaim. "Vultures feed on dead animals, not ones that are alive and well." OK, I concede the point. "Vampires" would be a better metaphor, because everyone knows that vampires attack the young and healthy and turn them into unfeeling parasites like themselves. Let's compromise: when there's great software happening, the vampires start circling.

    The vampire/vultures tend to be older than the programmers actually doing the work. Regardless of the facts, they like to complain about things like the lack of order, the lack of predictability, and the lack of control. They ALWAYS find lots of evidence that these supposedly indispensible things are missing. Things are moving quickly and the business is growing quickly, so of course the vampire/vultures find all sorts of evidence that the business, as currently "organized," isn't "scalable." Things may be going OK now, but we have just been lucky! There's too much at stake to leave our future to chance. We need some "experienced, senior" managers to assure the future of our important business.

    What happens next is simply awful. I've seen it too many times. I know all the script variations. But let's skip past the mayhem to the final stage, which can reasonably be called "the inmates are in charge."

    The Inmates are in Charge

    There are sure signs that the inmates are in charge. When this happens, you may as well put this sign over the entry door of your office: "Abandon All Hope Ye Who Enter Here."

    Following is a sample list of the reasonable-sounding things that people say in this horrible place, each followed by the reality.

    “After you work a certain number of hours, your efficiency simply goes down. You can’t concentrate as well."

    Everyone slips out starting at 4 p.m. Which would be OK if they hadn't wandered in around 10am.

    “This pushing to make unreasonable deadlines just exhausts everyone. They fight with each other, don’t get any better results, and then, after, everyone is so beat that nothing gets done for weeks.”

    Deadlines get established so far in the future that earthquakes, floods and fires can take place, and they can still be met. The meaning of the word “deadline” evolves from “I’ll make it or die trying” to “a dead person could meet the target.”

    “People who do nothing but work are unpleasant, narrow, one-dimensional people. They don’t work well in teams, and before long they just crack up and become unproductive anyway.”

    HR people start roaming the halls at 4 p.m. suggesting that people go home and spend more time with their families. Anyone who seems stressed is invited to take a week’s vacation.

    “Our products are sophisticated, complex pieces of technology. It takes a long time to understand everything. It’s hard to find enough experienced people to do the work.”

    The “go along to get along” people are promoted and salaries of senior people advance rapidly.

    “There is a huge amount of value in the intellectual property we have built up here over the years and embodied in our people and processes.”

    Suggestions for change are rejected without serious consideration.

    “Rodney isn’t really a team player. He makes other people uncomfortable, and sometimes says things that make other people feel bad. He’s bad for morale.”

    Rodney is smart, hard-working and wants to build a great product. But he doesn’t fit in well, and is asked to look elsewhere for employment.

    “Our customers have come to depend on our reliability and commitment to quality. We have to continue to pay attention to what we do, and continue to do it extremely well.”

    Don’t bother suggesting doing more, working in parallel, or getting things out more quickly.

    “Our customers don’t have the money to chase after every cool-sounding new idea that comes along, most of which fail anyway. They depend on us to incorporate new  technology for them once it actually becomes useful and proven.”

    We will continue to defend our approach, using words like “practical” and “proven,” even after it has become dead-and-buried obsolete.

    AARRRRGGGHHH! Get me out of here!!!

    Conclusion

    Thankfully, not all great software efforts follow this tragic script. But here's something I've learned: when things are going well in software terms, there are always vultures and/or vampires eager to attack and do their awful worst. Constant diligence is required to keep them frustrated and hungry.

  • Experience and Youth

    I don't like to talk about this subject, because it's one of those almost-always-lose topics, not unlike politics or religion. But unlike politics and religion, it comes up ALL THE %^&*%^ TIME in the computer-centric circles I travel in. And it has impact (in a bad way) all too frequently.

    Here's what got me going on this today:

    Dilbert 2010-12-23
    Credit: Dilbert

    I can relate to this cartoon. When I was younger than most of the people I dealt with, I frequently encountered ignorant older people who weren't great when they were younger, but now they're old and even more out of it, and desperate to maintain control and authority. That was the pattern.  As I've gotten older myself, I now see people substantially younger than I am playing the role of the old guy in Dilbert's cartoon to perfection. What's worse is when they're in positions of power, and they just get their way because of the power, without even needing to wave around modems in threatening ways.

    Robert-mankoff-what-lemmings-believe-cartoonbankcom-1230088780485
    On the other hand, there is no such thing (in reality) as six-month-old technology that simply springs out of thin air. All the stuff that seems new is in fact a relatively minor extension of concepts and components that have been around for a long time, in some cases reviving things that were invented decades ago, but the time wasn't quite right for them. The "kids" usually don't know that. They have no history, no real experience, and they work in a field in which those things are not valued. Every once in a while one of the cool new things that seems like it's going to take over the world really does, but all too often the lemming cartoon above applies. The kids could benefit from a bit of perspective.

    Does the old person want to hear it? Generally, no. Does the young person want to hear it? Well, sometimes they'll listen politely and hope the noise stops soon.

    I wish I had a trick or technique to solve this problem, but sadly I don't. I just try.

  • What is the Best Programming Environment?

    What is the best programming environment? Is it Microsoft C#? What about Java? On the other hand, there are the open source scripting languages: are they all about the same, or is python way better than php? While we're at it, how about databases and operating systems? Isn't it true that you really need Oracle if you want a truly scalable application? And if you're really serious, shouldn't you take a close look at DB/2?

    As a long-time techie who has the opportunity to work closely with a wide variety of software/hardware groups, and often has the chance to take a close look at yet more groups my firm is considering for investment purposes, I confront this question frequently. I also get it thrown at me, sometimes by anxious investors or business leaders. They are worried about the possibility of making the "wrong" choice. They are bombarded with conflicting advice, frequently from techies who are truly knowledgeable people and speak with authority and confidence. It's tough!

    OK, Mr. Smart Guy, dish it out! You've got inside information on all these efforts using the different tool sets. You see which ones are productive, and which are not. You see which scale and which can't. What's the answer?!?!

    The good news is, I do have the answer. And I'm going to tell you. But you have to sit through a story first.

    The scene is fifth grade. The playground was a competitive place for me. Running games were important. I was pretty fast, but not the fastest. I needed to get just a touch faster. After much begging, I finally got the new sneakers I had been pining for. The sneakers that would make me run faster, just like the ads said. I was real excited. I put them on and tried them out. Darn! It's true! I really can run faster in these sneakers. I would do little speed bursts, and was amazed at what a difference those sneakers made!

    Pro-keds-sneaker
    Then I went to the playground. I wore my new secret weapons and a smirk on my face. I felt no need to brag; I would let my amazing new speed do my bragging for me. Then the games began.

    Something was wrong. VERY wrong. HOW COULD THIS BE?? I just KNOW I run faster in these sneakers! But I'm not winning!? And with that experience I took a small step towards growing up…

    Thanks for sitting through that vignette from my childhood. Here's why it's relevant: programming environments are like sneakers, and many of the people who use them are like fifth grade boys who don't actually have to compete against other boys on the playground to find out how much difference those sneakers really make.

    Here's the answer to the original question: differences between sneakers (programming environments) are tiny compared to differences between kids (the skills, sophistication and raw horsepower of the people who use them). Put great shoes on weak, slow, unmotivated kids and it won't help them much; force strong, fast, passionate kids to wear crummy shoes and it won't slow them down much.

    This is not my "natural" way of thinking about this question. It is what scores of data points over many, many years have forced me to. The data points don't just come from things I've heard; they've come from things I know up close and personal. I could give loads of examples.

    That this conclusion about technology surprises many people tells us how isolated the field really is from normal human experience. Who, for example, would be surprised to hear that:

    • in baseball, the batter matters more than the bat
    • in art, the painter matters more than the paint or brush
    • in writing, the writer matters more than the word processing program

    Are there differences between the major programming environments? Yes. Can you "prove" that one is better than another for a particular task? Yes, vendors do this all the time. They want you assume that tools are like trains: all you have to do is pick the "fast" train and it will take you to your destination quicker than the slow one. But the reality is that tools are more like sneakers than trains — the tools are things capable people use to get their jobs done, rather than being machines that transport people to where they want to go.

Links

Recent Posts

Categories