The Black Liszt

Author: David B. Black

  • Medicine as a Business: Billing 1

    If you were to argue that my fascination with medical billing, including the endless-seeming minutia of it, is somewhat, well unusual, I could not dispute it. "Guilty as charged" (or billed?) would be my only response.

    I've learned that the obvious, annoying uniqueness of paying for medical services, different than pretty much anything else in our lives, is the tip of an iceberg of financial pain, metastasizing bureaucracies and festering IT dysfunction for the institutions and people involved in it. See this for some context.

    This is a strong, broad statement. Let's dive into some real-life, on-the-ground facts and see what we see.

    Background

    I’ve been under the care of an excellent doctor, now working at Northwell Health, for a super-rare, long-term condition. I called his office because there was evidence that the condition was worsening. His similarly excellent NP buddy responded to my phone call report by authorizing an MRI of the area in question, and making an appointment for me to see the doctor 4 days later. I truly appreciate how exceptional this accessible and responsive behavior was.

    I went to the MRI and then had a consult with my doctor. Sure enough, the evidence I noticed was confirmed by the MRI, and he immediately started the appropriate treatment.

    From a medical and scheduling point of view, this experience would be hard to beat. Anywhere, under any system.

    This post, however, is about the hospital system billing, with the heavy involvement of my insurer. Every step of the process was chock-full of stupidity, waste, friction and “opportunities for improvement.”

    The bills

    Here’s a bill I got for the MRI:

    MRI bill 1

     

    Here’s a bill I got for a visit with the doctor who ordered the procedure:

     

    Maki 1

    Take a quick look at the two bills, each issued from the same health system for visits which took place in Manhattan a few days apart. Could they be more different?

    Right from the get-go, we know we have a problem. Totally different systems producing bills that are radically different, involving different systems for generation, tracking, and collection. Wow.

    Totally different systems

    Even though both bills clearly have the Northwell Health logo clearly displayed at both the top and bottom of the page, nearly everything else about them is different. The return address is different, the address to which you send the bill is different, even the little box where the payment is defined is different. It blew me away that even the web page for electronic payment was different!

    This means different software systems for generating the bills and collecting the payments, among other things.

    The MRI bill

    Let's look at that MRI bill. It's written in the form of a letter, which is interesting. It was amazingly prompt: the "service" was "rendered" on Dec 7, while the bill was dated Jan 13, only 5 weeks later. In terms of medical billing, lightning fast!

    Who is the bill from? Clearly, Northwell Health. But that doesn't help, because Northwell is a giant system. The return address of the bill says "Lenox Hill Hospital," with a PO Box in New Hyde Park, NY. That's nice, except that there is no place called "Lenox Hill Hospital" in New Hyde Park — though there is a huge complex that is part of Northwell called "Long Island Jewish Medical Center." So where is "Lenox Hill Hospital?" That's easy. It's a big place on E 77th St in Manhattan. The addresses they give aren't helpful. What about the letter itself. Maybe there's a hint there?

    Now we're getting somewhere — the letter clearly states when and where the service was performed:

    111

    Excellent, the service was performed at Lenox Hill, the place on the Upper East side of Manhattan. The trouble is, I didn't go there for my MRI!. Not there, and not LIJ. I went to an imaging center run by Northwell in the Chelsea neighborhood of Manhattan.

    Maybe they're talking about something else than my MRI. Let's check out that bill again, and see for sure that it's a bill for my MRI. Uh-oh. Trouble. Nowhere is MRI mentioned in the bill, or medical imaging of any kind! Look again at the bill and the snip of it above. The words used are "services rendered." The wonderful people at Northwell go to the trouble of putting a Spanish-language version of the bill on the other side of the paper, but they can't be bothered to tell me what service was provided or where it was provided. Actually, it's worse. They clearly state in the snip above that the service was provided at Lenox Hill Hospital. Which it was not.

    Now try to pay the bill. Hah! They give a website. Let's go there.

    11

    I have to guess that they want me to pick "hospital," so I do and pick the one on the bill, Lenox Hill. I hit "submit." Here's the result:

    12

    Software is wonderful, isn't it?? Saves trouble, filling out paper, stamps, and all that old-fashioned stuff. Just go on-line and pay. Except when the software doesn't work.

    That's plenty for a single blog post. We'll have some fun with the other bill next.

     

  • Medicine as a Business: Billing Overview

    If you break your arm, broken arms are suddenly at the top of your list of least favorite subjects. But after the arm gets better, the billing process for the arm-fixing services is probably pretty high on the list.

    Medical billing is something too many of us are way more familiar with than we'd like, but nonetheless will serve as an excellent first subject for this series on the business of medicine.

    Let’s start by putting medical system billing in perspective.

    Going to a hospital or doctor to get a service is in most ways like going to any place and getting a service.

    • You’re hungry, you go to a restaurant and get a meal.
    • You’re shaggy, you go to a barber or salon and get a cut.
    • You’re bored, you go to a movie or show and get entertained.
    • You’re bored and hungry, you go a bar with a show
    • You need a book or chair or laundry detergent, you go to the appropriate store.
    • For most of the above, you just go. Sometimes, when the service is popular, it pays to make a reservation. You call or do it online, and your space at the place is assured.

    A pattern is clear here. When you need something, you go to the place that has, does or serves that thing and you get it. Pretty simple, and universal. Same thing with medical issues!

    • You broke your leg, you go to the hospital and get it fixed.
    • Your skin gets weird and painful, you go to the doctor and get it checked out.

    But there’s something really important that I’ve left out of the above list; I suspect you know what it is. Let’s see if you’ve guessed what it is. I’ll wait and give you some time. Click … click … click … OK, time’s up! Did you get it? You probably did, but just in case, here’s the answer:

    The billing and payment are totally different! And the scheduling/reservations are often a nightmare!

    Here's how it works for nearly everything:

    • For the movie, the price is posted, you pay when you enter.
    • For the restaurant, salon, and retail store, you choose from a menu or list or wander around picking things, and pay when you leave based on the services/goods you got.
    • For the bar with show, you pay the cover charge when you enter, and your bar tab when you’re done drinking/eating.
    • For all of these, you can pay with cash or card. The card could be a credit card, which enables you to pay later or make payments.

    Now, let’s look at the big exception: medical billing.

    • If it’s an emergency, you might go to the ER and wait for hours.
      • If not, making that “reservation” may require a “pre-auth” and a variety of other things that are often painful, and sometimes denied.
    • You don’t pay when you enter.
    • There are no posted prices, unlike the salon or movie. No menu. No price tags.
      • Some stores let you special order things, sometimes one-of-a-kind. Before getting it, you get a price and make a payment arrangement. Not in medicine!
    • There is no “check-out register” where you find out what the total comes to.
      • Unlike a restaurant, where no one has any idea what food and services you’ll receive when you walk in, there is no bill at the end.
      • In medicine, it’s common to leave after services have been rendered, and eventually a flurry of bills may arrive from different places and/or notices from insurance wanting information or telling you what’s been “covered” and what they’ll pay.

    Go. Get services. Leave. They're all the same. Schedule it and pay for it: it's totally different in medicine!

    • “I broke my leg” is like going into a car repair shop knowing you want it fixed, but not being able to do it yourself or knowing what it will cost. It’s your only car, so it’s got to get fixed.
      • The car place tells you about what it will cost and you agree to pay when done. Not so at the leg fixing place!
      • Bills from the ER and a couple doctors could show up over the next couple of months, with lots of fine print. The insurance company probably joins the fun with multiple pages.There's loads of stuff that has no counterpart in the real world of product and service buying.
    • “I have funny skin” is like going into a salon knowing your hair and makeup just doesn't work for you, and you don't know what to do. You need a new "look" and need help and advice. After a discussion you might look at some pictures and try a couple things out. A specialist may need to be called over.
      • You get your new look and pay for the products and services on the way out of the salon. Not so at the skin fixing place!
      • There might be "co-pays" that you owe someone, or maybe not if they're "in network." Letters about any "coverage" some family member may or may not have. Not to mention surprise bills. Nothing that any salon of any kind would ever try if they wanted to keep their customers.

    All of this is bad. Really bad. It's terrible for patients, terrible for medical people and institutions, and no fun at all for insurance companies. Believe it or not, the behind-the-scenes, underlying reality is even worse! Which is part of why it doesn't get better.

    In the later posts in this series, I will go into specifics with real-life examples. In each case, while there are clearly systemic barriers to improvement, there is a clear path to improvement.

    Among the issues I'll cover are these:

    • The delay between the service and the bill
    • Bills from different places
    • The difference between provider billing and insurer paying
    • What service was provided, where and by whom?
    • The challenge of paying the bill

    This is going to be fun!

     

  • Medicine as a Business: Overview

    Medicine is a business. It’s a big business. But it’s like no other business, often in ways that are not good for anyone involved.

    The good news is that there are many amazing people who work hard, have compassion, and do their best to make things better for their patients. More good news is that dedication and hard work have created drugs and procedures that can cure or alleviate awful conditions, conditions that resulted in pain, suffering and premature death in the past. We are truly blessed.

    It’s not news to say that the business of medicine could be greatly improved. When this subject is discussed, it’s usually done in terms of rocks and hard places, or irresistible forces and immovable objects. Then hands are waved, thrown up in the air, and nothing but desperate hopes are expressed.

    I believe that the business of medicine can be improved. Greatly. From the outside, it’s impossible. But when I look at the computers and software systems and procedures, I find a particularly ripe example of something I’ve seen many times: decrepit software running on outrageously expensive hardware, surrounded by ineffective business processes, and run by experienced, well-meaning people trapped on a steam-age island in a world that has long-since gone electronic. While there is no magic wand to fix everything in a flash, the technology is available to make dramatic improvements. Today.

    To a broadly experienced software person, the solutions are obvious – once the exact technical flaws have been precisely identified. Much of this series will be devoted to spelling out those flaws, not to wallow in them, but to put cross-hairs on them so they can be fixed!

    As context for the rest of the posts in this series, let me observe that the issues I will raise are waaaay too "little" to engage the big minds and the great powers in the medical industry. These prestigious and powerful people much prefer to give talks, issue press releases and hang banners about how they and their institutions are "innovative," how they are leading the industry in their use of "cognitive computing" or "artificial intelligence." They hold and/or attend conferences on the subject, and pour money in those general directions, each of which is sure to deliver real results real soon now — why, the trials are just so promising!!!

    2018-04-24 09.33.00

    That's not my perspective, to put it mildly. See this for some plain words and facts on the subject of fancy-pants innovation in healthcare. And see this for facts and logic that help explain why this subject gets almost no interest or attention from the higher-ups in healthcare management.

  • Getting Results from ML and AI: 3, Closed Loop

    Getting practical, real-world results with ML and AI involves more than getting data, doing calculations, and building models. You can do everything else right, but if you don’t get this last step right, you’ll join the rapidly growing ranks of people who may have tried hard, but ended up accomplishing little in real-world terms.

    The first part of this series laid out the issues and concentrated on the indispensable foundation of success, the data. The second part of this series dove into the analytic methods that can be used to generate value, with some advice about how to sequence the methods used.

    In this post, we’ll concentrate on the relationship between the real world and the back office analytical work. What we’ll find is that an integrated, collaborative, closed-loop relationship between measuring, calculating and real world application is the path to success.

    Loops, open and closed

    Whether you run an operation closed loop or open loop is one of those absolutely key concepts, highly correlated with success, that is rarely discussed. Generation after generation discovers it by itself, or not, nearly always without fanfare. Who talks about the key role played by the invention of the governor in 1788 by James Watt – the invention that made his steam engine practical? In that case, the governor was the newly-minted part of a steam engine that kept the pressure of the steam reasonably constant. With a governor, steam engines no longer blew up, as they regularly did before its use.

    Centrifugal_governor

    It’s important to understand that the reason a governor works is that it’s an integral part of the steam engine. Steam goes into the governor, which then controls the throttle valve of the engine, slowing it down when it’s getting dangerously hot. This is closed-loop.

    In more modern terms, running open loop means going on and on down a path without real-world feedback and testing of your work as it is being developed. It’s a little like trying to walk to a goal post at the opposite end of a football field with your eyes closed, using a carefully planned sequence of steps and turns. That’s open-loop, which essentially mean no feedback. But shockingly enough, a huge fraction of highly technical efforts in software and analytics operate in just this way!  The people in charge insist they’re experienced, they’ve got a thoroughly vetted plan, and everyone should let them alone to get their work done.

    There are many similarities between war-time software and running closed loop for analytics. Driving towards a goal, letting nothing get in the way. Optimizing for speed, not expectations. Leaping to a place that's better than today, and then cycling improvements.

    The easiest way to see the difference is thinking about the previous posts in this series. Have you spent lots of time with data, and applied simple calculations to it? If not, you should. Once you have, … you should put your new understanding into practice! It may not be the very best solution that’s possible, but if it’s marginally better than what’s in place today, you should roll it out at least in a limited way and see how the world reacts to it. You’ll learn stuff! You may end up learning there are more variables you need to account for, different ways it needs to be applied, all sorts of things! In other words, don’t sit on the beach by the water for months – wade right in and see what it’s like. That’s when you’ll really start learning.

    The World Responds and Changes

    The key concept to understanding why running closed loop is so important is that the “world” is an incredibly complex, ever-changing set of actors. When you do something – almost anything – the world changes in response to what you did, if only in a small way. You have to run closed loop to respond appropriately as the world responds to your actions.

    Oh, you may say, I’m just the genius in the back room who’s an expert in this or that branch of ML. I’m not acting on the world. I just need the time and support to get my amazing modeling work done.

    That may be true. And that’s the problem! The whole point of doing ML/AI/etc. is to change something in the world –  and it’s guaranteed that the world will change in response! Accounting for the responsive changes is just as important as whatever it is you first put out there. Even worse, the world constantly changes independent of anything you may do. So the solution you modeled for may not be valid, given the changes that happened.

    Think about the carefully planned walk on the football field to the goal posts I described above, and how hard it would be to accomplish with your eyes closed, i.e., with no feedback. Now think about the same situation, except there's an opposing team on the field! You carefully study everything about the opposing team. You know who they are and where they are. Then the play starts and you start to execute your exquisitely planned march to the goal posts. Here's the trouble: opposing team members see what you're doing, and they change their positions! They move! Even worse, they run towards you and try to tackle you. And you are helpless, because you are carefully executing your wonderful plan with your eyes closed, unable to react to the other team's movements. Is that stupid or what? It's not just stupid, it's inconceivably stupid. That's why I spelled it out, because that's exactly how most ML and other analytics efforts are carried out. Open loop. Assuming that the world does not change in response to what you do.

    Of course, the world is unlikely to be quite as single-minded and determined as members of an opposing football team. But you'd be surprised! You're making changes in the real world. Whatever you do, there are probably losers. Losers who won't be happy, and will change their behaviors so they become winners again. Or simply fail to act in the predicted ways.

    Conclusion

    Running closed-loop is absolutely indispensable to achieving success. Put something simple in the real world and then cycle, making it better and better, using increasingly sophisticated techniques. Whatever your final crowning technique is, whether it's ML, AI or something else, success will be yours, and you'll enjoy it all along, without the risk, anxiety and likely failures of the usual highly planned methods.

  • Getting Results from ML and AI: 2

    Getting practical, real-world results with ML and AI isn’t just a matter of hiring some people with the right credentials and throwing them at the project. Most such efforts start with fanfare but then fade into failure, usually quietly. The first part of this series laid out the issues, described the path to success, and concentrated on the indispensable foundation of success, the data. Data that has to be collected, corrected, enhanced and augmented – a time-consuming process that has no “advanced algorithm” glory, but MUST be done, and done well.

    In this post, we’ll concentrate on the analytic methods that a successful project uses to generate value from the data so arduously collected and corrected.

    A Little Background

    I say I’m focused here on ML and AI. I just said that because it’s what everyone is talking about. What I’m really focused on is algorithms for understanding and getting value out of data. So I lied. Even worse, I’m not sorry – because just thinking that what’s important is to use the latest ML and AI techniques is central to the failure of most such efforts to deliver value.

    I guess I can get over my programmer-ish prissiness that things are getting new names. What I refuse to get over is that lots of important, really valuable techniques are usually left out of the grab-bag of “ML and AI.” I won’t be comprehensive, but I think a glance at the landscape might help here.

    There are a couple different ways to understand useful algorithms and how they came to be. Roughly, they are:

    • Follow the algorithm, taking a fuzzy lens for the naming and details
    • Follow the academic departments that “own” the algorithm
    • Follow the problems the algorithm has proven to be good for

    These ways overlap, but provide useful angles for understanding the algorithms and where to find them.

    Let’s illustrate this with an amazing, powerful algorithm that is usually sadly ignored by people who are into ML and AI. It’s most often called linear programming (LP).Those who are into it think of it as being one of a category of algorithms called mathematical programming. More broadly, it’s normally “owned” by academic departments of operations research (OR). OR studies repeating operations like responding to repairs for appliances or controlling the output of oil refineries when prices and costs vary and optimizes the results. It’s been used for decades for this purpose in many industries, and is being rolled out today to schedule infusion centers and operating rooms in hospitals.  

    This isn’t the place to spell it all out, but knowledge of amazing algorithms like LP is scattered over departments of Engineering, Computer Science, Math, Operations Research, Statistics, AI and others. The point is simple: the world of useful algorithms and modeling techniques is vastly greater than ML and AI.

    The Natural Sequence

    There are dozens and dozens of methods that can be used to analyze and extract value from data, which after all is the point of ML and AI (and, by implication, all the other great algorithms). As I described in the prior post, there is a natural progression or sequence of methods, which roughly follows their order of discovery and/or widespread use. Success usually comes from using the methods in the rough order of their invention as you climb the mountain of understanding from simple and obvious (in retrospect) results to increasingly non-obvious and subtle results.

    I often see the following reaction to this concept, rarely articulated but often acted upon: “Why would I want to waste everyone’s time playing around with obsolete, outdated methods, when I’m an expert in the use of the most modern ML and/or AI techniques? I’m sure that my favorite ML technique … blah, blather, gobbledygook … will yield great results with this problem. Why should I be forced to use an ancient, rusting steam engine when I’m an expert in the latest rocket-powered techniques, ones that will zoom to great answers quickly?!”

    The unspoken assumption behind this modern-sounding plea is that analytical techniques, ranging from simple statistics and extending to the latest ML, are like computers or powered vehicles. With those things, the latest versions are usually WAY better than prior versions. You would indeed be wasting everyone’s time and money if you insisted on using a personal computer from the 1980’s when modern computers are many thousands of times better and faster.

    The trouble with this line of thinking is simple: the metaphor is inapplicable. It’s wrong! Analytic techniques are NOT like computers; they are like math, in which algebra does not make simple math obsolete – algebra assumes simple math and is built on it. Calculus does not make algebra obsolete – calculus assumes algebra and is built on it! And so on. Each step in the sequence is a refinement that is built on top of the earlier one. No one says, now that I know calculus, I refuse to do algebra because it’s old and obsolete. See this for more on this subject.

    So it does make sense to quickly apply simple methods to the data to get simple answers, and at the same time vet your data. No time is wasted doing this. On the other hand, if you jump straight to someone’s favorite ML technique, not only is it likely that inaccurate and incomplete data will render the results useless … you won’t even know anything is wrong! Because most ML techniques do nothing to reveal problematic data to the researcher, while simpler methods often do!

    Fundamental Analytical Concepts: Calculate it methods

    The simplest and most useful methods are ones in which you simply calculate the answer. There’s no modeling, no training, no uncertainty. These methods are highly useful for both understanding and correcting the data you’ve got. The basic methods of statistics like regression apply here, and so do the methods of data organization and presentation usually called OLAP, BI and dimensional analysis. The tools associated with a star schema in a DBMS apply here, which are roughly the same as pivot tables in Excel.

    Graphing and visualization tools are important companions to these methods; they help you really understand the numbers and see to what extent they make common sense and match reality. For example, you can see to what extent a doctor’s years of experience correlate with ordering tests or issuing prescriptions of a certain kind; or simply identify the doctors whose actions stand out from the rest. There could be a good reason why they stand out; wouldn’t you like to find out why? Maybe the doctor should be emulated by others, or maybe the doctor should be corrected; either way, you should figure it out.

    Until you’ve pursuing all lines of thinking based on these simpler methods, it’s premature to move on.

    Fundamental Analytical Concepts: Solve/Optimize it methods

    These are, IMHO, the gold standard of algorithmic improvement. When applicable, they tell you how to reach a provably optimal result! No training. It takes experience and judgment to apply the generic algorithms to a particular problem set, and sometimes the problem needs to be adjusted. But the results are stellar.

    First, you create an equation that measures what you’re trying to optimize. Is it fastest time? Lowest cost? Least waste? Some combination? Whatever it is, that’s what you’ll maximize or minimize as the case may be.

    Next, you determine the constraints. You only have so many operating rooms? This kind of machine failure requires a repairman with that kind of skills?  Then you put in the inputs and solve. While I’m leaving out lots of detail, that’s the basic idea.

    These methods, usually of the OR kind, have been applied with great success for decades. In certain fields and industries, they are part of the standard operating procedure – it would be unprofessional to fail to apply them. And you would rapidly lose to the competition.

    Fundamental Analytical Concepts: Train it methods

    The training methods all require sample data sets on which to “train” the model. Selecting and controlling the data set is key, as is avoiding over-training, in which the trained model can’t generalize what it’s been trained on, and thus loses most of its utility.

    Fundamental Analytical Concepts: Train it methods: white box

    What characterizes these methods is something incredibly important: what the model does can generally be explained in human-understandable terms, i.e., it’s “white box.” This has huge value, if only to gain acceptance for what the model does – but it may also bring up problems with data that can lead to further improvements.

    There are lots of ML algorithms that are in this category. All the decision tree methods are here, among them the very important random forest method, along with methods that arose within the field of statistics such as CART.

    0

    Fundamental Analytical Concepts: Train it methods: black box

    These methods can produce amazing results, and should be used whenever necessary, i.e., whenever earlier methods in the sequence can’t be used. The fact that the model is “black box” means that it’s difficult if not impossible to understand how the model makes its decisions in human terms – even for an expert.

    These methods include neural networks in all forms, including all the variations of “deep learning.”

    Fundamental Analytical Concepts: Rules

    Finally, I add the indispensable attribute of success in many practical systems: human-coded rules. These can be inserted at any point in processing, as early as enhancing the data before any methods work on it, and as late as modifying the results of final processing. While not often explicitly discussed, few practitioners with successful track records avoid the use of rules altogether. They may not be pretty or fancy or elegant – but they work, darn it.

    00

    More elaborate than sets of rules is the technique in AI of expert systems. This is a whole big subject of its own. Generally speaking, if you can get useful results from one of the sequence of methods up to and including white box training systems, you should do so. But important categories of problems can only be solved using expert systems, which ideally should be as white box as possible.

    Conclusion

    There is a broad range of analytic techniques that can be applied to a given problem. There is an optimal sequence for understanding the data and the problem. Going from one step in the sequence to the next, when done correctly, isn’t abandoning a method for something better, but first picking the low-hanging fruit and then moving on to catch tougher stuff. Prejudging the best technique to use before really getting your hands dirty is a mistake. Being a specialist in a particular method, e.g. “deep learning,” and confining your activities to that method alone can get you hired, paid and busy, but may lead to no useful results, or results far less useful than they could be.

  • Getting Results from ML and AI: 1

    We hear quite a bit these days about ML (machine learning) and AI (Artificial Intelligence), as the drumbeats of Big Data and Analytics fade away. You’ve got to be using these things to get great results and transform your business! You’d better round out the staff of Data Scientists you hired last year, and add appropriate numbers of ML and AI experts to the mix. Otherwise, you’re hopelessly behind the times, and you’ll eat the dust generated by the winners!

    Most efforts to apply these technologies fail. Not loudly, of course – no one admits failure. But after enough time passes with exciting results being right around the next corner, people stop talking about catching a glimpse of light from the end of the tunnel, and accept the fact that they’re digging a tunnel deeper and deeper, a tunnel to nowhere.

    Is there a way to get amazing value out of these exotic technologies? Yes. Decades of solid results show the way.

    Typical Failure Patterns

    In real-life cases, what happens all too often is that “data scientists” are called in to apply their magic. They take the available data and apply their favorite techniques. They may not produce results that are promising. If the results are promising, there is trouble applying them to the real-life situation. Or if they can somehow be applied, they don’t work or produced the promised results.

    The failures are the direct result of taking a naïve approach to applying these kinds of techniques in the real world. There are proven methods for attaining good results, but those methods are rarely discussed, for reasons I don’t understand. If they’re discussed, they’re lightly brushed over – instead of being given center stage as they deserve. Following are the ways to be successful with ML and AI.

    Build from the bottom up

    No one tries to sell a house based on how solid its foundation is; but a house with a crappy foundation will collapse. No one brags about their arithmetic skills when trying to get a job as an ML expert; but if you can’t add, how can you do ML? There is a clear sequence of learning ML and AI. You start with learning how to count; if you don’t know the numbers, you know nothing. Then you learn basic manipulation of numbers, like addition and multiplication. Then it’s on to algebra and the following stages. You don’t try to learn exotic techniques until you’ve mastered the more basic ones on which they’re based, and on which they depend. These are things that are nearly universally accepted.

    What’s not so common is applying the same sequence when analyzing any particular problem. The foundation of the sequence, the first step, is the numbers, the data. All too often, the data is incorrect and incomplete. If the data is bad, the results will be worthless.

    I’ve never found anyone who disputes this, once the issue is raised. But I also rarely find people who act on it, and take it seriously. Why? Among other reasons, it's a multi-trillion dollar issue.
     00

    I have worked with some of the most advanced people in the field, including someone who’s been the chairman of one of the top academic departments in the field. This person and his methods have been behind a few of the most widely used success stories. Here’s his “secret” for when he dives into a new problem: he loads the data into Excel and looks at it, first line-by-line, and then using functions and visualizations. Yes, I know, Excel is something accountants use and try to avoid. But it’s ideal for fast, visual analysis of data sets, and has some of the most advanced algorithms available as add-in libraries. Why would you start programming in Python when you can move quickly mostly without programming using a tool?

    The important thing isn’t the tool. The important thing is the activity – look at the data. Seriously look at it! Don’t just scan and move on. Understand it! What you’ll almost always find is that there are errors. Mistakes. Important stuff that’s missing. Graph it and look for basic correlations, and see if it makes sense. Make sure a true subject-matter expert is by your side.

    Then the fun begins. You have to fix the data. It’s the foundation of everything else. Without a solid foundation, nothing of value can be built on it.

    It’s also important to understand that this isn’t something mechanical, like spell-checking. What you often find is that really crucial data is missing, or that real important data can be added. This simple-sounding fact can be a project-maker. I have been involved with several projects in which the mundane-sounding effort of adding more relevant data has been the difference between failure and incredible, world-changing success.

    OK, your data’s pretty good. Time to dive into ML? No way!! Way too soon! We’re going to go in sequence here, applying the very most basic techniques to the data first.

    The point of all this is simple: you squeeze all the value you can out of a given “level” of technique before advancing to the next one. There are lots of reasons why this makes sense. The simpler techniques yield results more quickly than fancier ones. They tend to be larger and more obvious, which means the impact will be big. People will understand them, and so are more likely to buy into the changes needed to apply them in real life.

    I’m not going to spell out all the techniques you should apply and the proper sequence, but generally speaking, the order is the same as the order in which the techniques were discovered historically, which is the roughly the same order in which the techniques are taught in school. So you try simple linear regression before multi-variate, for example. And you always look and use visual methods, because a surprising number of the advances are often ones that people in the field know or expect, or that at least “make sense” to them.

    Finally, at long last, you get to use the fancy stuff you’ve been itching to use all along. But by then … your data’s in great shape. Your system is already up and running. People are already accustomed to change and the improvements that result from applying math. And interestingly, there may not be that much “juice” to be squeezed out of the system by then. Depending on your scale, that remaining juice may be tasty indeed, but it’s the icing on the cake.

    In applying AI, the pattern is the same, except that in addition to applying simpler analytic techniques, you may be writing common-sense-understandable rules by hand. Why not? It gets the job done, it’s simple and direct, and the AI can focus on that yummy icing.

    Other Issues

    We’re not done! There are a couple other major, overriding issues to be considered in order to get great results from these advanced methods. I’ll cover them in future posts. They are:

    • Most people have a favorite method, in which they have expertise and experience. That’s wonderful, except that there is a world of different methods, and many of them are simple inapplicable to certain kinds of problems – but great on others. Picking the right method (or methods) is absolutely key to success.
    • Closed loop. All too many projects run open loop. In my descriptions above, I sneakily assumed closed loop – that’s where the best feedback comes from.

    Conclusion

    Wonderful results can be obtained by applying modern analytic methods to real-world problems. But you have to choose: do you want an academic prize or do you want real-world improvements? Sadly, those goals don’t have loads of overlap. If you want real-world results, you should build your effort on a solid foundation of accurate and complete data, and move from simple to increasingly refined as you apply algorithms to it. If you do, you’ll see positive results fairly quickly, and those results will get better and better as you climb up the mountain of sophistication.

  • Are Cashless Payments a Good Thing?

    Cashless payments have been around for quite a while in the form of checks and credit cards. But even while physical cards are being re-issued with chips, there is huge innovation going on to bypass checks and cards, and go all-electronic. There is Apple Pay, remote sensing, QR codes and more. Lots of people are excited about a new wave of consumer convenience, most of it based on credit card technology. It's all good!

    Is it really ALL good? You never seem to hear about the downside of all this new technology. But there's actually lots of downside. Payments are a huge field. So I'm going to just point out a couple issues with credit cards, the basis of most cashless payments by consumers.

    The Volumes

    Let's start by getting a sense of how much money we're talking about. Look at this:
    000

    That's more than $5 TRILLION dollars in 2016 just in the US. How much is that? Well, for starters, it's more than the entire US Federal budget!! In fact, it's over a quarter of the US GDP.

    Merchant Discount rates

    This is an incredibly important subject — of which most consumers are blissfully unaware. The concept is simple. Suppose you buy something from a merchant for $100. When you pay cash, the merchant ends up with … $100! what a shocking concept! Well, it is shocking when you realize that when you pay with a credit card, the merchant will end up with less than $98, and often less than $97.

    That may not sound like a big deal. But what if you're running on thin margins, like many merchants, and are under intense competitive pressure to cut prices, have sale prices, etc. You can be doing well to end up with an overall margin of 10% on your business. This means that collecting $97 instead of the full $100 amounts to a margin reduction of … 30 percent!

    Merchants grit their teeth and take it. They have no choice. The giants don't care. But next time you go to a small shop or farmer's market and see a sign that says "cash only, please," you'll understand.

    This is a complicated subject, since the discount rate includes more than a dozen charges for various things, the largest of which is called "interchange." But it easy to boil it down. Consumers put about $5 trillion on their cards, which they have to pay — undiscounted. The discount is about 2-3%, which is between $100 and $150 BILLION dollars. A year.

    Some combination of issuing banks, acquiring banks and networks (like VISA) are splitting that money.

    Credit card debt

    Leading thinkers in payments love to talk about how consumer-centric they are, and how they're investing heavily to make using cards even more convenient for consumers. Aren't they just wonderful people?

    The magic of credit cards is that when you want to buy something, and you use a card to buy it, you can get what you want even if you can't afford it. Great concept, huh?

    Consumers in the US alone are burdened with about $1 TRILLION dollars of credit card debt. And it's not rich people who have the debt. It is overwhelmingly people with little to no net worth.

    What's worse is what they pay. Today's average of interest rates is around 19%. It hasn't been less than 15% in quite a while. So that means that consumers have to pay well over $150 BILLION dollars a year just to keep the debt current, not counting paying it off. Almost all that money goes to the banks that issue the cards, though various lawyers and collection agencies get their cut. 

    What cards are worth

    None of the institutions involved with cards care a whit about whether the card is physical or electronic. What they care about is protecting and preserving their more than $250 BILLION dollars a year in revenue. How much is that? Here is an excerpt of a list of the world's governments' revenue:

    00

    As you can see, the revenue up for grabs with today's credit cards in the US is about the same as the revenues of the government of Sweden, 15th out of more than 190 countries. Card revenues are worth, well, quite a bit…

    What this means

    What all this means is simple: the companies involved are living high on this revenue, and will go to great lengths to protect it. They will spend big on fancy consumer features — so long as those features are based on the card infrastructure.

    Why this matters is that there are incredibly inexpensive and fast methods of transferring money that are not based on the card infrastructure. Some of these are in widespread use outside the US. In the US, the least expensive such infrastructure in wide use is ACH, which is a government-run electronic form of checks.

    There is no technical barrier to upgrading this network to be near-instantaneous. Technically, it could have been near-instant 20 years ago! But no one involved felt like doing it somehow, and even today, it's getting faster at an incredibly slow pace.

    There are a couple modern peer-to-peer payment systems that are based on this technology. While they're growing, they are tiny compared to card volumes, and the card insiders are working madly to control and contain this threat to their massive revenues.

    Conclusion

    I'm strongly in favor of consumer convenience. My goal here is simply to point out that the convenience comes at an astounding cost that is hidden from consumers who don't carry a balance on their cards, and is a constant temptation and on-going burden to those who can't afford debt, particularly the expensive kind you get with revolving credit cards.

  • Blockchain 1.01

    The post explains some of the most basic things about Blockchain. For more detail and analysis, see this and the links therein. For a basic definition, see the one at the end of this post.

    Blockchain

    “Blockchain” refers to widely varying bodies of code ripped from the Bitcoin source code, always leaving out the currency, and usually varying other key aspects of the code. These subsets of the base code are typically described as an "Immutable distributed ledger" or some such. Blockchain implementations are often created to respond to some crucial defect of the base system for a particular application. Such implementations wander far from the features that Blockchain is supposed to have in the minds of most people.

    Ledger

    Blockchain is most often described as a distributed ledger. The fact that it’s a “ledger” refers to the fact that blockchain is an extremely primitive DBMS that lacks a query language. As a “ledger,” about all it can do is support data writing and reading. Not only does it lack a query language, but it’s effectively a key-value store. This means that you better know the key of the ledger entry you want to read or update; without this, you are totally without recourse. Some anguished early investors in Bitcoin discovered this when they saw that their original investments had grown into the equivalent of millions of dollars, but they couldn’t access the Bitcoin because they have lost their key. Many others have lost their investments due to other security disasters.

    As a result of this glaring deficiency, a couple resourceful people I have met who are building blockchain applications use a relational DBMS such as Postgres side-by-side with the Blockchain, copying all the data put into the Blockchain into the DBMS, along with the additional information required to make it a practical system. Why keep the Blockchain? Simple: who would invest if the venture didn't incorporate the hot new solves-all-problems miracle technology?

    Distributed

    Many Blockchain applications depend on the fact that it’s a “distributed” ledger to justify using the technology. Unfortunately, as implemented for Bitcoin, the fact that the ledger is distributed is a painful but necessary consequence of the design goal of having no one in charge of the system. It's a necessary evil! The distribution is due to an unknown number of “miners” who enforce the simple but computationally intense rules of the system; there may be over tens of thousands of miners, maybe even more than 100,000, no one knows. In the Bitcoin implementation, distributing the ledger in this way leads directly to the ten minute time typically needed to complete a transaction, and an unsatisfactory implementation of standard database ACID properties and the problematic CAP theorem for distributed databases.

    Many Blockchain implementations attempt to overcome these problems by using common-sense measures that effectively eliminate the key features of Bitcoin. The most typical compromise puts the Blockchain under the control of a single organization; that opens it to the same kind of insider hacking that has broadly affected large commercial and government organizations. See this, for example.

    Many users seem to think that being “distributed” is an advantage that makes Blockchain applicable to solving long-intractable problems involving related data in different locations managed by different organizations and systems. The fact that Blockchain can be implemented as a distributed ledger is completely irrelevant to solving problems of this kind. Blockchain's implementation of data distribution is vastly less effective and efficient than proven methods, and its general deficiencies make it far less useful than standard DBMS’s in such applications.

    Immutable

    Blockchain gets sold based on the notion that it’s “immutable,” implying highly secure. In the context of Bitcoin, it’s true that the only way an outsider could come in and change the ledger would be to organize a conspiracy of the miners. However, even with the incredible security provided by an army of miners, people have still lost their investments in Bitcoin by a variety of prosaic means, such as corruption or dysfunction in the wallet or system that represents you for holding or buying/selling bitcoin. There have been a number of such incidents that have been publicized, and more that have not.

    The second you move from Bitcoin to Blockchain, the level of immutability plummets to little more than rhetoric. This is because the main security in Bitcoin is provided by miners, who are incented to keep the system secure. Once the miners are gone or under the control of a central organization, the usual methods for subverting software come into play, only without the layers of security that have prevented central bank systems from being hacked, at least so far. The typical Blockchain implementation is less immutable than a properly managed RDBMS with an appropriately replicated and secured log.

    Given the widespread computer security breaches that have taken place, and will continue to happen, it's comforting to imagine that having an "immutable" distributed ledger would solve the problem. However, what the Blockchain does is replace the database in a normal system. So the question is, is the pervasive lack of security in today's enterprise systems due to vulnerable databases, or some other reason? The answer is clear: the database is the MOST secure aspect of today's systems, and security in it is rarely breached, and in no case of a properly administered one of which I'm aware. The security breaches are always "top down," i.e., the hacker gets into the system as though he were a legitimate user by one of a wide variety of means, and sometimes is an authorized but crooked user. Then he sucks data out of the system. The DBMS has NOT been breached in this kind of attack, and Blockchain is equally vulnerable to such exploits. And that's not even getting into the ultimate attack, which is an insider modifying source code, which has been the source of some of the most inventive banking attacks. SInce the database is not the problem, Blockchain is not the solution.

    Smart contract

    A “smart contract” is a key feature of the Etherium cryptocurrency and several Blockchain implementations, and is the foundation of many of the proposed applications of Blockchain. A “smart contract” is the exact equivalent of a DBMS stored procedure. In other words, it’s a body of code buried inside the blockchain (DBMS) that goes beyond storing and retrieving, implementing any logic the creator chooses. Most “smart contracts” today are written in newly created software languages; for example, Solidity is the one used in Etherium. These languages and systems are relatively new, and have flaws that have already been exploited by hackers. Using the name “smart contract” is effective rhetoric, but that’s all it is. It’s an immature programming language running inside a seriously deficient database.

    If the version of Blockchain is run like a typical DBMS, with master/slave – in other words, going against all the principles that are supposed to distinguish Blockchain – then a Smart Contract could work like a stored procedure, though crippled by lack of functionality in the DBMS. Worse, there are serious unaddressed issues maintaining the basic ACID properties that any normal person would expect – things like either the money is removed from account 1 AND added to account 2, an absolute guarantee that BOTH happen or NEITHER happens, even in the face of machine failure. If it’s run like a “distributed ledger,” then you’ve got even worse problems.

    Computer History and Context

    Blockchain is experiencing a huge uptick in interest, involvement and investment. Many people and organizations are convinced that it is the key to solving many long-standing problems in various industries. Generally, the fact that there are huge problems of long standing and no clear path to solving them is correct. The thought is that, with so many obviously smart and hard-working people involved with software, the solution must be elusive because of a fundamental technology barrier. The solution must hinge on the arrival of some transformative new technology that will change the game. Blockchain!

    Sadly, the long-standing barriers have nothing to do with technical break-throughs, and everything to do with incentives and standard industry practices that inevitably yield such results. A study of computer history and the way things work in various industries — something that is rarely done — would make this clear. While it's a huge subject, here are some posts (including the many embedded links) that can provide a starting point:

    In spite of all the above, there's a way that Blockchain can be a big win in certain applications, when applied in smart ways by top-notch teams. There's a clear historical pattern to follow. Here's how.

  • Who Owns your Health Data?

    You own all the data about yourself, right? It's your blood pressure, your date of birth and your test results, after all.

    Forget about owning your data — just try to get your hands on it! You may think you own it, but the sad fact is that you don't possess it! (Remembering hearing about how possession is nine tenths of the law? It applies here in spades.)

    Health systems like to appear to be great folks who really care about you. These days, they all talk about how great their patient information portal (or whatever) is. Hah. Just try to use it!

    I've talked about how essential it is to have a personal EMR. I've sarcastically described a big break-through in EMR interchange with details from a personal example. And most recently why portability is essential for the EMR.

    I've just had another MRI and want to get my hands on it, so I can see the details. So I went to the hospital system website, and found right on the first page that they offer a patient portal that lets you get your information. Great! Here's what they say:

    Follow my health

    I took the next step and immediately ran into a bit of a problem:

    Gotcha

    Hmmm. I wonder which of these my doctor uses? And which one the MRI will be under?

    After lots more work, I finally got to the right portal that had my stuff. I dove right in. I found my list of test results:

    Results

    Hmmm, something is wrong here. There's got to be more than that! And what's that second item, the path report?

    The path report, somehow dated both 11/23/2016 and 1/06/2017 is actually a copy of a report that was done at my first hospital, Mount Sinai, shortly after the biopsy procedure on Feb 21, 2014:

    North path results

    Bad data!

    The first item is the most recent MRI I had done at Northwell. When you read the report:

    MRI 2017

    You find that Northwell has a prior MRI they did dated 9/14/2016, and two earlier "outside examinations" dated 9/24/15 and 1/22/2015, in other words, MRI's done at some other hospital they choose not to name. But they had those MRI's in their system! They just chose not to show them to me.

    Even worse, they admit that they performed a prior MRI themselves on 9/14/2016, and somehow it isn't made available to me, though it was in the system and available to the person who wrote this report less than a month ago.

    They've thrown away or (more likely) withheld from me, either maliciously or incompetently or some combination, a report they did and two reports sent to them by Mount Sinai, helped in part by the miraculous digital system I described here.

    Well, at least I should be able to go to the Mount Sinai portal and get the two missing MRI's, right? Let's see. Here is the top of the Mount Sinai patient portal test results page:

    Mychart test results

    Score: one out of two. The MRI from 1/22/2015 is available, but Mount Sinai has decided I don't deserve to have the latest one, 9/24/2015. The reason is clear. That is the very MRI that was transferred to Northwell by the amazing digital process I described. When I removed the MRI from Mount Sinai on the DVD, I guess it was no longer there, right? That's how computer data works, right, when you take it from someplace it's no longer in the original place? How can it be???

    Just in case, I decided to put in a request to the portal to supply the missing information. I was constructive, polite and provided the facts. Here was the cheerful, non-helpful response:

    11 mychart

    In other words, not my problem. As though the providers had anything to do with what information in the EMR makes it into the patient portal.

    The only possible explanation for all this madness is that the hospitals involved are using some cheap new software written by a bunch of hacks. They can't possibly be using any of the most famous, widely used, expensive enterprise software systems, can they?

    Let's see. Here's what you see at Northwell:

    Followmyhealth northwell

    And here's what you see at Mount Sinai: Mychart Epic Mt Sinai

    Oops. Two of the premier, widely used, non-cheap vendors. Each of whom is committed to modern, state-of-the-art EMR's with rich portals to enable patients to access their own data. Except on days that end in "y." Or when it's too hot or too cold outside. Or something.

    Oh, I know! Their backs must be against the wall, humping to get all those wonderful features into their EMR's so that doctors can spend even more time staring at screens instead of being bothered by patient contact. They must be stretched so thin, they just can't afford to get the work done, and all my sarcasm is just nasty and uninformed!

    Let's see. Here's AllScripts:

    Allscripts

    Lack of money is not the issue at Allscripts. Epic is privately held, so there's no way of knowing their profitability, but by every indication, they're doing just fine. 

    I guess it all comes down to the simple fact of who owns the data: possession being 9/10's of the law, the fact that they have it means that we patients can have what we fantasize to be our own data when we are able to pry it out of their cold, dead hands.

     

  • Blockchain: a Sailboat without Sails

    The interest and investment in blockchain continues its exponential growth. It seems there's no end to the long-intransigent problems it will solve!

    There are just a couple little issues. First and foremost is that the virtues of blockchain, such as they are, have little to do with the problems it is supposed to solve. Second is that other technologies are better than blockchain, often by a factor of 1,000 or more. Finally, the most-discussed blockchain virtues solve problems that don't happen to exist in the real world. Other than that, blockchain is great!

    Why then all the interest in blockchain, you might reasonably ask? You might as well ask lemmings why they are following the lead lemmings as they run off a cliff — it's what everyone is doing!

    Let's dig into just one aspect of this widespread delusion.

    What does everyone say blockchain is? Some combination of the following:

    • A distributed, immutable ledger
    • The foundation of Bitcoin, only without the currency

    We all think we know that Bitcoin involves cryptography, somehow making it safe, and that it electronically transfers data and value between distant parties. It's kind of like email in that regard, only so solid and safe that it works for money as well as a bank. Since blockchain is the "foundation" of Bitcoin, let's get blockchain to solve the many problems that involve getting data between places, and getting information exchanged. Let's go!

    The widespread picture is that Bitcoin is just an application built on top of blockchain, kind of like how Oracle Financials is just one of many applications that are built on top of the Oracle DBMS. The idea is that blockchain is a platform on which applications can be built, like an operating system, DBMS, etc. That's what everyone is pushing, but it's a bogus idea!

    The reality is that what we call "blockchain" is an artificial slice of an amazing piece of software that was artfully designed to solve a VERY hard problem — and each piece of it depends on the others in order to work properly. Once you take the currency away, the whole thing falls apart. "Blockchain" is part of a whole, just like legs are great transport mechanisms, but only work as an integral part of a whole body.

    Here is a sailboat:

    Foggy-Frank-Gehry-boat-01

    The sailboat is like Bitcoin. It's got a hull, rudder, masts and sails. It's moving. It's an amazing invention. All the parts are designed with the others in mind.

    Here is the same boat at the dock:

    FOGGY_at_BBY_6-2015_cPanbo-thumb-465xauto-11556

    The sailboat at the dock would be like blockchain if you removed the mast, the furled sail and all related ropes. There is no doubt that the sailboat is "built on" the hull.

    But what a ridiculous thing! What good is a sailboat without its mast and sail? There's no motor. It isn't even set up to be rowed with oars!

    The reason is simple: the hull of the sailboat wasn't designed in isolation. It was designed as an integral part of a sailboat. You know, one of those things with sails. And without those sails it's nearly useless!

    That's blockchain. A sailboat without the part that makes it useful.

    The currency is what drives the blockchain forward.

    Let's start by reviewing the capabilities banks have for normal currency. Let's remember that banks do DDA's (the thing we draw checks on and deposit checks into) just fine.

    While we often hear people complain about not having enough money, we never hear it's because the bank has somehow lost it — that their ledger has been falsified, so that money we deposited and thought we had disappeared. Bank ledgers are already "immutable." It's just not a problem.

    Paper checks can take a couple days to clear. But certified checks can be turned into cash immediately, and you can get cash from any ATM. You can also use a PIN debit card to instantly access or transfer money. Fast access to and transfer of money is widely available. That's not a problem.

    Banks have their electronic ledgers in multiple places, so that nothing is lost when there's a computer failure. Money can be moved between accounts in a bank in seconds. Banks already have distributed ledgers, and every modern DBMS supports replication of various kinds.

    So what about Bitcoin? Has it invented things that aren't needed or already exist? Bitcoin represents an amazing way of implementing the following:

    • A virtual currency
    • which no single entity controls
    • which incents "miners" to do the hard work of assuring that transactions are consistent and secure by rewarding them with newly-issued Bitcoin

    It is not an advantage that Bitcoin has a "distributed ledger" for much of anything: it's got a "distributed ledger" so that no one entity is in control of Bitcoin. That's it! The "mining" is an extremely clever mechanism to pay groups to keep the ledger, stored in thousands of copies in many locations, up to date and consistent. That's why there's a consensus mechanism in Bitcoin, so that new transactions go into the ledger only when most of the miners agree they should.

    In a normal bank DBMS, money is moved from one account to another in a fraction of a second. The transfer conforms to the traditional ACID properties of a DBMS, which assure for example, that the money is taken from one account and added to the other — either both transactions take place or neither takes place. Any widely-used DBMS can do this many thousands of times a second. In Bitcoin, it takes on average 10 minutes for a single transfer to take place, with total throughput a tiny fraction of any modern DBMS. This is the cost of having a "distributed ledger," which is required to meet the key design goal of Bitcoin of having no single entity controlling the currency.

    If you take away the Bitcoin, you take away the reward mechanism, and all you're left with is an insecure ledger that performs worse than a database on 50 year old computers. Worse.

    The fact that the ledger is "distributed" is supposed by blockchain advocates to solve the problem of resolving data in different places, like between stock trade systems and hospital EMR's. Ridiculous. Blockchain is distributed only to meet the requirement of having no one in control. Any modern DBMS supports replication, which can keep remote DBMS's in sync while performing thousands of transactions a second. This works today. Blockchain is not an advance in this regard. In the end, all parties to a transaction have to get their data to a single place, and then have a couple copies of that single place for fault tolerance. DBMS technology is optimized for this use case. Blockchain accomplishes the same thing … eventually.

    Each case in which blockchain supposedly solves long-standing problems can be proven false, which is why you always hear about the wonders of blockchain:

    • using the future tense, or
    • in a proof-of-concept, or
    • in an application that could have been built faster, cheaper and with better performance using modern DBMS technology.

    When will the blockchain insanity end? Here are my earlier thoughts on why blockchain is so hot. Here is my analysis on blockchain applied to the stock transfer problem. Here's the upside, how blockchain will in the end deliver great value (heh).

    I realize I haven't covered all the issues here, but when you're confronted by an entire mental hospital's worth of insanity, covering a single floor's worth of problems feels like a lot.

  • The IRS Anti-fraud Contract with Equifax is Good

    First there was the furor that Equifax was hacked, putting millions of confidential consumer records in criminal hands. Next there was the furor about Equifax's response. Now, our in-bred elites are outraged that the IRS would award a sole-source contract to Equifax for, of all things, anti-fraud! Outrageous! Equifax can't protect itself, and now our genius IRS awards them millions of dollars?!

    Sadly, this is yet another example of pathetically ignorant people expressing outrage about a perfectly normal and sensible action by the IRS that has nothing to do with Equifax's inexcusable malfeasance in protecting consumer data.

    Here's the story in a nutshell.

    Equifax

    Equifax is one of a handful of companies that gathers and sells information about consumers, much of it confidential. It is a public company that provides an essential service to its customers, which are predominantly credit-granting businesses. The core of their business is receiving detailed transaction data from banks, aggregating it and selling it.

    The Equifax breach and follow-ons

    As usual with breaches, it happened long before the company became aware of it. Also typically, the company waited a long time before making an announcement. Equifax executives added an extra unsavory twist to the events by selling stock before the breach was announced. The response of Equifax to the event, which included a bogus offer of consumer protection against identity theft, was awful. Extremely little hard-core information about the breach has been released.

    With this breach, Equifax joins the ranks of large institutions, private and government, that demonstrate their inability to keep their data assets safe. This is an ongoing scandal for which there are solutions, but none that major institutions care to use. I have written extensively about this.

    The IRS contract

    The IRS awarded a sole-source contract to Equifax for access to confidential consumer credit data — exactly the same kind of service that Equifax provides to most of its customers. Public figures were outraged!

    Capture

    If the IRS contracted with Equifax to help apply its expertise to keeping IRS data secure, the outrage might have been justified. But Equifax does not sell those services. What they sell is data, data that the buyer can use for many purposes — often for credit-worthiness, but sometimes to help verify consumer identity. The data was valuable for this purpose before the breach, and remains valuable today.

    The data that was stolen was, of course, a snapshot of what Equifax had at the time of the theft. Since then, data has continued to pour into Equifax, updating and augmenting the data it already had. By using this additional data in special ways, the IRS could improve its ability to prevent identity thieves stealing taxpayer refunds, for example. I have no idea if the IRS will be smart enough to do this (I suspect not), but in any case they need the data! Without it, the IRS will be even more vulnerable to theft and fraud than it already is.

    For the Senators to castigate the IRS for buying data from Equifax shows that they don't have a clue about computers and software — they don't care to know the difference between services and data, for example. But we already knew that.

    Clueless about Technology

    What this is really about is that most people, including business, government and media elites, are clueless about technology. Which doesn't stop them from pronouncing about it with great confidence. As it turns out, I wrote about before, using the IRS and e-mail to illustrate the hapless opining of public figures about Bitcoin and Blockchain.

    When things go wrong, "experts" are called in, and more money is spent doing the same useless things that let the problem happen in the first place. With the side effect that everything is even slower, more error-prone and vulnerable than it is today. The current round of posturing by public figures helps nothing. Sad.

     

  • How to Avoid Cutting off Breasts by Mistake

    I think we can all agree that if a breast is not diseased, it should not be removed. High on the list of modest, achievable healthcare goals should be avoiding misdiagnosing breast cancer, and avoiding cutting off healthy breasts.

    There are widely known methods and proven technologies that are available for achieving this goal. Instead of applying them, the leading minds in the healthcare technology field ignore them, and instead invest billions of dollars in exotic, cool-sounding AI software that may deliver benefits sometime in the future. Something is deeply wrong here.

    Yes, breasts are cut off by mistake

    Perhaps you think I'm exaggerating? Here is a recent cover from the NY Post:

    NY Post cover

    The woman felt a lump and had a biopsy. The diagnosis was cancer. She was sent to another hospital for surgery.

    Authorization

    Makes, sense, right? Someone should double-check. The rule in carpentry is "measure twice, cut once." When breasts are involved? Double-check should be the minimum; triple-check would be nice.

    Checklists

    Wait, hasn't someone gone into this already? For healthcare, specifically? Oh, yeah, there's that book written by that famous doctor on this very subject, Atul Gawande:

    Checklist cover

    He dives real deep into checklists in many fields, medicine in particular. Here's a partial summary by Malcolm Gladwell he puts on his site:

    Gladwell review

    The hospital at which the healthy breast was removed knew about checking.They wrote a procedure for it!

    What was the procedure? More paperwork. Another in the long list of things doctors have to plow through before they can do their jobs. If doctors actually took all the paperwork seriously, they'd get nothing done. In other words, the "procedure" was just paperwork created by lawyers and bureaucrats that makes life harder and helps nothing. As usual.

    Effective, automated workflow checklists

    What the healthy-breast-removing hospital did was not what Gawande had in mind, of course. He was thinking group meetings at which everyone goes through each step and signs off. Something meaningful. But that doesn't exactly apply here, since the double-check pathology exam should have taken place before surgery was scheduled. Thinking about it, there are many similar situations in which the checklists are spread out.

    There's a concept and proven technology that applies here: workflow. Think of a flowchart with worksteps, lines and conditional branches. Here's a trivial example:

    Workflow

    Think of the workflow as being driven not by paper diagrams like this, but implemented in software that interacts with EMR's and people, and tells them what to do.

    In this case, when the patient was referred for breast surgery, a proper workflow would have scheduled the pathologist to examine the cancer biopsy data before anything else happened. If the results were negative, as they would have been, the patient would have been notified and that would be the end of it. Only if the results were positive would the system allow breast surgery to be scheduled.

    Workflow systems assure that all the work that should be done is done. They eliminate paperwork by putting the checking and doctor sign-offs into the system. No lawyers. No bureaucrats. Near-zero errors. Corrections and enhancements can be made to the workflow without massive re-training and human error. Workflow is like a system-wide, automated checklist, with all the checklist benefits and more.

    Let's do AI instead!

    Workflow software exists. It's implemented in many industries, even sometimes in healthcare. It works. It reduces human labor and improves outcomes. Why isn't it ubiquitous in healthcare?

    Workflow works, solves problems and is understandable. AI, by contrast, is cool, the coming thing and few of the people yakking about it understand it. AI is exploding:

    AI market

    I see a recurring pattern here: people jumping on the new trend, the leading edge, the cool new stuff that "everyone says" is the coming thing, that none of them really understands. And ignoring relatively prosaic technology that works and can provide real benefits today.

    Conclusion

    Cutting off healthy breasts tells us everything we need to know about exotic new AI, which is at the far end of the innovation spectrum I've described for healthcare here, and illustrated here. Workflow is at the simple end of the spectrum: proven, works, understandable and understood, delivers immediate benefits. I guess that's why the leading thinkers ignore it and all the "smart money" avoids it. 

  • Government Cyber-Security tops the Oxymoron List

    Some cynics keep an ordered list of the "top" oxymorons. Long-term members of the list include "business ethics," "military intelligence," "northern hospitality" and "southern efficiency." While "government efficiency" has a permanent hold in the top ten, "government cyber-security" has leaped to the top.

    Cyber-security is a huge problem for large organizations in general, for deep systemic reasons, as I explain here. The US federal government is making a big effort to get ahead of the problem. By doing so, it is embarrassing itself and making things worse.

    The government is engaged in a giant effort to systematize cyber-security and train professionals in how to do it. Various claims are made about salaries over $100,000 and a need for over a million qualified people. I first encountered this effort in an article about a local community college that had received certification for their program:

    CCM

    This sounds prestigious: both the NSA and DHS are jointly behind it! This must represent the best training there is!

    So I looked into it a bit. Here is what the NSA has to say:

    NSA

    Here is what the DHS has to say:

    DHS

    I noticed a couple things:

    • Each organization has its own page to describe the joint program. Of course.
    • Each organization lists itself first in the description. At the NSA, the program was created by the NSA and DHS, while at the DHS, DHS is listed first. Of course.
    • Each describes the programs and its goals differently. Of course.

    This is bureaucracy as usual.

    Then I decided to find out about the program itself, so I clicked on the link at the bottom of the NSA site. Here's what I got:

    IAD not secure

    No kidding! That's why in the image above, I clipped to the top of my browser, so you could see the URL and see that I wasn't fooling around. This is exactly what I got by clicking on the NSA site shown above! Maybe it's just the NSA that's screwed up. DHS probably has a better link, since their website was updated less than two months ago. Nope! Same result!

    Makes perfect sense. The NSA can't keep itself secure. We already knew this from the Edward Snowden problems, and more recently from their role in the world-wide Wannacrypt virus attack. The DHS? Even government investigators have concluded that its cybersecurity efforts are worthless. So why shouldn't their joint website fail the most elementary security test?

    I dug and dug, trying to find what was actually taught, and what the cybersecurity standards and practices actually were. In particular, I was curious to find if there was any mention of the NSA's role in supplying the weaponry for the Wannacry attack by means of gross deficient internal cybersecurity. I was also curious to see what level of acknowledgement there might be of their problems.

    Result: a couple hours of digging resulted in amazingly little of substance. I'll just end with an interesting comparison.

    Remember the world-wide outcry about the guy being dragged off a United airplane? The CEO stepped up quickly and defended his employees. Then he took it back and abjectly apologized, and there followed a stream of discussions about how other airlines did it and specifically how United was going to change to prevent a repetition of the drag event.

    Compare this to Wannacry. Not just one guy, but tens of thousands of organizations, including most of the UK's NHS — resulting in massive patient issues, many of which were far worse than United's dragging event. Who was put on the carpet? Who apologized? Any word from the NSA about their security breach that greatly magnified the problem? Of course not! Don't be silly! This is an august government organization: no one apologizes, no one loses their job, and nothing changes. Got that?

  • Software Myth-Conceptions: Building Materials

    An amazing amount of discussion about software is governed by metaphors. This is not surprising, since software is so abstract, and invisible to most people. So a shared physical metaphor helps everyone communicate.

    The trouble comes when the metaphor doesn't match with the reality of software, as it often doesn't. In those cases, what usually seems to happen is that what everyone understands about the metaphor is assumed to be true of the software. Bad things then happen.

    One of the most common metaphors for software is a physical building. Buildings normally have architectural plans that are determined before ground is broken, with the architect being in charge. Then some digging happens and a foundation is created. From then on the framing is built, the electrical and mechanicals are installed, and the finishes are applied inside and out. Now the building is ready for use. Something everyone can understand, right? It makes sense that in software we also have architects, carefully planned and laid foundations, workers who fill things out according to plan, and finally finishes applied.

    Software is usually built according to the physical building metaphor. Things like Agile methodology don't change things very much — they just mean that instead of a complete plan of everything from the beginning, things happen in phases. The physical building metaphor for software is unacknowledged, unchallenged … and pernicious. Ignoring the metaphor is one of the main things that smart software people who follow Wartime Software practices do to beat the competition and win.

    Temporary physical buildings

    When we think of software in terms of the building metaphor, we nearly always think of "permanent" buildings, ones with poured foundations and all the rest. Of course, you darned well better have it all planned out right — altering a foundation is no easy job!

    What about the technology used to build temporary buildings? These aren't just tents for camping; some of them are extensive. A great example is the Frieze Art Fair, an international art fair held in NYC every year. It's huge; over 200 galleries exhibited last year. Here is a view of the extensive building from the air:

    Frieze matvel_cropped

    Here's a view of part of the exhibition area on the inside:

    2016 05 06 Frieze art fair 2

    There were even a couple of restaurants:

    2013 05 11 Frieze art fair 123108

    All this was assembled in under a week, used for the time of the fair, and then torn down and carted away. No poured foundations. No big deal!

    Architectural programs

    Architects use software to do their work today. There are varying levels of programs they can use. I used a semi-professional one to design a house. The program was amazing. It helped me with everything. Here's part of a drawing from my project:

    161

    See the steps in the lower left? The software made it easy. I didn't draw every step. There was a staircase module which did most of the work for me. Same thing with roofs, walls, windows, doors, appliances and pretty much everything. At any time, it could generate a list of the building materials required.

    In the physical world, I would need those building materials. But what if I were designing a building that existed only in a virtual reality world? I wouldn't have to go farther than the design — the design itself is all that would be needed! Now let's turn to software. Suppose I'm designing a UI and I want a footer on every page, and a logo on the top right corner. If I'm reasonably competent with software, I don't have to count the pages and place the logo and footer as many times as there are pages — I do it once, in a page master!

    In the plans for the physical house, while it's easy for me to pop 10 identical windows into the drawing, when it gets built, someone has to build and install 10 physical windows. In the software world, the realization is done by more software. In the physical world, if I want a change to all 10 windows after they're installed … ugh! Go back to all 10 and re-do them. In software world, I go to the footer or logo definition, change what I want, and boom! It's done!

    Yes, you can design software that doesn't work this way. Yes, all too many people do it the wrong way. They're stupid. What's important is that software can be designed the smart way, and it's not hard, so often enough, it is.

    The building metaphor

    So what does this mean? A couple things.

    Using the building metaphor encourages software to be built the wrong way. It encourages people to think about it inappropriately. The smarter you are in building software, the farther away from the physical building metaphor you get. The metaphor even gets you to think about roles badly — the "architect" is someone who does lots of planning but no real work. Lots of workers are needed to build, workers who follow exact instructions. Then checkers who walk around with the plans making sure the workers did their job. All of this is counter-productive! It leads to bad process and bad software.

    Let's knock the building metaphor down, bury it and move on.

     

  • Computer Security: Problems and Solutions

    Computer security problems keep piling up like dead bodies during trench warfare. Is the problem evil Russian (or Chinese or whatever) hackers? Of course! But there have always been bad guys and always will be. The real issue is, why do our corporations and government agencies steadfastly refuse to apply the known, proven methods that would prevent most of the problems?

    The problem

    When a jewelry store has been cleaned out, whose fault is it? Of course it’s the bad guy’s fault. But what if the people who run the store went out for lunch, left no one in the store, and left the doors and safes wide open?

    If a high-end clothing store is robbed blind, whose fault is it? Of course, it’s the bad guy’s fault. But what if store management failed to install the widely used tags attached to articles of clothing that are removed when you pay for them –and ring alarms if they leave still attached to the clothes?

    If a big store with lots of cash registers has a huge loss of cash stretching over many months, whose fault is it? Of course, it’s the bad guy’s fault. But what if the store failed to count the cash at the end of each shift, and paid no attention when a check-out person when from register to register during his shift, spending some time on each one?

    Computers are different

    Yes, computers are different than jewelry stores, clothing stores and big box stores. You may think that the computer security problem is way harder. Here’s what you might think:

    Data is invisible! When someone walks out the front door with a pair of expensive sneakers, you can see the goods if you’re watching. But if someone “steals” some files, no one can see the bits pouring down the tiny wires, or flying invisibly through the air. So you can’t see the stuff going out through your doors. Surveillance cameras don’t work for data.

    The thief can be remote! When a cyber-criminal does the bad deed, he could be right next door – or on the other side of the planet. He could be anywhere!

    This is what most people think. So they write piles and piles of regulations that mostly call for the computer equivalent of putting big walls around your computers and call it good.

    Use Computers for Computer Security

    Computers are complicated. Very few people understand even a modest portion of what goes on in modern computers. So most people are clueless, and the ones who have a clue tend to learn a specialty and stick with it. The incredible complexity makes change surprisingly hard – most of the change we see is due to the unprecedented growth in the speed of the underlying hardware, not advances in software.

    The vast majority of modern computer security is a rough translation of methods that have evolved in human warfare. The trouble is the bad guys have evolved and the defense remains stuck in decades-old, long-obsolete concepts.

    Here's a wild thought: let's use computers for computer security! Let's have them do the computer equivalent of how security is implemented in places where we really care about it, places like jewelry stores, clothing stores and stores with cash registers. Even libraries! And, please, people, how about attending to the simple stuff, like keeping your software up to date. If software had been up to date, the recent Ransomware attack would have been nearly a non-event. And worst part of it (the worm) wouldn't have happened if the agencies in charge hadn't let it walk out the door via the Manning method (see next section).

    A Big Fat Example

    Chelsea (f.k.a. Bradley) Manning was convicted in 2013 of releasing nearly three quarters of a million military and diplomatic documents and was sentenced to many years in prison. How it was it possible for a low-level Army private stuck in a remote outpost in Iraq to access all these documents and make them available?

    Here is a book:

    Private book

    … that tells you most of what you'd want to know about Ms. Manning, who was Mr. Manning at the time of her/his noteworthy actions. And how he (at the time) got away with it.

    The giant security hole that Manning waltzed through to do his damage was still open years later for Edward Snowden to do what he did. The security hole is wide open today at most government and commercial computer systems. It's the same hole that doesn't exist in places where people actually care about security, places like retail stores…

    By their actions, we can only conclude that the security "experts" in charge are criminally ignorant or just don't want to fix the hole.

    I'm sorry to report that I've seen no evidence that the situation will change any time soon. I've had occasion to talk with some leading security experts about the subject. I can definitely report that they exist on some planet — unfortunately it's some planet that is not planet reality.

  • The Value of Computer Industry Advisory Groups

    The value of the famous computer industry advisory companies is much less than most people think. Take the example of Gartner Group, whose exemplary customer service I discussed elsewhere. Gartner employs some highly knowledgeable, helpful people. Gartner wants you to think is that it's the place to go to find experts. And you can find them there, as I explain here. But as a company, Gartner and its kin are mostly formalized gossip services for big-company IT folks.

    The origins of advisory services

    Imagine, in pre-Gartner days, groups of IT execs getting together by geography or at industry conferences. They all naturally want to learn what others have done in terms of purchasing gear, because it’s expensive and things change a lot. As they exchange information, it’s clear that some are kind of behind the times, others have made similar choices to everyone else, a few are out there – with new stuff from the usual vendors, or with something new from a new vendor.

    Everyone wants to avoid career risk. There’s a strong tendency to reversion to the mean, i.e., doing what most of the others are doing. The tendency is strongest concerning vendors, and after that products within a vendor; e.g. “I always buy GM cars, when I get more money I’ll upgrade to a Caddy.”

    Gartner comes along with a deal: you tell me what your choices are and a bunch about your business, and I’ll put everyone’s choices together and feed back to you a better, broader-based version of the results of the networking and gossiping you used to do, so you don’t have to spend the time. Even better, I’ll put it in an authoritative package so that if you’re ever questioned about your choices by non-IT people in your organization, you have our endorsement to fall back on; e.g., “yes we had a disaster, but it happened to everyone who made the best available choice at the time.” What a huge win, and cheap for the value.

    For example, here's how they explain their most famous graphic, the "magic quadrant."

    Gartner quadrant

    What is it? Yes, there's some dressing, but a vendor only gets to the best place, the upper right, if lots and lots of people buy their products. It's little but a graphical illustration of products by popularity.

    This is the value they add. All their categories and analysis is just a prettied-up version of what everyone tells them.

    The value of advisory services

    So who actually listens to Gartner and follows their advice? Exactly the kind of buyers who avoid risk at all cost. They go to Gartner, who tells them what people like them are buying. So they can buy the same thing. And be safe!

    If you're running a big-organization IT operation, commercial or government the way most people do, that's exactly what you want. Your operation is, almost by definition, bloated, inefficient, over-the-top expensive and riddled with problems. Saving a few bucks or doing things a little better isn't going to get you promoted. When a disaster strikes, the fact that your decisions were "mainstream" tends to bullet-proof you against recriminations.

    In this context, Gartner is indispensable. Your decisions weren't just mainstream; you can point to Gartner — Gartner says they're excellent decisions. So there!

    Advisory services and innovation

    What if you run an organization and for some reason are really motivated to innovate? What if you're a hot young tech group building next-generation products and want to find buyers? What is Gartner's role?

    It's simple: if you really want to innovate, avoid experts at all cost. Period. Gartner and anyone like them included. Here is lots of detail and a juicy example of why.

    The people and organizations who value Gartner are least likely to buy from a 1% market share vendor with a product that’s ahead of the market. Who is most like to buy such a product from such a vendor? Someone who is in big trouble, desperate, or one of those thinks-for-themselves buyers at the front end of the Geoff Moore adoption curve.

    How about the small to mid-market? Similar rules. It's true that the buyers mostly don’t know or care about Gartner. Who cares about people who buy giant, expensive systems from HP, IBM, EMC and the rest? They focus on their business, and don’t give a hoot about products and vendors – though someone they’ve heard of would be nice. They’re like homeowners buying a heating/cooling system – mostly they’re buying from the local dealer, who they depend on to sell them something good and then support it. The dealer matters as much as the product. Gideon Gartner can just Giddy-up out of town, he doesn’t matter in this world. But at the same time, the buyers are still mostly failure-avoiding. They don't want innovation. They want works and cheap.

    Product Innovation

    There aren’t a lot of ways to break in with a new technology product. The ways I’ve seen are pretty much summarized in this blog post, which includes links to further material, including my book on the subject.

    The most important concept is simple. 95+% of the market will never buy from you. Ignore them and their gossip-aggregators. The vast majority of the big, Gartner-esque buyers won’t give you the time of day. You need to find some narrow market niche to focus your energy on, and dominate that tiny sub-market. Then you can grow from there.

  • The Ransomware Hack Attack: Lessons from the Experts

    The Wannacrypt ransomware attack is in the news because it's causing havoc world-wide in major corporations and government institutions. It's a textbook lesson in a number of subjects including (but not limited to): the hopeless incompetence of major institution management in general, and IT management in particular; the worthlessness of most people said to be experts; how dead simple most cyber-security is; the rank illiteracy of otherwise highly educated journalists about computing; the incompetence of our super-spook institutions.

    The authoritative New York Times

    Of course, we turn to the venerable NYT to get the facts about this important story. Here's the head:

    A1

    It's clear from the headline that the substance of the story is beyond the grasp of the generally super-bright Times authors (look at the bottom of the story, the author had lots of help), so we're going to have a treat: lots of experts!

    First some facts

    Let's start with a couple simple facts.

    The software in question is "ransomware" that users are tricked into running on their computers. The software is normally an attachment to an email message that an unwitting user (being kind here) clicks on. Once it runs, the software encrypts all the files on the computer, making them unusable. It then displays a helpful announcement of what it's done and how to get your files back. Here's a sample, taken from a nice summary of the situation:

    A2

    At this point, most people panic. Loads of hospitals in the UK were infected, for example, and mostly shut down.

    There's more! Once installed, the software probes all the computers connected to the same network, and tries to infect them with the ransomware using an error in some deep-in-the-guts thing normal users would never encounter called SMB. This means that once a single user in an organization has fallen for the bait and gotten the software, it quickly spreads. This part of the evil software is the "worm."

    Here's how the New York Times describes it:

    A3

    The underlying reality — the important facts

    Here are the most important things to know about this "audacious global cyberattack."

    • The ransomware spread by the usual means: emails to gullible users. To their credit, the Microsoft Windows Defender group quickly identified the problem and released an update that detects and removes it.
    • Only obsolete and improperly maintained Microsoft Windows computers were affected by the worm. Loads of systems were hit in hospitals running Windows XP, which Microsoft stopped supporting years ago. Supported versions of Windows that had installed all recent patches were not hurt. The relevant patch was released months ago. It is the worm part of the malware that infects servers, which is particularly harmful.
    • The bad guys are only charging $300 in Bitcoin to unlock your computer. That's a small price to pay to learn the lesson of keeping your system up to date!
    • If you really don't want to pay, all you have to do is wipe your machine and restore it from a backup. And then maintain it properly. I gather from all the furor that on top of using obsolete software, the affected sites fail to follow standard backup procedures.
    • The bad software itself has been publicly available for months, ever since being walked out of the NSA and published. It was only a matter of time.
    • It's not exactly genius software. A clever guy managed to do a simple thing that disabled the worm aspect of it worldwide! Details here from the guy himself.

    The Experts weigh in

    Since my regard for experts could hardly get lower, the NY TImes article changed nothing. But perhaps some examples might be amusing.

    I love this one:

    A4

    The price goes up to $600 if you delay. Let's assume everyone delays but pays. That means no less than $1B/$600 = 1,666,667 sites would have paid, if the experts are right. I checked the relevant Bitcoin accounts a few minutes ago, and the total had yet to exceed $30,000. Way to go, experts!

    I also love the choice given: "pay the digital ransom or lose data." Right. First of all, you're stupid because you're running obsolete software. Then, you can't restore from a backup? You deserve to lose all your data, and then your job — remember, we're not talking about naive consumers here, we're talking about richly paid computer professionals!

    Our next expert dares to be named:

    A5

    Here's the part I like: "Despite people's best efforts, this vulnerability still exists…" Of course it does! Updating Windows makes the problem disappear. You can't make people update their software — even though it's their job to maintain it!!

    "…experts said that computer users in the United States had so far been less affected than others because a British cybersecurity expert inadvertently stopped the ransomware from spreading."

    First, the guy who stopped the worm part was brilliant. He did what he did very much on purpose — he just referred to what he did as something "accidental," being sleep-deprived and modest. Second, what he stopped wasn't the initial infection into a site, but the spread of the worm once it was in. There were loads of US sites infected — the numbers are random, as you would expect from whatever email list the bad guys used, and the odds of professional users clicking on the attachment.

    The Times itself attempts to explain how the clever guy managed to halt the worm aspect of the malware. Completely screwed it up. Sorry guys, maybe you should stick to quoting experts who get it wrong instead of being obviously wrong yourselves.

    Then we have security experts weighing in:

    "Yet security experts said the [Microsoft] software upgrade, while laudable, came too late for many of the tens of thousands of machines that were locked and whose data could be erased."

    The Microsoft software upgrade was made months ago. It was not too late. It's the people responsible for the machines in question who are too late. If they let their data be erased it's on them — either pay up, wipe and restore from backup, or slink away in shame.

    As to the NSA that created and released the software in question: shame on you. You probably have yet to implement the measures that would prevent more of the same in the future.

    Summary

    When you read stories like this, it's natural to form a set of impressions, including:

    • There are mysterious hackers out there who are really smart and really bad.
    • The evil hackers can cause havoc.
    • All we can do is bring in experts and try to clean things up quickly.
    • Let's hope it's not worse next time.

    All these are reasonable thoughts for a layperson to have, reading the published material.

    The truth of the matter is closer to the following:

    • The richly funded NSA develops evil software and can't keep it secure, in spite of having a budget larger than most countries.
    • Opportunistic hackers comb through stuff and sometimes put together something that could make some money.
    • A shocking fraction of the big government agencies and corporations fail to follow the most basic computer maintenance procedures (keeping software up to date and making backups), in spite of spending megabucks on IT, and so are vulnerable.
    • The experts quoted in news stories are ignorant and/or wrong, along with the stories themselves.
    • The guy who stopped the worm part of the software from working was at the opposite end of the competence spectrum from all the highly-paid executives who weren't doing their jobs.
    • Most organizations will change nothing, so something very similar will happen again.

    Sigh.

  • Hospital Computer Disasters and Iatrogenic Disease

    I finally figured out why hospital administrators (1) don’t care that their computer systems go down, and (2) refuse to admit that they go down. It’s been driving me nuts. When the systems are down, patient health and lives are put at risk! Why the lack of concern and the secrecy?

    The answer is simple: people die in hospitals all the time! And on top of the people who would have died anyway, hospitals kill people all the time! Yes, “kill” – either through negligence (such as failure to sterilize) or positive error (wrong treatment). In fact, medical error is the fourth leading cause of death in the US! And how often do you hear about that?! In that context, a few more people dying because of an unavailable computer system is a drop in the bucket. And refusing to talk about it fits the pattern.

    Hospital Computer Failures

    Hospitals know they have to have computers. They know computers are complicated and they know software is hard. So nearly all use a couple simple, common-sense strategies:

    • Give high salaries to the top IT people to make sure they’re getting the best
    • Spend lots of money on computing for the same reason
    • Buy only what the best places are buying to reduce risk

    Sounds good, right? Sure, if you don’t know the facts. The result of these common strategies is consistent: most hospitals use incredibly expensive, generations-old technology that regularly fails. Comforted by the fact that they’re no worse than most of their peers, the hospital big-wigs nonetheless keep a solid lid on the problems and outages that are so severe, they would cripple any other business.

    Here are details about disastrous computer systems decisions in hospitals. Here is the story of my personal encounter with outages and keeping the problems secret. Since hospitals know they can’t count on their computers, in practice they’re all about the paper.

    BUT … when real people are dying, the “death” or severe crippling of a piece of electronics is, like, who cares?

    Causes of Death: the Public Information

    Tracking causes of death makes perfect sense. It’s something we want to know for each person who dies; it’s recorded on the death certificate. Here is the relevant section from my great-grandfather’s death certificate in 1923:

    Death

    It’s something we want tracked, so we can see trends and know where to put our efforts to reduce the causes of death. Here is recent data from the CDC, the US government agency that is supposed to track these things:

    FastStats - Leading Causes of Death 2015-07-01 12-36-04

    Heart disease and cancer are the biggies, right?

    Unfortunately, that’s not the whole picture. Hospital executives and government bureaucrats go to great lengths to make sure we don’t know the full story. People might get upset if they knew how many of them had relatives who died in the hospitals as a direct result of negligence and/or medical error! Important people could conceivably lose their jobs!!

    Causes of Death: the Information they try to Hide

    Even hospitals that have great doctors and special experts can have horrible levels of medical error. But the ones that are really bad? They’re really, REALLY bad. Here’s an example of a death-trap masquerading as a hospital in Brooklyn:

    Brookdale

    Unfortunately, it’s not just a couple of bad eggs that are the problem. We know it’s widespread and big, but unfortunately because of all the secrecy, it’s hard to know exactly how bad it is. We know it’s bad enough that the equivalent of a jumbo jet of people die of medical error every day:

    Jumbo

    This makes medical error the fourth-leading cause of death, shown here in terms of deaths per 100,000 in 2010:

    Jumbo 2

    Because the people in charge are so anxious to keep the facts from us, the only thing we can be confident about is that, whatever the reported data on medical errors is, the actual number is far greater.

    Then there are preventable deaths. Even when there are strict rules about reporting problems, hospitals break them. Reuters did some investigating in 2016:

    Reuters

    Among other things, they discovered that there was a superbug outbreak in a New Mexico nursing home the January 2014 that was supposed to be reported. There was repeated delay and denial. In the end, here’s what happened:

    Reuters dead

    What about the CDC? The agency that publishes the bogus statistics that ignores all the hospital-caused deaths? They don’t seem to care:

    Infections

    Things are getting worse:

    Superbugs

    A book has just come out by Dr. Robert Pearl, CEO of the Permanente Medical Group, responsible for the health care of 3.8 million Kaiser Permanente members. He states that medical error is directly responsible for more than 200,000 deaths per year, and another 200,000 preventable deaths as a result of substandard care.

    The theme is clear: suppress the information and focus elsewhere.

    Conclusion

    With so much energy going into suppression and denial of hospital-caused deaths of human beings, it’s finally clear to me why hospital administrators, bureaucrats and most of the “big thinkers” in the medical profession can’t give the time of day to concerns about computer systems. First, they use paper anyway. Second, it’s someone else’s problem. Finally, however bad the computer systems are, they don’t hold a candle to the ongoing death-and-destruction horror show of real human beings dying. Human beings who, except for the incompetence and ineptitude of some doctors, hospital staff and administrators, would be alive today.

  • Security Regulations vs. Security

    Maintaining the privacy of personal data is important is many industries. This aspect of computer security has received lots of attention from regulatory agencies, who have issued massive bodies of regulations that must be followed in order to achieve security. They've done it in fintech (for banks) and in healthcare, for example HIPAA. But there's a little problem: if you follow all the security regulations with absolute perfection, you can still be hacked. In many cases, following the regulations makes you less secure! Here are details.

    Are you going to be secure, or are you going to adhere to the security regulations? It's a choice no one should have to make, but it's exactly the choice forced on us by supposedly well-intentioned government agencies and industry groups.

    The Simple Solution

    There's actually a simple solution to this problem, though it's likely that a well-known really hot place run by things with horns and tails will freeze over before it will be accepted. Why the resistance? It's simple! It's inexpensive! It's marvelously effective! It enables innovation! And most of the people currently involved in the regulatory nightmare would be out of jobs. Sound like a good reason to find fault?

    I've described this amazing solution here. It's pretty simple. The regulations declare that consumer's personal information shall not be disclosed without their explicit approval to any entity, whether on purpose or as a result of error or negligence. Make the penalties severe and personal. Exactly how this is accomplished is up to you. And unleash a torrent of fast, effective security measures. Ones that work!

    Why should such a radical approach be tried? Well, among other reasons, the current approach to mandating security just isn't working. Period. Here's a reminder of just how bad it was a couple years ago (and it's not getting better):

    WSJ hacked

    What to do while the hot place remains hot

    Ok, that's a nice fantasy, but what do you do now? I'm going to be inspected for regulatory compliance, and I've got to pass! Here are the basics of a sensible approach to pass audits and achieve actual security at the same time.

    Information and systems security is incredibly important. No one wants systems to be down for any reason or information to fall into unauthorized hands.

    Creating systems that can evolve quickly, scale and survive systems failures, while maintaining good performance and near-perfect up time, is really hard, but is core to business success.

    • Small organizations have trouble maintaining speed, flexibility and quality as they grow.
    • Large organizations rarely are fast and flexible.

    Achieving “basic” security (things like firewalls and access control) is easy and normal. Basic measures protect against most threats.

    When you go beyond basic security, there are measures that organizations can take that are sensible, proportional to realistic threats, and supportive of the business. The measures are in the spirit of fast, flexible and high quality systems development that lead to business success.

    As organizations grow, there are pressures on them to “grow up.”

    • In development, organizations adopt industry-standard development processes, and see costs explode, time-lines stretch out and quality plummet. The “solutions” usually make the problem worse.
    • In security, organizations call in the experts, get audited, and change lots of things in order to comply with all the lawyer-written regulations. The net result is normally an additional big tax on development (making it even slower and costlier), with dramatic reductions in the actual security of systems and information.

    The painful fact is that complying with security regulations is not highly correlated with actually being secure, whether it’s keeping patient records confidential in healthcare or financial information secure in banking and commerce.

    • Smart organizations can recognize this and have two efforts: one to maintain actual security, and another to achieve compliance that is “good enough” to pass any audits that may be required.
    • Large organizations are more likely to have industry consultants and security specialists who see their jobs as being expert in the regulations and complying with them. This can create the illusion of security without achieving it, while placing an ever-growing burden on the business.

    There are many reasons why security regulations are ineffective at achieving their goal. They include:

    • Bad guys are always inventing new ways to be bad, and the regulations tend to lag far behind them.
    • The regulations tend to be voluminous, detailed specifications for how to achieve security rather than plain statements of what to achieve, which would leave room for innovation and automation.
    • You can have highly automated, more effective security measures than specified by the regulations and still fail to be in compliance.
    • Achieving compliance tends to be so hard and costly that there is usually little appetite for supporting actual security.
    • Meeting the regulations is often so burdensome that compliance in practice tends to be tardy and/or incomplete, further worsening the effectiveness of the regulatory approach.
    • Many regulations are written assuming (demanding!) the obsolete, document-heavy waterfall style of software, making compliance while running fast, modern iterative development nearly impossible.

    Truly bad things happen when you have actual security breaches, not failures of compliance. Therefore:

    • Top priority should be achieving actual security, because failing to do so can seriously harm if not kill the business.
    • Second priority should be running the business effectively and efficiently.
    • Then should come achieving enough regulatory compliance to stay out of the news and out of serious trouble. There are ways to accomplish this.

    Basics of Effective Security

    The most important aspect of security is establishing a culture of security throughout the organization. Security is not principally a technical issue—it is a cultural one. It doesn’t matter if you use 128-bit encryption on all of your “data at rest” if your implementation associate puts a million patient records in a Dropbox or a customer service representative emails those records to someone pretending to be an employer. High "walls" don't protect against the bad insider.

    As a company with a larger profile, you do need to check the boxes—for example in healthcare, having an assigned HIPAA security officer, do an annual HIPAA risk assessment, go through an SSAE-16, etc. But you should fill that role with someone who truly thinks about it from a risk-appropriate basis. This is no different from the idea that the ideal Director of QA doesn’t fundamentally think of themselves as the police; the best QA organizations are the small, highly automated ones that are highly integrated with the development team and who are equally talented.

    The ideal Security Officer is someone with the intellectual flexibility and horsepower to understand and mitigate the real security issues in the organization while also being able to speak the language of the auditors. Those people do exist (although they are rare)—but they think less about how they are responsible for “policing” the organization (which quickly leads to multiple layers of dysfunction, cost, and distrust) and more about how they can work as part of the organization to mitigate issues.

    [Thanks to Ed Park for this formulation!]

    Security is tough, particularly when the regulations are burdensome and ineffective. The approach and realizations described here are the only ways I've found to be secure while minimizing the regulatory tax.

  • Managing Software that’s Invisible to You

    Many things follow from the fact that software is literally invisible to most people. If you can't see it, how can you manage it? As I've detailed before.

    In some cases, normal people can, at least, see some evidence of the software, like for example when it's an app or a web site. They may not choose (!) to try it themselves, but it's not too much of a stretch to think that they could.

    But for some important classes of software, that's not really an option.

    As usual, Dilbert explains it to us:

    2016 12 20 Dilbert software finished

    Note that the big boss has no easy way of knowing whether the software is "done." If it were a new web site, perhaps he could navigate to it. But what if it's a new internal process management system? Even a new customer service routing system? You could walk into a service center and see lots of people on the phone. Just like before. Is the new software done? How would they know?

    Is the operating system upgrade done? How about the migration of certain applications to a virtualized environment? No one except the internal people would know the difference, unless of course a disaster happened! Hmmm… Given the track record of these things, maybe the fact that everything continued to run smoothly, with no break at all, means that nothing was changed!

    There is a solution. It's pretty simple. The solution is to acknowledge that knowledge of the substance of software is more important — by far! — than supposed "knowledge" of generic management techniques as taught in business schools.

    Coal mining

    Knowledge of substance trumps knowledge of "management" even when you can see what's going on!

    My great-great-grandfather was James Law McMillan. He was born in Wanlockhead, Scotland in the 1830's. After he came to this country, he went to work in the anthracite coal mines in the Pittston area of eastern Pennsylvania. He knew how to read and write, but had essentially no schooling. Here's part of his 1907 obituary:

    Coal

    He knew mining — because he was a miner. He was there, deep in the pits. No one can make the argument that mining and coal is invisible in the way software is, but James was promoted because he could get it done.

    Perhaps we could learn a little from the experience of the Pennsylvania company. Get people who can do the work to do it. When you find one like this: "with such energy did he labor and such an intelligent interest did he take in his work, that he soon acquired an excellent practical knowledge of mining," that person should take on management.

    In software, most managing is done by people to whom the software is invisible. And what happens too often is that the people who aren't good at programming become managers so they don't have to do it any more!

    Maybe those people who ran coal mines are worth emulating, at least in this respect.

Links

Recent Posts

Categories