The Black Liszt

Category: Data Science

Summary: AI Machine Learning Big Data Math Optimization

This is a summary with links to my posts about Big Data and the fancy algorithms and math that are variously called Artificial Intelligence, Cognitive Computing, Machine Learning, Big Data Analytics, math optimization, etc. I have worked as an insider in these fields for over fifty years.

Huge amounts of time and money are spent on these things, but getting practical, real-world results is not so easy. You can make things worse.

https://blackliszt.com/2023/03/making-things-worse-with-ai.html

How do you make things better? It starts with data.

https://blackliszt.com/2018/03/getting-results-from-ml-and-ai-1.html

Everyone knows that data is important. There’s even a field called Big Data.

What people call Big Data often isn’t nearly as big as people think.

https://blackliszt.com/2013/02/big-data-some-little-observations.html

Do the arithmetic!

https://blackliszt.com/2012/04/im-tired-of-hearing-about-big-data.html

For all the talk about Big Data, it’s clear that making sure that the data is accurate and complete is too much to ask.

https://blackliszt.com/2019/03/nobody-cares-about-data.html

Among other things, the status hierarchy in Data Science makes it clear that anyone with ambition needs to get as far away from the actual data as possible.

https://blackliszt.com/2019/03/the-hierarchy-of-software-skills-data-science.html

Without data integrity, your analytics is screwed. It’s worse when the data is about your health. The integrity of data in healthcare electronic medical records is a major issue.

https://blackliszt.com/2016/06/healthcare-innovation-emrs-and-data-quality.html

Even simple insurance provider network data is too often wrong, putting yet more obstacles between patients and the services they need.

https://blackliszt.com/2017/01/my-cat-taught-me-about-the-state-of-healthcare-provider-data.html

Even if all the data were perfect, no one seems to ask why exactly is big data better?

https://blackliszt.com/2015/07/fatal-flaws-of-big-data.html

Is Big Data delivering results? Here’s what happened with Big data and Hadoop at Yahoo.

https://blackliszt.com/2019/01/using-advanced-software-techniques-in-business.html

With all the money going to Big Data, people are ignoring the massive benefits of leveraging little data.

https://blackliszt.com/2016/08/little-data-vs-big-data.html

The few people who study computer history have noticed that Big Data is remarkably similar to what used to be called EDW, Enterprise Data Warehouse.

https://blackliszt.com/2015/10/big-data-and-data-warehouses.html

Same thing with a new name. Hmm. Could it possibly be that Big Data is just a fashion trend?

https://blackliszt.com/2013/02/the-big-data-technology-fashion.html

https://blackliszt.com/2015/03/big-data-the-driving-force-in-computing.html

At least for insiders, there's lots of humor to be found in data. There's even a book of the fun highlights.

https://blackliszt.com/2022/01/data-humor-book-by-rupa-mahanti.html

After data, you move to the amazing variety of techniques available, each of which has specific areas of applicability.

https://blackliszt.com/2018/04/getting-results-from-ml-and-ai-2.html

Even a narrow-sounding field like Machine Learning has incredible variety and lots of things you need to do until the algorithm can be effective.

https://blackliszt.com/2017/02/learning-machine-and-human.html

The simple concept of closed loop is essential to getting and keeping good results, but isn’t applied as often as it should be.

https://blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html

The application of the principles to healthcare has proven to be challenging, with high-profile failures. But there are successes.

https://blackliszt.com/2018/08/getting-results-from-ml-and-ai-4-healthcare-examples.html

I've discussed the application of the principles to fintech, with a focus on anti-fraud. The current leader in the field displaced the incumbent who was using neural network technology. Here is the background on HNC, which crowed about how it used neural networks to catch credit card fraud. Which it did. Sorta.

https://blackliszt.com/2016/06/how-blockchain-will-deliver-value.html

Here's the story of how the absolute winner of credit card fraud detection was displaced by a superior algorithm and concentrated on more relevant data.

https://blackliszt.com/2019/12/getting-results-from-ml-and-ai-5-fintech-fraud.html

Natural language AI has proved notoriously hard to make practical. Here’s an example of how it’s been done in fintech customer support.

https://blackliszt.com/2019/12/getting-results-from-ml-and-ai-6-fintech-chatbot.html

All too often, AI research is conducted in isolation. There is a great deal to do to assure that any results that are achieved can be integrated into production systems.

https://blackliszt.com/2022/06/how-to-integrate-ai-and-ml-with-production-software.html

Given all the issues in making AI work in practical reality, how likely is it that generative AI, the hot fashion in 2024, is likely to decimate the job market?

https://blackliszt.com/2024/07/how-many-jobs-will-ai-eliminate.html

As is true in most fields, in healthcare the money and attention tends to go to expensive, fancy methods requiring PhD’s instead of simple things that actually work.

https://blackliszt.com/2015/07/cognitive-computing-and-healthcare.html

https://blackliszt.com/2016/09/healthcare-innovation-from-washing-hands-to-ai.html

https://blackliszt.com/2016/05/healthcare-innovation-can-big-data-and-cognitive-computing-deliver-it.html

You don’t need AI or cognitive computing to discover or promulgate the new discoveries that humans make.

https://blackliszt.com/2015/08/human-implemented-cognitive-computing-healthcare.html

There are no technical obstacles to having computers do what doctors do. There is a clear path to getting it done. But the people in charge can't or won't make it happen.

https://blackliszt.com/2025/01/ai-can-automate-what-doctors-do.html

There are barriers to medical innovation, but there are terrible risks.

https://blackliszt.com/2025/02/can-ai-improve-medical-diagnosis.html

And then there's generative AI, which spews out expert-sounding language like a doctor. Is it better?

https://blackliszt.com/2025/07/will-ai-give-better-healthcare-advice.html

https://blackliszt.com/2025/07/chatgpt-covid-vaccine.html

There are many ways to spend lots of time and not get practical results.

https://blackliszt.com/2019/03/the-hierarchy-of-software-skills-data-science.html

Even when an optimization technique is perfected and proven in practice at scale, it can take decades for it to be used in other relevant fields.

https://blackliszt.com/2019/08/the-slow-spread-of-linear-programming-illustrates-how-in-old-vation-in-software-evolution-works.html

Impressive analytics performed by recognized experts are subject to getting the results the Expert wants.

https://blackliszt.com/2017/04/big-datas-big-face-plant.html

Corruption of data or process can lead to bad results anywhere.

https://blackliszt.com/2019/10/surprising-bugs-at-amazon-shows-how-ai-can-lead-to-disaster.html

For cynical definitions in the tradition of Ambrose Bierce of Big Data, Machine Learning, Deep Learning, Cognitive Computing and AI, see these.

https://blackliszt.com/2017/01/devils-dictionary-for-21st-century-computing.html

https://blackliszt.com/2017/01/devils-dictionary-for-21st-century-computing-2.html

https://blackliszt.com/2017/03/devils-dictionary-for-21st-century-computing-3.html

August 29, 2023
Summary: Software People and Management

Guess what – software people aren’t like regular people! There are differences in the ways they think and act, judging results can be difficult, and hiring and managing is challenging. This is a summary of my writing on the subject.

Skills, Training and Status

The education establishment has taken over the world of skilled jobs. They've convinced the world that getting a degree in Computer Science is the only way to learn how to be a computer programmer. They are wrong.

https://blackliszt.com/2025/01/how-to-learn-to-be-a-computer-programmer.html

When you get a degree in CS or learn a programming language, you’re all set, right? Actually, those things are pretty much the same as knowing how to hammer a nail when building a house. It’s a nice start, but there are levels and levels of skill beyond the basics.

https://blackliszt.com/2018/09/the-hierarchy-of-software-skills.html

There is perennial conflict between young and old in software, perhaps more than other fields.

https://blackliszt.com/2010/12/experience-and-youth.html

You might think that getting a degree in Computer Science prepares you to be an effective programmer, much like getting an MD prepares you to be an effective doctor. A comparison of the two makes it clear that CS education is sadly deficient.

https://blackliszt.com/2020/07/job-requirements-for-software-engineers-should-stop-requiring-cs.html

Does status in software correlate with your skills, kind of like how hitting more home runs gives you status in baseball? Sadly, no. One aspect of software status is that the more layers there are between you and real customers, the more status you have.

https://blackliszt.com/2011/10/status-in-software-silliness-and-stupidity.html

There are other dimensions, for example the more management layers between you and fingers-on-keyboard programming, the greater the pay and status. And there are others!

https://blackliszt.com/2018/10/the-hierarchy-of-software-status.html

There is a similar skills and status hierarchy in data science

https://blackliszt.com/2019/03/the-hierarchy-of-software-skills-data-science.html

One of the sad things about software is that writing great code and getting credit for it are largely unrelated to each other.

https://blackliszt.com/2020/10/elizebeth-smith-friedman-the-cancelled-heroine-of-cryptography.html

Programming skill and power in the programmer is far more important than the differences between programming environments.

https://blackliszt.com/2010/05/what-is-the-best-programming-environment.html

At least in some environments, high-performing nerds are highly valued.

https://blackliszt.com/2011/07/the-new-top-gun-is-top-nerd.html

Understanding Programmers

When you think about hard-core programmers, the word "nerd" comes to mind. In the poplar imagination, "nerd" often seems to mean guys who play lots of games. And have attitude. Of course there are such people. But the very best programmers, in my experience, don't match that description.

https://blackliszt.com/2012/02/most-nerds-are-introverts.html

Some programmer-nerds go beyond introversion.

https://blackliszt.com/2012/03/nerds-autism-deficiency-advantage.html

Many of the best programmers rub "normal" people the wrong way. They're not "team players," They refuse to follow industry-standard norms, etc.On the other hand, what's so great about being "normal?"

https://blackliszt.com/2025/07/whats-so-great-about-being-normal.html

Temple Grandin's views are always valuable.

https://blackliszt.com/2010/02/nerds-and-the-yakkityyaks.html

In the end, the best nerds like to get stuff done — and have fun.

https://blackliszt.com/2011/07/top-nerd-activities-work-hard-save-the-day-and-have-fun.html

https://blackliszt.com/2010/03/nerds-in-norway.html

https://blackliszt.com/2011/07/top-nerd-nerd-values-and-attitudes.html

And of course, normal people can react to a passionate desire to get things done in the best way possible as arrogance.

https://blackliszt.com/2012/02/when-you-call-the-programmer-arrogant-are-you-committing-libel.html

Hiring

Hiring software people is a challenge. There is widespread bad advice that interviewing for a software job should involve lots of chatting and relationship building.

https://blackliszt.com/2017/02/how-to-get-a-software-job.html

There are definitely better ways to select software people than the normal interview process.

https://blackliszt.com/2011/12/interviewing-software-people.html

The CTO of an organization is a special case. The requirements for the job are frequently impossible to meet.

https://blackliszt.com/2014/10/hiring-a-cto-the-impossible-dream.html

While CTO’s should be tops in technology, the best of them understand finances.

https://blackliszt.com/2012/10/cto-cfo-cfbco.html

Managing

Managing software people isn’t easy. Most managers tend to focus on process instead of substance.

https://blackliszt.com/2013/03/process-and-substance-in-software-development.html

https://blackliszt.com/2013/11/people-and-substance-in-software-management.html

Many companies have what's called a software architect. What should that person do?

https://blackliszt.com/2010/04/the-chief-architects-role-in-a-tech-company.html

If instead of just paying attention to abstract process managers paid attention to what the programmers are actually doing, things would improve.

https://blackliszt.com/2010/04/what-are-all-the-programmers-doing.html

One of the reasons software managers tend to focus on process instead of substance is that they have zero experience building software. Worse, it's literally invisible to them.

https://blackliszt.com/2016/12/managing-what-you-cant-see.html

As you can see from Dilbert and coal mining, you do a whole lot better managing something you've done or at least can see.

https://blackliszt.com/2017/04/managing-software-thats-invisible-to-you.html

Non-technical managers naturally want to find a way to feel better about what they're doing. Here are some of the favorite methods.

https://blackliszt.com/2015/06/software-problems.html

All too often, nontechnical managers make important decisions instead of software people.

https://blackliszt.com/2014/01/who-makes-the-software-decisions.html

When non-technical people do all the managing, the programmers who know stuff either leave or learn to keep their mouths shut in the face of stupidity and lie.

https://blackliszt.com/2024/06/well-trained-managers-create-software-failure.html

In other fields, it’s mostly people who DO the highly skilled thing who become managers of it. It should be the same in software.

https://blackliszt.com/2014/10/joe-torre-and-software-development.html

Non-technical managers think that MBA skills are the main thing that's required to be a good manager. Wrong.

https://blackliszt.com/2016/01/software-business-and-business-school.html

Here's how I learned from a course at Harvard Business School that the MBA teaches confident ignorance in terms of software.

https://blackliszt.com/2023/01/software-management-and-business-school.html

So with all this, should a skilled software aspire to be a manager? Dilbert makes an excellent argument.

https://blackliszt.com/2023/02/should-a-software-engineer-aspire-to-become-a-manager.html

It’s easy to focus on relationships instead of technical substance when managing.

https://blackliszt.com/2017/02/software-management-and-relationships.html

There is also the classic case of ego. It's been a big problem in other technology-based fields, and it's a big problem in software.

https://blackliszt.com/2012/07/a-lesson-from-joseph-lister-ego-the-killer-of-software-projects.html

https://blackliszt.com/2012/07/what-can-software-learn-from-steamboats-and-antiseptic-surgery.html

The way group management changes as the group grows can be destructive.

https://blackliszt.com/2011/05/how-great-software-teams-can-go-wrong.html

It’s clear from the pay scales that most companies value management much more highly than the people who write code. Unlike baseball, for example, where the people who play on the field get the highest pay.

https://blackliszt.com/2016/04/software-people-you-get-what-you-pay-for.html

Does having a CS degree lead to the best pay? Not as much as you’d think.

https://blackliszt.com/2015/05/how-much-is-a-computer-science-degree-worth.html

There are many ways otherwise good programmers can go wrong. One big one is the result of perverse incentives.

https://blackliszt.com/2015/04/software-problems-the-role-of-incentives.html

Programmers who are considered to be “stars” often have issues unique to software that hurt group efforts and are rarely recognized.

https://blackliszt.com/2015/04/high-iq-programmers-the-problems.html

On the other hand, the whole process of training and promoting the most talented programmers is broken. It can learn a lot from ballet.

https://blackliszt.com/2013/04/what-can-software-learn-from-ballet.html

Book

The above are highlights of what there is to learn about software people. For more, see my book.

https://blackliszt.com/2015/03/software-people-book-just-published.html

May 17, 2023
Data Humor Book by Rupa Mahanti

There's a new book out about Data Humor.

If you like data, you will be amused by this book. If you feel tortured by data, join the crowd — and read this book, it will relieve some of the stress. If you were wondering what nerd humor was all about, read this book — better to learn about nerd humor by getting the giggles.

The author searched far and wide for data humor. She stumbled upon a blog that had some bits she thought were funny — this blog, yes, the one you're reading now!

She contacted me to ask permission to quote me. After thinking hard about whether I should grant permission — for about a microsecond — I gave it. She asked me to check out a draft of the book. I guess she liked what I said because my quote went on the back cover and was the first of the quotes on Amazon.

"This is a brilliant book. The title says it's humorous. It's hilarious! But even more valuable is the sad-but-true insights it conveys about humans, lost and wandering in uncharted forests of data, anxious to escape."—David B. Black, Technology Partner, Oak HC/FT Partners

It's quite amazing how widely she searched for quotes, from places I never would have thought to look:

… book containing a collection of more than 400 funny and quirky quotes, puns, and punchlines related to data, big data, statistics, and data science, from different sources and a wide array of cultural figures, thought leaders and key influencers across the world- William Edwards Deming, Charles Wheelan, Brené Brown, David B. Black, Tim O’Reilly, Jill Dyché, Evan Levy, Gwen Thomas, George Mount, David Shenk, James Gleick, Jim Barksdale, Vincent Granville, Cathy O’Neill, Dale Carnegie, Martyn Richard Jones, Timo Elliott, Mark Twain, Phil Simon, Lewis Carroll, Oscar Wilde, Thomas H. Davenport, DJ Patil, Damian Mingle, Thomas C. Redman, Cassie Korykorv, Brent Dykes, Guy Bradshaw, Scott Taylor, Susan Walsh, Winston Churchill, Ronald Reagan, Arthur Conan Doyle, and many more.

Here's just one:

Data isn't information, any more than fifty tons of cement is a skyscraper.
Clifford Stoll (Stoll 1996)

Don't you need some light in your life? I promise, it's lighter in every way than fifty tons of concrete…

January 18, 2022
The Hierarchy of Software Skills and Status in Data Science
There is a striking hierarchy of skills in software, as I've explained here. When you dive into any particular aspect of software, you usually find that it's got a hierarchy all its own. Data science is a subject of intense interest these days, so in this post I'll explain some of the basics of the data science skills hierarchy.

A skills hierarchy is very much an insider's game. What most people care about is status. I talk about the basics of software status here. Remember, the skills hierarchy is a whole world away from the status hierarchy that most people care about. Don't confuse the two!

Data Science skills

First and foremost, it's important to understand the incredibly broad range of subjects covered by the term "data science." I attempt to explain the basics of the range in this blog post. You can be just amazing in one of those subjects, while being a neophyte in one that the outside world may consider to be "related," but which in practice is not just down the hall, but in a different building on a different campus.The general understanding of this range is SO pathetic that "data science" is typically managed as a completely independent, free-standing group. Which makes about as much sense as believing that a sous-chef belongs in something that isn't a kitchen. Or that everything that everyone does in a kitchen is basically the same thing.

Here's one cut at the hierarchy in data science, starting from the base:
1. Tool users.These are people who have learned how to use some software tool, or maybe a couple. Most "data scientists" fall into this category.
  - They don’t understand how the tools are built or any of the underlying software
  - They may know their tool, but aren't real clear on what the other tools are about, much less when you might consider using one.
2. Some have broader knowledge of tools
3. Very few have the sophistication to understand real data analysis, per my series of AI/ML; see this and the links in it: https://blackliszt.com/2018/04/getting-results-from-ml-and-ai-3-closed-loop.html
4. Of those, even fewer can understand the underlying algorithms and follow the latest research literature
5. Of those, even fewer can make real algorithmic advances and implement them as tools and deliver practical value.
6. Of anyone who can do all of the above, it is rare to meet anyone who can address and solve deep tool-level problems in other domains needed to make their code practical in the real world.
7. Finally, it is extremely rare to meet people who, in addition to their deep and broad prowess in data science and relevant skills needed to make it real-world practical, have similarly deep skills in an associated domain needed to make the data science fully effective for a business.
The issues of this hierarchy are compounded by the usual over-selling by people who are good promoters but little else, and corporate/government big-wigs who don't want to be bothered with details, but are keen to be seen as "doing something" on such a high-visibility topic. Getting results that make a difference is pretty low on the typical priority list.

And then of course there are the "data scientists" themselves, who most often are sincere people who are trying to do a good job as they've been taught to do it — mostly by professors and others who have no idea what real-world success looks like, much less how to bring it about.

Finally, there is the usual "manage something that's invisible to you" phenomenon I have often discussed in this blog, which leads to so much dysfunction and so many wonderful Dilbert cartoons.

Conclusion

People talk as though "data science" were a thing, with the usual kind of hierarchy based on level of management and/or "experience." Those typical patterns of hierarchy just don't cut it for understanding what's going on in data science, just like they don't cut it for understanding software development. We will continue to see waste and dead-end efforts until we at least make a start at making our understanding of data science more sophisticated, and aligned with the facts on the ground.
March 12, 2019