Joel Spolsky's article The Iceberg Secret, Revealed is, for me, one of his most memorable articles. He frames the situation as follows:

"I don't know what's wrong with my development team," the CEO thinks to himself. "Things were going so well when we started this project. For the first couple of weeks, the team cranked like crazy and got a great prototype working. But since then, things seem to have slowed to a crawl. They're just not working hard any more."... Meanwhile, of course, the development team has no idea that anything's wrong. In fact, nothing is wrong. They're right on schedule.

Then he reveals the Iceburg Secret.

You know how an iceberg is 90% underwater? Well, most software is like that too—there's a pretty user interface that takes about 10% of the work, and then 90% of the programming work is under the covers. And if you take into account the fact that about half of your time is spent fixing bugs, the UI only takes 5% of the work. And if you limit yourself to the visual part of the UI, the pixels, what you would see in PowerPoint, now we're talking less than 1%.

That's not the secret. The secret is that People Who Aren't Programmers Do Not Understand This.

He also enumerates some entertaining corollaries.

If you show a nonprogrammer a screen which has a user interface which is 100% beautiful, they will think the program is almost done.

Joel's article focuses mainly on the lack of understanding managers have when they have not come from the software development ranks. This includes people managers, product managers, project managers, and even executives. I think the Iceburg Secret also applies to at least three other aspects of software development. The first is people who are not experienced in software development thinking that they are software developers because they've done some programming. The second is understanding the scope and cost of technical debt. The third is related to software developers wanting to play with the latest-and-greatest technology.

Understanding Software Development

There are many people working in the software development industry that have no formal education in software development. It is very common for people studying engineering, science, or economics, to be exposed to programming and enjoy it so much that they make it the focus of their career. I fall into this category. I studied engineering. I decided to pursue software development as an occupation after spending two years in graduate school constructing an elaborate software simulation and developing a number of programs for data analysis and modeling. The exposure to programming in these situations is usually pragmatic and procedural—writing some C, Python, or Matlab to manipulate data, generate empirical models, build simulations, or interface with hardware for data collection or control. Very little attention is paid to concerns like architecture or maintenance of the software over time.

These experiences translate well to writing a few proof-of-concept, hello-world applications for a small and growing company. Quick-and-dirty code, when successful, can take on a life of its own. Suddenly, important parts of the business are dependent on these applications and you start to think that you are a software developer. Because of the important impact you have had on the growing company, perhaps you are promoted and have great influence over how software is developed in critical parts of the organization. You know how to program a computer, but you don't understand software development. Utilitarian concerns will dominate the software. The architecture of a software system developed in this manner will be a Big Ball of Mud, as described by Brian Foote and Joseph Yoder:

A BIG BALL OF MUD is haphazardly structured, sprawling, sloppy, duct-tape and bailing wire, spaghetti code jungle. We’ve all seen them. These systems show unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated. The overall structure of the system may never have been well defined. If it was, it may have eroded beyond recognition. Programmers with a shred of architectural sensibility shun these quagmires. Only those who are unconcerned about architecture, and, perhaps, are comfortable with the inertia of the day-to-day chore of patching the holes in these failing dikes, are content to work on such systems.

Writing a prototype application, even if it is running in production and the business is depending on it, is tip-of-the-iceberg stuff in terms of professional software development. For example, getting from an application that handles a thousand transactions per second to one that can scale to millions might require an entirely different design. The majority of software development is taking an infant application and figuring out how to make it run reliably and be highly-available; meeting business goals in terms of performance and scalability; making the application secure; making it easy to deploy, update, maintain, and evolve over time; and doing so without compromising functionality or reliability. It takes years of effort to become good at these critical aspects of software development.

Evolving beyond a Big Ball of Mud takes discipline in terms of design, refactoring, consolidation, unit and integration testing, security threat modeling, fixing defects, documentation, and so on. The 90% of software development that lies below the surface. A person who hasn't been exposed to these aspects will have difficultly appreciating them. Dave Farley believes that the vast majority of people in our industry have never seen what a good software project looks like and that if these people have grown successful careers doing the wrong things, they will continue doing the wrong things. This is what Erik Dietrich calls an Expert Beginner—a person that believes expert status has been reached and, thus, believes further improvement is not possible. If care is not taken to address the Big Ball of Mud and prevent the software organization from being dominated by Expert Beginners, the development team will rot, alongside the software.

Understanding Technical Debt

Technical debt is a metaphor, widely used in software development, to describe the work that would need to be finished in order to consider something complete. Carrying technical debt is not a bad thing, in and of itself. It serves a purpose, just like financial debt. Too much debt, poorly structured debt, or an interest rate that is too expensive, however, can become a serious threat to the viability of a product, or even a business as a whole. Technical debt is repaid through practices like refactoring, automating manual tasks, or developing characterization tests—activities that reduce software entropy. Foote and Yoder acknowledge that paying down technical debt is a necessity after creating a Big Ball of Mud:

During the PROTOTYPE and EXPANSIONARY PHASES of a systems evolution, expedient, white-box inheritance-based code borrowing, and a relaxed approach to encapsulation are common. Later, as experience with the system accrues, the grain of the architectural domain becomes discernable, and more durable black-box components begin to emerge. In other words, it’s okay if the system looks at first like a BIG BALL OF MUD, at least until you know better.

Non-programmers do not understand what technical debt is or what it costs. I think even for a software developer it is difficult to understand the full cost of technical debt, unless you've worked on a team that has had to carry a high debt-burden. It is easy to think you'll get to that refactoring, automation, or testing later on, but, of course, there is never time, and later never comes. Things are only temporary, unless they work. You don't appreciate the full cost of the debt until it becomes paralyzing.

I once worked on a foundational product for a company. The product had evolved significantly over the course of more than 20 years, but it also had over 20 years of accumulated debt. A lack of investment in refactoring and test automation meant that some components of the product had not evolved in years. The original authors were long departed. These components sat alongside new and continually evolving components. Despite making significant progress in developing tests and refactoring some legacy components, there was still a significant debt, and our team had a reputation for being very slow to deliver new features. The reason was this enormous debt. Engineering managers, even ones promoted from the software development ranks, could not understand the size of this debt, or the impact that it had on the team's ability to iterate quickly, and deliver new features. They couldn't even execute on plans to retire deprecated components, in favour of their replacements. Only software developers who actually worked on this team could understand the true cost of the debt.

I think this leads to another corollary to the Iceberg Secret:

For a component that appears to be working, a non-programmer will assume the technical debt is minimal, or nonexistent.

Even software that is "done" still accumulates debt. Frameworks evolve. Languages evolve. Business needs evolve. An aspect of the system that works for 10 concurrent users may not work for 1000 concurrent users. The analog is closer to something that you own that depreciates and requires upkeep, like a house or an automobile. If you don't invest in a bit of regular maintenance, just to get started working on a legacy component may mean you have to get up to speed on a programming language or framework that is not one of the main ones used by the organization, and you may need to sort out things like licenses for dependencies that haven't been managed for months or years. Even then, you still can't get down to work, because you have to write characterization tests just to figure out how the application currently works. If you've been asked to fix a bug or add a feature to a legacy component like this, how is anyone to know how long this will take, or what risks are involved?

If software developers who have not experienced technical debt do not understand its weight, how are decision makers in the organization that have not come from the software development ranks supposed to develop an understanding? Take, for example, the Product Owner role employed by many "Agile" teams. Ron Jeffries points out that it is the responsibility of the Product Owner to own the entire product backlog, including new features, defects, and technical debt.

Wait, what? You thought the Product Owner should just pile on more new stories and the team should magically fix defects in their copious free time? You were mistaken. As Product Owner, your job is to present to the team all the functionality-based backlog items that need to be done. New features, fixes to old ones: those are yours to decide upon.

If a product is lacking quality or has too much technical debt, this is as much the responsibility of the Product Owner as anyone. And this raises the question: can a Product Owner who does not understand the amount of technical debt, nor see how burdensome it is to the product, be an effective Product Owner?

Developers and Shiny New Objects

Joel describes the Iceberg Secret in terms of the understanding of non-programmers, but there is a case where I think it applies to software developers as well. Our industry evolves rapidly. It seems a new technology or application, with lots of promise, is announced every few days. It is easy to be distracted or overwhelmed by all of these choices. Two areas that are in a lot of flux at the moment are platforms for processing streaming data and software for service orchestration—essentially data center operating systems. Both of these domains are rapidly evolving, with a very diverse set of choices.

It is easy to be faced with a paradox of choice, to the point of being unproductive, ineffective, or even unhappy, constantly playing with shiny new objects that do 79% of what you need, at the expense of executing on your business goals, or developing a deep understanding of your problem domain. There is an opportunity cost, and we certainly need to strike a balance in terms of staying current and not becoming an Expert Beginner, as mentioned earlier. But playing with every new technology is, again, tip-of-the-iceberg stuff in terms of software development. Being an effective software organization often comes down to being disciplined—focusing on a few technologies and then getting really, really good at them. Even in terms of your own software development skills, I think you are probably better served at having an in-depth understanding of a few domains, rather than knowing how to write a hello-world, tip-of-the-iceberg application in every new technology.

I'll relate this point to another one of my favourite articles by Joel. When hiring software developers, he describes how you want to hire people who are smart and get things done. He says that knowing a lot of facts does not equate to being smart, or to getting things done. Similarly with knowing every new framework. Take some time to focus on a few domains. Develop an expert understanding of those domains and make a significant impact on the business. Let your career go through periods of expansion and consolidation. Foote and Yoder quote Daniel H. H. Ingalls when they discuss evolving critical software systems:

Probably the greatest factor that keeps us moving forward is that we use the system all the time, and we keep trying to do new things with it. It is this "living-with" which drives us to root out failures, to clean up inconsistencies, and which inspires our occasional innovation.

Seeing More Than the Tip of the Iceberg

Good software developers have an internal quality bar that they will not work below. Workmanship cannot be sacrificed for functionality. Good software developers will not work on software that is a Big Ball of Mud if it continues to be a Big Ball of Mud, despite their best efforts to evolve it into something more mature. Good software developers will not tolerate managers, especially ones who do not understand technical debt, asking them to cut corners, dictating technical decisions, never providing time for refactoring, and doing things right. Good software developers need time and focus to develop a deep understanding of their domains.

Foote and Yoder again:

Often, the people who control and manage the development process simply do not regard architecture as a pressing concern. If programmers know that workmanship is invisible, and managers don't want to pay for it anyway, a vicious circle is born.

We can't expect everyone in our software organization to have come from the ranks of software development. Our organizations need a diverse set of talents. To be effective, however, we need to ensure that the direction and decision-making of our organization is balanced with perspectives from people who can see the tip of the iceberg, but also the enormous mass that lies beneath the surface, out of sight. If not, the organization, and the software it produces, will be mediocre at best. At worst, we risk compromising the entire business.