Is the Scientific Method Valuable in Engineering?

In the novel Zen and the Art of Motorcycle Maintenance, Robert M. Pirsig states: “The purpose of scientific method is to select a single truth from among many hypothetical truths.” He summarizes the scientific method as follows:

(1) statement of the problem, (2) hypotheses as to the cause of the problem, (3) experiments designed to test each hypothesis, (4) predicted results of the experiments, (5) observed results of the experiments, and (6) conclusions from the results of the experiments.

The scientific method is held in high regard as a reliable process for acquiring and refining knowledge. It is arguably valuable in science, so people accept the scientific method as a universally good thing, and even aspire to it, without questioning its value. In engineering, fixating on a hypothesis, and working to confirm or falsify it, can lead us astray.

Question-Driven Engineering

In Bryan Cantrill’s excellent talk Things I Learned The Hard Way, he talks about effective debugging.^[1] Provocatively, he explains how people say they debug using the scientific method, but asserts this comes from an inferiority complex of software engineering not being a real engineering discipline—we try and sound scientific to desperately up-level what we are doing. He questions the value of the scientific method for debugging:

Do we use the scientific method? Does science use the scientific method? I would argue, no. We form a hypothesis, then conduct an experiment to attempt to contradict the hypothesis—bull shit, I don’t debug that way!
—Bryan Cantrill

He goes on to describe how a hypothesis is a guess and debugging with hypothesis-centric engineers drives him crazy: “I know this! It’s this! Oh, it wasn’t that, it’s this!”^[2] Instead of guesses, he encourages asking questions of the system:

What question do you want to ask of the system? Even if there is no way to ask it. Then get creative about how you ask it.
—Bryan Cantrill

He concludes by explaining how guess-driven debugging focuses on the things we already know, whereas question-driven debugging invites creativity.^[3] Perhaps the most valuable part: even when existing systems cannot answer the questions we want to ask, we can adapt them to leave artifacts to answer these questions, like adding a new metric, log message, or tool. Essentially, using engineering to answer those questions and improve the system over time.

Similar to debugging, the same thing can happen when we are trying to optimize the performance of a software program. It is not fruitful to guess where the CPU is consumed: “It must be serialization and deserialization! No, it’s the spin lock! Wait, maybe it’s inefficient memory management!” Instead, we can simply ask: “How is the CPU time being used?” From there, we can use a CPU profiler, like a Flamegraph, to measure how the CPU is being used. The path to improving application performance may be obvious, or we may need to control certain variables, modify the software, or use our intuition gained from experience to narrow down the possibilities. But there is no need to guess.^[4] We can experiment and measure, following a sound engineering methodology. Even once we have found an area for improvement, making it more efficient won’t necessarily be obvious. For example, serialization and deserialization of data can be expensive, but is often unavoidably necessary. Actually making a performance improvement may require even more questions and creativity.

Another situation in which engineers ask questions rather than form hypotheses is the discipline of continuous improvement. Continuous improvement is a technique popularized in the process industries and manufacturing to understand how to operate a process more efficiently by manipulating control variables using designed experiments while keeping the product within specification. For instance, operating a chemical reactor at a slightly lower inlet temperature might produce a product of higher quality, or a product of equal quality more cheaply. Again, no one is forming a hypothesis, they are simply manipulating process variables around control setpoints to better understand the process. They are asking a question of the system through controlled experiments—“What happens if I decrease the inlet temperature by one degree Celsius?”—and observing what happens.^[5]

Modern Software Engineering

I recently read the book Modern Software Engineering by Dave Farley. He explores whether software engineering is a real engineering discipline. He makes the argument that it is when we approach software development and operations using engineering methods that produce superior outcomes. Farley argues that as software engineers, managing complexity is one of the most important parts of our job. We do this by applying established techniques, including automated testing, and continuous integration and continuous delivery (CI/CD), that encourage modularity, cohesion, separation of concerns, and rapid feedback. I think it is an excellent book and an important contribution to our industry.

Farley defines engineering as follows:

Engineering is the practical application of an empirical, scientific approach to finding efficient, economic solutions to practical problems.
— Dave Farley

Farley quotes Richard Feynman on the scientific method and notes the importance of forming a hypothesis and performing experiments. On the surface, it might seem Farley has a different viewpoint than Cantrill, but after a more careful reading, Farley is arguing for a scientific approach—controlling variables, testing assumptions, and performing experiments—to iteratively solve problems and improve processes, not a rigorous application of the scientific method.^[6]

In addition, Farley puts a lot of emphasis on the importance of repeatable, automated testing to encourage modularity, cohesion, and a separation of concerns. Using testing as a feedback mechanism is further enhanced by using techniques for CI/CD, shortening feedback cycles and improving repeatability. But these tests are not being used to disprove a hypothesis or discover a fundamental truth of the system, they are being used to test the invariants of the system and provide the guardrails for continued experimentation, discovery, refinement, and iteration.^[7] Consistent with Cantrill’s statement on improving systems by making them receptive to the questions we want to ask, Farley notes that in order to write tests, we must control variables and establish measurement points where the results are visible and measurable. This encourages modular design, and incorporating these tests into CI/CD establishes a repeatable process which allows for comparisons and improvements over time.

Hillel Wayne also explored the question of whether software engineering is a real engineering discipline by interviewing people who have worked as software developers and in traditional engineering fields, like civil, electrical, mechanical, and chemical engineering. A summary of this work is presented in the talk Is Software Engineering Real Engineering? Wayne concludes software engineering is an engineering discipline, sharing similarities with traditional engineering disciplines, like being iterative, involving uncertainty, and using a mix of formal and informal methods. Software engineering is different from traditional engineering disciplines in that we can iterate much faster due to the flexibility of software and the minimal capital investment required for most experiments.^[8] Another advantage is deterministic code will always return exactly the same result, not just a result within certain tolerances. Farley also notes these two unique advantages of software engineering. Wayne reports the one thing people consistently said traditional engineering could learn from software engineering is the practice of version control. Farley would probably go even further and say version control in concert with CI/CD.^[9]

Most comparisons of software engineering to traditional engineering focus on production engineering, like the building of bridges, buildings, or widgets. More common to each engineering discipline are processes, not products. These can be processes refined in order to build products like bridges, buildings, and widgets, or processes in themselves, like in the production and management of chemicals, heat exchange, or electricity. Furthermore, engineering is often applied to incrementally improve an existing process rather than invent something entirely new. This highlights the importance of establishing a repeatable process, because without it, there is nothing to engineer. In software engineering, we can establish processes around version control, change management, testing, refactoring, modularity, scaling, and so on, and we can even combine aspects of these into a more comprehensive process we call CI/CD. Once we have a repeatable process, we can apply engineering to make it safer, cheaper, more scalable, more secure, and so on.

It is easy to write software to solve a problem. It is much harder to write software that can be adapted over time to accommodate new requirements, scale to meet growing needs, satisfy industry compliance and regulation, be maintained and modified by someone else, or retire the old thing and not just introduce the new thing.^[10] It is valuable to draw the distinction between a software developer and a software engineer. Anyone can write software, therefore, anyone can be a software developer. In contrast, a software engineer applies established engineering processes that result in superior outcomes over time. ^[11] Software engineers apply a scientific approach, through controlling variables and performing experiments, and by applying mathematics and statistics, but they don’t use the scientific method.^[12]

The Way of Zen

Although The Way of Zen, by Allan Watts, is not a book about science or engineering, it has a lot to say about both, including subjects like rational decision making and intuition. It also describes the limitations of the scientific method:

The rigorously scientific method of predicting the future can be applied only in special cases—where prompt action is not urgent, where the factors involved are largely mechanical, or in circumstances so restricted as to be trivial.
—Alan Watts, The Way of Zen

In science, contradicting a hypothesis can be hard because experimentation is challenging and because it can be difficult to control variables. In addition, a lot of effort often goes into explaining the errors in measurements.^[13] Engineering is different in that we are not working to prove or disprove a hypothesis, but rather iteratively and empirically improve a process by applying proven methods.^[14] Software engineering is unique in that a lot of our experiments are deterministic, perfectly reproducible, and cheap, fast, and easy to reproduce.

We feel that we design rationally because we base our decisions on collecting relevant data about the matter at hand. Yet we might ask whether we really know what information is relevant, since our plans are constantly upset by utterly unforeseen incidents. We might ask how we know when we have collected enough information upon which to decide.
—Alan Watts, The Way of Zen

An important difference between engineering and the scientific method is that in engineering we need to have a sense of quality—we need to be pragmatic, we need to understand what is good enough. Science is about seeking the truth and the search for the truth can go on a long time and without valuable outcomes. Engineering is about solving a problem or making something better, sometimes from first principles and sometimes empirically, but without necessarily understanding a fundamental truth. Engineering is about delivering a certain level of quality—making something faster, cheaper, cleaner, or more reliable—but doing so within constraints like cost, time, resources, physical constraints, or product specifications. In science, we form a hypothesis and design experiments to falsify that hypothesis.^[15] In engineering, we establish a process, and apply methods rooted in science, mathematics, and statistics, to ask questions of the system, run experiments, and improve the process.

I began with a quote from Zen and the Art of Motorcycle Maintenance describing the scientific method.^[16] Zen and the Art of Motorcycle Maintenance is ultimately a book about quality, not the application of the scientific method, and I will return to it in conclusion:

The difference between a good mechanic and a bad one, like the difference between a good mathematician and a bad one, is precisely this ability to select the good facts from the bad ones on the basis of quality...This is an ability about which formal traditional scientific method has nothing to say.
—Robert M. Pirsig, Zen and the Art of Motorcycle