← Back to writing

Notes /

So what is engineering anyways?

We call a lot of people engineers these days. Software engineers, data engineers, machine learning engineers, platform engineers... the list goes on. But what exactly makes the work they do engineering? Is it writing code? Building something complicated? Using maths? Having "engineer" somewhere in their job title?

If I write a script that gets a job done, have I engineered something? If that script then becomes important enough that hundreds of people depend on it every day, does that change anything?

I've been thinking about this for a bit, and I think one way to answer it is to forget about software for a moment and zoom-out to see to what engineering exists to solve.1

The physical world we live in is not black and white - measurements are imperfect, machines wear out, people make mistakes, and requirements change. But with all that, we still manage to build bridges that stay up, aircraft that fly, and software systems that people can depend on.

So, what are engineers actually doing then? Here's a first stab at a definition...

Engineering is the process of taking a problem in an uncertain and constrained world, and creating a solution whose behaviour we can depend on.

Engineering starts with something we want to achieve, like we need to get people safely across a river, or we need to process payments without losing money.

Notice that the starting point is never "build a bridge", "use Kafka", or "create a data lake". We start with the problem first, and this is the difference that separates engineering from technology. An engineer can know a tool very well and still decide not to use it if it's not suitable in solving the problem.

Similarly, engineering is not science. The National Academies' Reference Guide on Engineering makes the distinction that science generally tries to understand the world, while engineering starts with a goal and tries to create something that satisfies it.

For software, the difference might be something like understanding how a database behaves under concurrent writes, then deciding how to design a payment process so that two requests don't charge the customer twice.

So, we can say that the engineer is trying to get from a problem to a dependable solution.

The thing about reality...

If the world behaved exactly as expected, engineering would be much easier. Imagine designing a bridge where every vehicle had exactly the same weight, the wind never changed, every piece of steel was identical, every measurement was perfect, and nothing corroded. We could make some very precise assumptions in such a world... Alas, that is not the world we live in!

Engineers therefore work with uncertainty. They try to understand enough of it to make reasonable assumptions, then design around the parts that can bite them. The engineers that solve problems in the physical world use tolerances, inspections, safety margins and tests to stop normal variation from producing unacceptable outcomes.

Software has the same problem, even though the code itself can give us the feeling of determinism. Ultimately, networks fail, dependencies become unavailable, users do unexpected things, data arrives late etc... and production is where our tidy, well-crafted program meets the messy world around it.

Software engineers we use things like database transactions to give us guarantees about how changes are applied. Schemas that constrain the shape of data. Timeouts to put a bounds on how long we're prepared to wait, and tests to challenge assumptions before changes reach production. At the same time monitoring tells us when the real system has shifted beyond the range of behaviour we expect.

The practical job of engineering is to create useful guarantees around uncertainty. Those guarantees don't make a system perfect though, but they do make it predictable enough to depend on.

Let's not forget about constraints!

Every real engineering problem has constraints. There is limited time and money, and we tend to inherit systems we can't easily replace. There may also be some regulatory requirements, performance targets, deadlines, or operational limits that constrain the work of engineering a solution to a problem. Additionally, requirements can also conflict, where making one property better makes something else worse.

A system can usually be made more reliable, but reliability costs something - a pipeline can run faster with more infrastructure, and a platform can support more use cases at the expense of more complexity. In data engineering, we can validate more properties of a dataset, but there comes a point where the checks cost more than the risk they are reducing.

When thinking of possible solutions, there usually isn't one correct design waiting to be discovered. There are multiple potential designs, each with different properties, and the job is to choose one that satisfies the important constraints well enough 2.

Google's Site Reliability Engineering material on risk expresses this very aptly (I think) for software. Reliability isn't something to maximise without limit. The useful question is how much reliability a particular system needs, given the cost of achieving it.

So, good engineering means understanding which properties matter, how much they matter, and what we are willing to trade to get them.

Learning by building...and breaking

Anyone who has operated a production system will recognise that some things become obvious after the system has been running for a while and real users, real traffic and real failures have had a chance to expose the assumptions made when building it. Engineers learn through the act of building and operating things. Failure is useful in the sense that when something breaks, we learn where our model of the system differed from reality. The engineering comes in in knowing what needs to be measured, and what needs to be invalidated to then pivot to a more suitable design. 3

Importantly, the point isn't that failure is desirable, as some failures can be extremely expensive. The useful part is the evidence it provides.

I like to think about it this way -

  • A test challenges an assumption before production does
  • Monitoring compares expected behaviour with actual behaviour
  • An incident shows us a failure mode that we misunderstood, didn't anticipate, or didn't protect against strongly enough

Final thoughts

When I'm looking at a system, I find it useful to ignore the technology for a moment and ask what problem we are solving for, and what people genuinely need to be able to rely on. From there, I want to know which constraints are real, which assumptions might be wrong, and what failure could cost us. Finally, I want to know how we will see the system behaving in production, and what we will do when reality disagrees with the design.

Thanks for reading.More from the blog ↗