For a long time, I have rejected integration tests as overly costly and with very little benefit. Friends of mine have a similar argument about unit testing. They say that it does very little and slows you down.
I'm still not sold on the "unit testing slows us down" position. The negative side effects people mention (like impeding refactoring) line up more with testing implementation details than with unit testing, itself.
However, in the modern era, I'm starting to come around on integration testing.
First, I'll lay out my reasoning for why unit testing is better:
- Properly specifying how each individual behavior works allows you to build a foundation of working behaviors.
- Defining behaviors relative to other behaviors (rather than as aggregations of other behaviors) stems proliferation of redundancy in your test code.
- How two or more behaviors are combined in a certain case is, itself, another behavior; see #1 and #2.
I still believe all of that. If I had to choose between only unit tests or only integration tests, I would choose only unit tests.
In addition to providing better feedback on whether or not your code (the code you control) works, they also help shape your code so that defects are harder to write in the first place. By contrast, when something you wrote breaks, an integration test failing unlikely as it's very hard to create exhaustive coverage with integration tests. Furthermore, even if an integration test does fail, it's not very helpful, diagnostically-speaking.
In addition to providing better feedback on whether or not your code (the code you control) works, they also help shape your code so that defects are harder to write in the first place. By contrast, when something you wrote breaks, an integration test failing unlikely as it's very hard to create exhaustive coverage with integration tests. Furthermore, even if an integration test does fail, it's not very helpful, diagnostically-speaking.
Yet, I don't have to choose. I can have both. As we've seen in previous posts and will continue to see, I can have one scenario be bound as both.
So what is it that integration tests tell us above and beyond unit tests. My unit testing discipline makes sure that I almost never break something of mine without getting quick feedback to that effect. What do integration tests add?
The spark of realization was a recent discovery as to why a feature I wrote wasn't working but the evidence has been mounting for a while, now. It took a lot of data to help me see the answer even though to you it may prove shockingly simple and maybe even obvious.
Over the last year or so, a theme has been emerging...
- A 3rd party layout tool has surprising behavior and makes my layouts go all wacky. So I have to redo all my layouts to avoid triggering its bugs.
- "Ahead of time" code fails to get generated and makes it so I can't save certain settings. I have to write code that exercises certain classes and members explicitly from a very specific perspective in order to get Unity actually compile the classes I'm using.
- A Google plugin breaks deep linking - both a 3rd-party utility and Unity's built-in solution. I have to rewrite one of their Android activities and supplant the one they ship to make deep linking work.
- A "backend as a service" claims that its throttling is at a certain level but it turns out that sometimes it's a little lower. I have to change how frequently I ping to something lower than what they advise in their documentation.
- A testing library is highly coupled to a particular version of .NET and seems to break every time I update.
- Et cetera. The list goes on...and on...and on.
Unit tests are good at catching my errors but what about all the other errors?
When you unit test and do it correctly, you isolate behaviors from one another so that each can be tested independently from another. This only works because you are doing it on both sides of a boundary and thus can guarantee that the promises made by a contract will be kept by its implementation.
That breaks down when you aren't in control of both sides of the contract. It seems like we live in unstable times. You simply can't count on 3rd party solutions to keep their promises, it seems.
This realization led me to a deeper one. It's about ownership. If you own a product, your customer only cares that it works. They don't care about why it doesn't work.
Telling them "a 3rd-party library doesn't function as promised and, as a result, deep links won't work for Android users. I'm working on it," sounds worse than just saying "Deep links don't work for Android users, I'm working on it." What they (or, at least, I) hear is "Deep links don't work for Android users. Wah, wah, wah! It's not my fault! I don't care about your inconvenience. I only care about my inconvenience. Feel sorry for me."
Even though you don't own 3rd-party code, to your customers, you may as well. You own the solution/product/game in which it is used and you own any failures it generates.
That extends well beyond whether or not the 3rd-party component/service works. It includes whether or not a component's behavior is represented correctly. It includes whether or not you used it appropriately. It even includes the behavior of a component or service changing in a surprising way.
So I finally realize the point of integration testing. It forces you to deal with the instability and fragility of the modern software development ecosystem. It's not about testing your code - that's what unit tests are for - it's about verifying your assumptions pertaining to other people's code and getting an early warning when those assumptions are violated.
Integration testing - whether it's how multiple microservices integrate or how all your components are assembled into an app - is essential. Just make sure you are using it to ask the right questions so you can actually get helpful answers:
"Is this thing I don't control properly functioning as a part of a solution I offer?"
