This pattern of integration testing applies to any RPC call, so the RPC calls
made by a backend server to another backend can be tested just as well as
front-end client calls. When we apply this approach consistently, we benefit
from smaller tests that still test correct integration behavior, and make sure
that the behavior we are testing is "real".
To arrive at this solution, I had to build, evaluate, and discard several
prototypes. While it took a day to build a proof-of-concept for this approach,
it took me and another engineer a year to implement a finished tool developers
could use.
Adoption The engineers embraced the new solution very quickly when they saw that the new
framework removes large amounts of boilerplate code from their tests. To further
drive its adoption, I organized multi-day events with the engineering team where
we focussed on migrating test cases. It took a few months to migrate all
existing unit tests to the new framework, close gaps in coverage, and create the
new tests that validate the mocks. Once we converted about 80% of the tests, we
started comparing the efficacy of the new tests and the existing end-to-end
tests.
The results are very good:
The new tests are as effective in finding bugs as the end-to-end tests are.
The new tests run in about 3 minutes instead of 30 minutes for the
end-to-end tests.
The client side tests are 0% flaky. The verification tests are usually less
flaky than the end-to-end tests, and never more.
Additionally, the new tests are unit tests, so you can run them in your IDE and
step through them to debug. These results allowed us to run the end-to-end tests
very rarely, only to detect misconfigurations of the interacting services, but
not as functional tests.
Building and improving test infrastructure to help engineers be more productive
is one of the many things test engineers do at Google. Running this project from
requirements gathering all the way to a finished product gave me the opportunity
to design and implement several prototypes, drive the full implementation of one
solution, lead engineering teams to adoption of the new framework, and integrate
feedback from engineers and actual measurements into the continuous refinement
of the tool.