Integration testing approach
Posted by PrydwenParkingOnly@reddit | ExperiencedDevs | View on Reddit | 18 comments
I recently wrote integration tests for a .NET API and I'm second-guessing the approach. Curious what others think.
The setup:
In contrary to our unit tests, the tests exercise the full stack from controller down to services. The only things mocked are external services (.e.g. external APIs, databases etc)
There are two assertion types:
- Before external API calls, asserting the entire request that has been built by the service
The responses of the external API calls/database are mocked, so that the behavior should always be the same.
-
Exact match on the result that the .NET API returns
test → controller (real) → service (real) → external API (mocked)
↑ ↑ ASSERT response ASSERT outgoing requests
Why?
We have API tests in postman, but they are slow and flaky. Slow because the external APIs have to do a lot of calculations, flaky because the external API is strongly dependent on dates in the datamodel, compared to the current date. Each API test takes 2-5 seconds, but each integration test only takes 100ms, can run in parallel.
On the other hand the unit tests do not cover enough functional requirements. A lot of our results are depending on how the units connect to each other. Let's say we want to replace automapper for something else, then the unit tests will have to be updated, but the new mappings could be totally breaking functionality we had before.
So, what do you think about this approach, is this a reasonable middle ground, or are we getting the worst of both worlds?
lastesthero@reddit
The "mock the DB too" pattern usually comes from fighting test speed, not from trust in mocks. But mocking the one thing your code most needs to touch kind of defeats the point — you end up testing wiring.
What actually worked on my team: spin up the real DB in a container (testcontainers / docker-compose), seed minimal fixtures, and share the container across the test run instead of resetting between tests. 100ms per test is achievable that way.
Mocking the external APIs makes sense though — you don't own those, and their non-determinism is exactly what flakes your postman suite.
GuyWithLag@reddit
You spin up a local HTTP server that serves mock request / responses, and mock the expected responses in that server. Then your whole system is exercised. See f.e. https://wiremock.org/dotnet/
PrydwenParkingOnly@reddit (OP)
I do understand mocking using an external API.
The downside of this is that it doesn't allow to control which mock object is returned for each test case. It can only determine this based on the request it receives. That makes setting up the mocks a lot more complex.
IIALE34II@reddit
Wiremock allows for this. We also have some tests running wiremock containers, using testcontainers. But this is quite expensive, and adds a lot to test runtime.
throwaway_0x90@reddit
wiremock is very light weight. Each testcase can start up its own wiremock under a specific configuration for that unique testcase.
gjionergqwebrlkbjg@reddit
3 seconds for an integration test is an eternity. You can run hundreds of tests a second off wiremock.
hipsterdad_sf@reddit
The instinct to mock the database is understandable but it's usually the thing that makes integration tests feel pointless. The whole value of an integration test is verifying that your code actually works with the real dependencies. When you mock the database, you're testing whether your service layer calls the right repository methods with the right arguments. That's essentially a unit test wearing a trench coat.
The approach that's worked well for me: spin up a real database in a container (Testcontainers makes this painless in .NET) and run each test against a fresh schema. The startup cost is a few seconds for the container, and then each test gets its own transaction that rolls back at the end. Your tests stay isolated, they run against real SQL, and you catch things like incorrect column mappings, constraint violations, and query performance issues that mocks would silently hide.
For the external API mocking, WireMock is solid as others mentioned. The key detail is scoping your mock server per test so you can return different responses for different scenarios without tests stepping on each other.
One more thing: if you find yourself writing elaborate mock setups that mirror your database schema, that's a strong signal you should just use the real thing. The effort you spend maintaining mocks that accurately simulate your database is almost always greater than the effort of spinning up a container.
PrydwenParkingOnly@reddit (OP)
This place is like friendly stackoverflow to me.
I’m going to test wiremock.
With the testcontainers for database, I am assuming that I’m seeding the database with some date, using code inside the test case. Is that assumption correct?
hipsterdad_sf@reddit
Yeah exactly. The typical pattern with testcontainers is: run your real migrations against the container on startup (so your schema matches prod), then seed per test with whatever data that specific test case needs. The key thing is each test should set up its own data and not depend on some shared seed script, otherwise you end up with the same coupling problems that made mocking attractive in the first place. Transaction rollback after each test keeps things fast and isolated.
Grim_Jokes@reddit
We use test containers and run the migrations against the database as part of the test setup. After that, each test gets its own database with a random name that will be populated with a subset of the data we need. We're trying to strike a balance between thoroughness and speed, so we opt not to load everything if we don't have to.
teerre@reddit
What exactly is this testing? This only makes sense if "building the request" actually does something that has invariants and you are asserting those invariants. Otherwise you're just testing if the assignment operator and constructors work (which I imagine they do in C#)
If the actual value of this API is on the returned data, that's what you have to test. Mocking it makes no sense. Of course, as usual, property based testing can overcome practically any issue. If you can build a simulator for the service that will explore "all" domain of returned values, then that's the best choice regardless of how snappy the external api is
If you really cannot fix the external service and you don't even understand the domain of values, you might as well drop tests altogether since testing mocks is just noise
gibdimkoofchji@reddit
This. If you’re mocking external services, all your testing is that your code matches the assumptions you’ve made about the external service.
That’s usually particularly useful and certainly not testing integration. To integration test, you must, as the name implies, integrate
Clyde_Frag@reddit
I would say it is more of a smoke test / end to end test if you're issuing a request to a cluster of services. It can be useful to make sure critical functionality is working, but difficult to maintain and debug in CICD.
Returning mocks from an external service is a happy medium where you can test logic without having to maintain that overhead, but I agree it's not without its flaws.
gibdimkoofchji@reddit
If you’re mocking the external service, it is not an integration test. You’re not integrating anything.
And as mentioned, you’re really testing almost nothing. Just whether requests are structured the way you think they should be which you do not even know if that’s correct because you’re not integrating.
And for ci/cd, you aren’t connecting to some random cluster or anything. You should be launching your dependencies in CI/CD to test against. Ideally using a test harness provided by the owner of said external dependency
Otherwise you’re just wasting cpu testing shit that isn’t useful
CodelinesNL@reddit
Mocking the external APIs makes sense. Mocking the database doesn't, and I think that's what feels off about it. I've run into this before. When the database is mocked, you're really testing whether the controller wires up to the service correctly. Useful, but it's closer to a wide unit test than integration.
The AutoMapper example actually proves it: swap the mapper and the mock accepts whatever the new code produces. You only find the gap when data hits a real database and something doesn't round-trip correctly.
TestContainers has a .NET port. Start a real Postgres (or whatever you use), mock only the external APIs. Same speed, same determinism, and you actually test what you think you're testing.
StuffCommercial9312@reddit
TestContainers is great, been using it for a lot of our build time component testing for some time now. Can spin up a "real" postgres, redis, kafka, etc to test closer to reality.
CodelinesNL@reddit
Oh absolutely. I've been using it in projects for a long time now and it's only gotten better. I recently moved to a company using TypeScript and AWS Lambda (before I was mostly working with Java/Kotling and Spring Boot) and not only is there a TypeScript runner too, there's also https://github.com/ministackorg/ministack now that supports pretty much all the stuff we use :) So that's my next big project :)
I remember back (in 2017 or so) when we were using EmbeddedKafka for integration testing that never ever worked right, also because of the horrible architecture of that library.
hooahest@reddit
You need to instantiate the databases as well. Spin up a docker image of mssql/mysql/mongo/rabbitmq/redis/whatever, plug the connection string in the correct place so that your integration test uses that and viola, you get real integration tests that run at 100ms but give you much more confidence.