How does your team decide what is the "right" amount of coverage?
Posted by secretBuffetHero@reddit | ExperiencedDevs | View on Reddit | 194 comments
What are the discussions? Do you make exceptions? Are mocks allowed? Is it a % by line? is it branch coverage? Are there other static analysis metrics that you consider?
One exception I made was - "don't unit test data transfer objects and don't count them in metrics"
Excellent_League8475@reddit
There is no right amount of coverage. A better practice is to track how many bugs are reported and how long it takes to fix bugs.
Important code should have tests covering edge cases. Code review should focus on the cases covered, not necessarily the coverage.
Bug fixes should have a test added (or fixed!) as part of the code change to prevent a regression.
_ezaquarii_@reddit
And if the team becomes sharp and competent and produce zero bugs with robust testing practice, we scrap all tests because they are useless. :)
dustywood4036@reddit
Is this a joke?
MattTheCuber@reddit
Yes
dustywood4036@reddit
I was hoping but who knows.
secretBuffetHero@reddit (OP)
this is the way
mmcnl@reddit
They are not equivalents. Test coverage is early feedback, bug metrics are late feedback. Prevention is always cheaper than curing.
Excellent_League8475@reddit
High coverage != high quality. Bug metrics actually tell you something about your quality. So if you're going to track something, make it a useful metric that can drive change.
Engineers need to use their critical thinking cap to determine the right *cases* for each code change. Not just just look at some useless metric spit out by a report that knows nothing of the domain.
oupablo@reddit
Tests should cover the naughty bits plus a little extra for decency. That's why they call it coverage.
Ok-Yogurt2360@reddit
I agree with the part where a bug fix should almost always result in added tests. If the test is nicely written it also functions as a form of documentation.
Excellent_League8475@reddit
Yep! Someone else in here mentioned tests are a form of documentation. I 100% agree.
secretBuffetHero@reddit (OP)
code without tests is legacy code though. corner cases aren't documented well
secretBuffetHero@reddit (OP)
interesting response
PuzzleheadedReach797@reddit
If you dont say "frequently", that bug easly found with unit tests, i think its rigth amount of coverage
danielt1263@reddit
What are the tests for? Are you doing TDD? Knowing the answers to these questions will tell you the answers to the questions you are currently asking.
secretBuffetHero@reddit (OP)
isn't TDD just a design method with tests as byproduct?
the question is how do You decide. what is important to your org?
danielt1263@reddit
Yes, TDD is a design method, but tests are not the. byproduct, they are the inciting incident.
That said, The questions about test coverage and whether you make a test at all is answered differently if you are using TDD vs some other reason for writing tests.
And the question still comes back to, "why are you writing the tests in the first place?" Nobody can know how much coverage is enough or what static analyses are needed until they have decided what the tests are for; the reason for writing them.
cheolkeong@reddit
TDD seems so joyless tbh. I don't care for how much it puts the focus on tests over source code. Like it sort of turns the source code into the byproduct.
Lords_of_Lands@reddit
That's kind of the point. You write the test first so you know the test properly fails. The new source code simply stops the tests from failing. It results in the minimal amount of real code with your requirements all written as tests.
cheolkeong@reddit
I know you’re saying it’s the point to focus less on source code, but it’s funny to read it as joylessness being the point. I definitely understand what the point of it is, I just think it takes out the joy and satisfaction of getting into a flow. I also think it makes writing the tests even less fun.
I still greatly prefer to have net negative contributions of lines of code. I still can apply various levels of questioning of what’s a good and bad test. I can apply code quality concepts to tests and separate concerns. You can do these things without making yourself write the test first.
The tests have more value for a refactor imo, and when the goal isn’t to change the functionality the tests would already have been written with TDD. So when you refactor you aren’t writing the test first and nobody dies.
I suppose the thing I just can’t shake is the idea of TDD being a punitive policy. The only time I worked with someone that was excited about TDD was really early in my career. But even then this was a manager that was just excited about coding in general and excited about the ideas behind TDD. On a team that bought all the ideas and still just preferred to write the new code first.
Lords_of_Lands@reddit
I'll agree with you that it isn't fun. I have to remind myself that I'm engineering the code and have to take pride in that.
cheolkeong@reddit
Glad we agree on a thing!
Have you gotten to a point where it doesn’t disrupt your flow at least?
I always liken it to those standardized tests with scantrons. It was always recommended that to answer all the questions on the booklet first, double check, then finally go through and fill in the scantron based on your answers. To maintain flow because filling in bubbles on the scantron was a context switch that used a different part of your brain.
Don’t get me wrong, I love an iterative approach, but I’ve never found flow doing TDD. I only ever found flow once I technically wrote all the tests I was going to need, at which point the TDD part is effectively over.
danielt1263@reddit
Got it, but that still doesn't answer the question. What is the purpose of unit tests in your code base? Why do you write them?
Maybe I should create my own post with the question.
recycled_ideas@reddit
TDD is a design methodology that involves writing tests first which naturally leads to high test coverage. The tests aren't a by product they're the starting point.
squidgyhead@reddit
This makes sense if your tests are simple enough that they get run every single time. If the parameter space of test choices is too large (eg full tests take a couple of days to run) then I don't see how TDD is supposed to work. I also like having a test infrastructure which isn't coupled to the code - it should have a broader view, allowing it to be more feature rich.
recycled_ideas@reddit
First off, if you haven't got a clue what TDD is, just stop commenting on it.
Second, if your tests take a couple days to run, you're basically not testing at all and just wasting your time.
Dry_Hotel1100@reddit
If you replace an asynchronous, stateful logic with a mock, all bets are off - especially in OOP and when you implement it in the typical OOP way.
A high percentage of test coverage is often used to impress the stakeholders, it may not test anything. In OOP, in many cases, due to injection, your SUT is not deterministic by design anyway, and tests that pass with a mock may fail with the real dependency.
So, better you start to think differently:
Treat your stateful thing "SUT", as a deterministic state machine, separate pure logic and side effects. Only mock the real side effects. When a SUT has a dependency, leave them connected, don't use a mock for the related dependency. Instead, connect with the real thing, preserving its pure logic, and only mock the real side effects of the dependency.
kitsnet@reddit
100% line and decision coverage in addition to functional tests covering all the requirements.
This code drives cars, so surely you don't want us to test less.
xFallow@reddit
You’re testing your log statements and config?
hooahest@reddit
I've had logs that threw null pointer exceptions due to lack of tests. That was not fun.
xFallow@reddit
That's fucked what library was that? Can't imagine throwing exceptions in a logging library
hooahest@reddit
it wasn't the library's fault, the code was trying to access a field from a null object for the string interpolation
xFallow@reddit
Ah gotcha I've had a junior break prod with the same thing lmao gotta love javascript
_ezaquarii_@reddit
Absolutely.
Config files are tested, because there are some configuration rules (like "no X on prod should be enabled") and stupid typos can spoil deployment.
If logs are used for operations - like triggering alarms - absolutely yes.
yetiflask@reddit
I do actually. Why not?
Oatz3@reddit
If the logs are business critical, you definitely do want to test that they are being output.
fabulous-nico@reddit
👏
xFallow@reddit
Fair enough
eatin_gushers@reddit
My code flies airplanes. Look up MCDC, 100% required.
Abject-Kitchen3198@reddit
That seems reasonable for the purpose.
_ezaquarii_@reddit
100%, no exceptions, including
main().There is a small % of low-value tests, like getters nor nops, but they are treated as a price for the benefit.
Substantial amount of tests tests are BDD-style with documentation, but overall we use palette of techniques depending on situation.
Benefits:
yourparadigm@reddit
_ezaquarii_@reddit
I call your bullshit bullshit.
daredeviloper@reddit
We aim for 100% coverage. Yes, some test code is dumb and useless.
MinimumArmadillo2394@reddit
I tend to aim for 80% or higher because not all code paths are covered.
As long as the happy paths and most likely sad paths are covered, we are good. Edge cases will always exist but its impossible to imagine all of them.
_ezaquarii_@reddit
Funny thing with that 'reasonable' threshold is that people hit it by testing 80% of the easiest code. The missing coverage is 20% most difficult and error prone.
That's we radicalized and forced 100% to create pressure to deal with complexity in the problematic 20%.
Dziadzios@reddit
The problem of 100% is inability to write defensive code/defense in depth. Some error checks are redundant - which is fine, better safe than sorry, but then you won't be able to ever cover them.
_ezaquarii_@reddit
What you are described is a symptom of high cyclomatic complexity. Fixing that fixes the problem.
ReginaldDouchely@reddit
That's an interesting generalization that doesn't seem consistent with my experience. If you've got devs that actually care, they'll inherently write more tests for the harder code. And in PRs, they'll challenge when the complicated code isn't tested enough.
But yeah if no one cares, then no one cares, and you have to make rules that attempt to legislate caring but never succeed.
MinimumArmadillo2394@reddit
That just encourages me to not include edge cases though
codescapes@reddit
People are lazy and for whatever reason substandard practices with frontend development are tolerated in a way that they mostly aren't for the backend which frustrates me.
For reference the UI project I lead has Jest / React Testing Library and Playwright. The Playwright tests are instrumented and contribute to code coverage stats with an overall 80% tollgate.
Yeah it's not bulletproof and the number is pretty arbitrary but it works fine. Before I joined no such testing existed and UI crashing bugs happened semi-regularly.
Frankly I don't get it when people don't do some basic frontend testing. It saves you time, gives you peace of mind and makes your work better. It doesn't need to be over complicated (and really shouldn't be), just basic testing pyramid shit we've known since forever.
Zero219@reddit
Same with the same testing stack. it’s crazy that most of the frontend devs are not familiar with anything besides easy jest unit tests… I’am always the one initiating these testing practices whenever I join a project.
fabulous-nico@reddit
Oof.. let me introduce you to 90% of the backend devs I've ever met (some who hate on frontend). No ability to actually program or design the solution, just rote Java junk with terrible writing. 🤮
1000% agree, and usually the testing misses the key aspects anyway. It's usually one of my first fixes on a new team
coryknapp@reddit
I have found bugs with tests that I wrote thinking they were dumb and useless.
Peach_Boi_@reddit
Sounds like my last company
randomInterest92@reddit
Cover the business logic and use static analysis to eliminate runtime errors. Especially in large codebases you have a lot of boilerplate code that is useless to test. Don't ever use a fixed %. It's either too high or too low
renq_@reddit
It’s not the right question. The real question is how your team should shape its process to produce high-quality code. Code coverage is best viewed as a team-level process indicator. If you practice approaches like TDD and trunk-based development, you’ll naturally end up with excellent coverage.
Gofastrun@reddit
It really depends on the use case. Testing is an investment. You need to make sure you’re getting the ROI you expect.
If I’m building the BE for a game and a regression would merely inconvenience users, the testing requirement is far lower than if I’m building something that handles money, or medical data, or could potentially cause someone physical harm.
We look at our actual regression tolerance, err on the side of slightly too much testing, and then tune from there.
On most of the apps that I work on (established startups) we primarily use unit testing with mocks, integration tests on major cases, and a handful of e2e tests on critical use cases.
We ship the occasional regression and the world keeps turning.
Sometimes (especially with govt work) the testing requirement is built into the contract, so you follow that.
When I worked for an auto manufacturer the testing requirement was very robust and we had dedicated QA teams.
When I’m at very early startups we intentionally do not test. The code life cycle is too short. Everything is basically a proof of concept until theres at least a 1-2 year product roadmap.
smarkman19@reddit
What’s worked for me is using mocks only at true external edges (HTTP, queues, payment SDKs) and rely on contract tests between services so you can swap internals without breaking callers. Spin up real deps in tests with testcontainers to avoid flaky fakes. Keep DTOs out of metrics and focus on invariants-money never disappears, idempotency holds, pagination is consistent-property-based tests shine there. In CI, run race detector and shuffle; then use mutation testing (e.g., Stryker) to measure test quality instead of raw %. For very early stage, keep a thin safety net: post-deploy smoke tests, canary, feature flags, and a one-click rollback. On gateways and smoke, I’ve used Kong and Postman, and pulled in DreamFactory when I needed a quick REST layer over a legacy DB to exercise real flows. Target risk, not line coverage.
Historical-Hand8091@reddit
Our team focuses on mutation coverage over line coverage, as it better reveals test effectiveness by showing if tests can catch injected faults.
xFallow@reddit
Blanket coverage percentages are bad practice imo
Whitchorence@reddit
I get that there are issues but is there a better way that isn't just "everyone relies on their judgment"?
mountainunicycler@reddit
Especially after AI.
AI is extraordinarily effective at creating useless tests that hit 100% coverage without actually really testing the code.
bluemage-loves-tacos@reddit
This is too true. Only this morning I watched claude undo my test changes instead of updating the code. Made the tests pass, but the code was still broken. If I hadn't been paying attention I'd have looked mightily stupid when we had a test session.
831_@reddit
This might be a dumb question but what's a "test session"?
bluemage-loves-tacos@reddit
When we have enough of something useful done, we get people who care about it (could be a customer, could be a colleague, could be us) to give it a test drive so we can get feedback and find issues. We don't always do it, but it can be useful to do some sanity checking and see how something feels. It's also useful for showing our CS team new things that make their lives easier.
831_@reddit
Ah ok, I understand. I thought you meant something like a session during which everyone looks at the tests and comment on them, which seemed a bit unusual.
mountainunicycler@reddit
Yeah…
“Wait let me check the behavior of the code”
…. Thinking
“Now I understand the implementation, 2+2 should equal 6. I’ll fix the tests”
wubrgess@reddit
So am I when CI has a gate on 98% test coverage.
empiricalis@reddit
I am a government contractor and have a mandatory 80% line coverage rate, and they're very particular about what we're allowed to exempt from coverage. Guess what ends up happening to the tests?
danintexas@reddit
We do 100%. As someone that has been a dev for 5 years and in QA for 20. That is PURE INSANITY and will be the #1 reason I leave.
kemitche@reddit
The only valid way to have a % goal is to make it 100% BUT ALSO empower every dev to annotate code as not needing to be tested. In this case, 100% means "every line of code the dev checked in is either tested or deliberately not tested."
Even that is questionable and requires the right attitude and environment. I was in such a scenario once. It was alright.
GRIFTY_P@reddit
The feeling when you refactor clean code to be worse so that it can be more easily tested lol
danintexas@reddit
LITERALLY on some of the most complex endpoints I spend more time rewriting it so as I can hit the 100% test coverage metric. It is annoying as hell for me because I am forced to remove defensive clean code just for unit tests.
random314@reddit
We aim for blanket 99% coverage.
You're welcome to add exclusions for review in your PR with a reason though.
actionerror@reddit
Gotta test those log statements mirite? 😜
random314@reddit
If you have customized logger logic then yes, you should test those.
KronktheKronk@reddit
Insane. 80/20 rule
random314@reddit
Well you can always exclude files and add exceptions.
caboosetp@reddit
We expect 99% but put the rule at 80%. It stops people who just plain forgot unit tests but means people don't need to request exceptions.
jl2352@reddit
It’s good as a compass. Today your project has 10% coverage and it shows through support tickets. So you say ’let’s aim for 70%’, and you really lets write tests as standard.
Once people are writing tests, then you can shift to writing good tests.
Deranged40@reddit
100% code coverage metrics creates bad tests every single time. All of the absolute worst unit tests I've ever seen were when I worked at a place with a 100% coverage requirement.
AWildMonomAppears@reddit
Not sure what's worse, 100% or 0%...
_ezaquarii_@reddit
I'd like to reply with "every single time you did it." :)
There are other experiences.
But yeah, people generally struggle with tests. We have to put them through training.
DaRadioman@reddit
No, requiring any blanket coverage percentage welcomes gaming the system and useless tests with no clear Adrian to satisfy the numbers.
I can cover 100% of the code and not really test what matters.
andlewis@reddit
That’s why I aim for 110%. That 10% covers the bad parts.
TitusBjarni@reddit
Perhaps it's good for forcing devs who don't know how to unit test to learn to unit test
SideburnsOfDoom@reddit
So they learn how to go crazy with mocking frameworks and close-coupled tests. That's a way to unit test, but IMHO not a good way, and to get better it will have to be unlearned. It seldom is.
xFallow@reddit
I assume that’s what they’re there for my company just made a GitHub group so that a senior needs to approve a PR before it gets through
mirageofstars@reddit
And yet so often requested by management.
metaphorm@reddit
agree. "once a metric becomes a target it ceases to be a good metric" applies here.
Ok-Yogurt2360@reddit
I usually only look at coverage after i think that i made good tests. In that case the metric can tell if i left some important gaps in my tests. The value of the metric goes down the more often you look at it.
itsgreater9000@reddit
Coverage can only tell you what areas of code have not been exercised by your tests. There are ways to write tests that cause coverage to increase but do not actually "test" the code (at least, not what we want a test to do).
DaRadioman@reddit
This. It's a signal when you are building the code to see how tested the code is, but it's a relative metric, you make a judgement call not a hard number
systemnate@reddit
Not all teams are the same. If you have a good team, with a good process, that values testing, and picks up lack of code coverage in reviews, and encourages deleting low value tests, then code coverage metrics don't make sense.
If you have a junior team that doesn't have good test coverage, then enforcing something like 75% coverage can be really helpful.
100% coverage doesn't make sense for 99.9% of products.
actionerror@reddit
I 80% agree with you
RedditNotFreeSpeech@reddit
Testing needs to add value.
FutureSchool6510@reddit
In my opinion, mutation testing is the key. We have line coverage metrics for our codebase but I don’t place much value on them. Line coverage is too easy to cheat by adding tests with few or no assertions that simply execute the code.
The metric we pay attention to is the mutation coverage reported by PIT. If you aren’t familiar, it makes small changes to the code and runs the tests to check if any fail. It properly measures the QUALITY of your tests. If your tests don’t really test anything, mutation coverage will let you know.
In terms of what code to include though, generally everything except application config (eg spring bean config), properties classes, and any generated code such as Lombok or Jooq.
jcgl17@reddit
Mutation testing is the answer. For go, I can mostly recommend gremlins. Development has seemingly stalled and it's missing some important features (most notably ignoring specific lines). But it works and is useful.
Best_Kaleidoscope_89@reddit
Never heard of this. Mutation testing as a concept sounds interesting.
danielt1263@reddit
I routinely do this (after writing the tests, I will subtly alter various lines of code to see if the tests catch the change), but I've never heard of an automatic tool that does it for you...
TitusBjarni@reddit
That's good, but writing the test first (TDD) is better. Similar benefit but fewer steps. Test fails then you write the code to make it pass. Sometimes you may even find that the edge case you were thinking about was already working so no production change necessary.
FutureSchool6510@reddit
pitest dot org for a Java based one. Not sure if other languages have options.
PandaMagnus@reddit
stryker-net for c#. I haven't personally used it, I just introduce own variances in my code. My coworker highly recommended it, though, and he's usually pretty picky, so I need to give it a shot.
PandaMagnus@reddit
Absolutely support mutation testing! I used to do it manually, but there are tools to automate it.
OdeeSS@reddit
I absolutely agree that 100% line coverage is meaningless without mutation tests. I also find that resolving mutations in tests is a fun way to guide people how to make tests that are meaningful.
Unfortunately, they still can't beat a bad test culture. If people mindlessly rewrite the tests everytime they change the code without defining why the tests needed changed, or aren't able to write tests that describe the requirements, then mutation tests become the new, new metric.
FutureSchool6510@reddit
Nothing can beat bad test culture except education. You can only automate your way so far around a knowledge/skill gap.
rag1987@reddit
Unit tests should be written to do the following in order of importance:
Most developers can write mostly correct code without tests, but breaking something you forgot to think about is very hard.
Don’t use mock frameworks, at least in the beginning if possible, they make very fragile white box tests.
thewritingwallah@reddit
Well write tests that will save you time later.
My favorite part of tests is that it allows you to update code later without the fear of breaking something related. If you find yourself constantly testing related things after making updates, you probably want a test.
imagebiot@reddit
imo just max out your integration test coverage and the rest just helps you sleep better
k8s-problem-solved@reddit
I lean further into this now. For a rest api, spin up the whole process, stub out its dependencies, test the api at contract. Get a lot of value from this, certainly for happy path.
Then, unit test all the edge cases, easier than trying to do over http.
There's a balance in there where you get the best of both, and makes your tests less brittle to changes.
hooahest@reddit
This is how we used to work at my former workplace. Almost 0 bugs, developer experience was great, PRs were lean. Loved it.
New place has mostly mocks and 100% required code coverage. It's a big stinky mess and I hate it. Half of my time on every feature is writing meaningless tests that do more harm than good.
AbstractLogic@reddit
Coverage % are bullshit with AI these days. I can ask an Aai give me 99% coverage and it will write the tests over night to satisfy that.
Lords_of_Lands@reddit
It was bullshit before AI. Line coverage doesn't mean branch coverage so 100% line coverage doesn't mean you're testing all your code. Even then, branch coverage doesn't hit edge cases you didn't check for (like overflows or missing files) and knows nothing of your business logic. Your code might be bug free, but that doesn't matter if it's doing the wrong thing.
Finally, your tests are only as good as your developers. If your developers are writing buggy code, your tests are going to be filled with bugs as well. At least TDD tries to force you to test your tests, but again, that does nothing for when you forget an edge case.
AbstractLogic@reddit
It’s just a matter of being better than what we had before which was nothing.
Lords_of_Lands@reddit
Fair enough.
marquoth_@reddit
Goodheart's law: "When a measure becomes a target, it ceases to be a good measure"
It's tough, because there is no denying that if you arbitrarily mandate 80% coverage you create an incentive for developers to write garbage tests just to make the linter happy.
But on the other hand, if you don't apply some kind of linting rule you allow for untested code to be rushed through in the name of expedience. "We'll come back and write the tests later" but later never comes.
What really needs to happen is for reviewing tests to be just as important a part of the PR process as reviewing the code itself; PRs can and should be rejected on the basis that the tests aren't thorough enough or don't cover the right use cases even if you're pretty sure the code is sound. But this still relies on people to do their job properly and not wave things through because they don't have time.
In short, I think it's better to have a coverage requirement than not to have one, but with the caveat that it should not be relied upon as a substitute for properly reviewing your colleagues' work.
As for the actual number, it doesn't matter too much. 75 + d20
lordlod@reddit
It should increase, it should very rarely decrease.
If it keeps going up you will naturally find the useful level.
You discourage it going backwards so that when people make changes they keep the tests up to date. Red text in the merge request report works wonderfully for this.
ThouCodingDrummer@reddit
Side note, if your code coverage primarily comes from AI unit tests, then you might as well not be testing at all.
tobega@reddit
We're actually going away from general testing and focusing on getting good signal from tests, types and other things like contracts. Some of these ideas you can get from my blog post https://tobega.blogspot.com/2023/09/using-programming-structures-to.html
bluemage-loves-tacos@reddit
Is the behaviour covered? That's what we care about. I DGAS if there's zero % coverage on code that doesn't get executed, and I care very much that a behaviour that *should* be happening when something else happens, is actually happening.
Our standards are: Mocks are allowed, but should be *outside* of the code that's being executed to show a behaviour. So, for example, if I'm taking a basket and making an order, and if I have an API client to make that order to a third party system, I want the boundary to be the POST request, so I can check we're able to put the order together, get it posted (API client code is our code, so we should test it), and the expected mock response data parsed and further actions executed using that response.
Metrics are fine and all, but what matters most is are we able to test behaviour. We don't track test metrics, we track issues, incidents and focus on understanding if what we did is working or not. So we like to look for observability of the system over test metrics.
steliosntilis@reddit
What about mandates from SOC2? Actually I am not sure if PCI or SOC2 have a mandate for code coverage but this is what our Security and Compliance team says. "We need to have above 70% of code coverage and less than 3% duplicate code"
Any thoughts?
Outrageous-Bath-3494@reddit
I prefer 100%. I see the annoying cases like DTOs as a small price to pay. 100% forces me to think about structure, and if I am writing meaningful tests and making code appropriately modular, it is not even hard to hit.
I can get why some people hate it though. And I don't think 100% makes you write good tests. It's just that 100% is easy to hit when writing good tests, so long as everyone cares.
gomsim@reddit
Wo don't decide on right "amount" of coverage. Coverage % is pretty much meaningless in my opinion. Checking coverage per line of code however is a good tool to see what lines are covered by tests, in case something was forgotten.
Otherwise testing should be done just the way you designed a function. What are the potential outcomes and errors. Test that it handles all that could be thrown at it, edge cases and edge values, values that are on the edge of what makes the function give a different outcome.
And this is a personal preference, but trivial code I don't bother testing. I a function returns a hard coded value any test testing the function will just be a copy of the code. It was a silly example.
thefightforgood@reddit
80% because that's the default setting in our tooling. It's not a hard gate, but we're usually over it. We don't stress too much about it, mostly there just to remind us to write tests and we focus on the content of those tests more than the actual coverage.
Zotlann@reddit
I just don't think coverage can ever be a good metric to judge testing by. It's too easy to write completely worthless tests to achieve coverage, especially with coding agents.
Ideally you have a work culture that values meaningful tests and that is just enforced socially through code reviews. My previous job had almost 0 testing before I joined and I had to set up the infrastructure for it and try to convince other devs of the value. Its not easy.
I think the most successful thing we did specifically with regards to code coverage as a metric was a ci/cd check that would not allow prs through if they lowered overall code coverage for the repo by some amount. This at least made sure that large amounts of new untested code wasn't getting added while we worked on backfilling tests.
omz13@reddit
My rough rule of thumb is that about 50% coverage means you’re hitting the hot path - so good enough for regression testing. And then another 10% or 20% so to cover testing for stupid footguns like bad parameters and obvious error handling. After that, its diminished returns territory. As always, YMMV depending on the circumstances.
Yes, use mocks so you can test quickly in isolation - catch the stupid stuff early and quickly.
Use mocks when you start to enter integration territory - this is when you really see if things are compatible or your architecture is built in quicksand.
And somewhere you do end-to-end testing and hope not too much smoke escapes.
Coverage is one thing to consider (metrics), but it’s only one aspect.
OhhWhales@reddit
Wouldn't you want integration tests to actually hit the respective "units" rather than mocking the response?
liammyles@reddit
Coverage is important until you reach 100% and doesn't mean anything once you reach 100%.
It's just a tool that tells you if every line in your code ran, it doesn't tell you if the interface or contract with the module/API has been tested.
My background is web development and I like to be explicit about what we remove from coverage reports for node based test runner.
For example if there are expectations that the code is tested in E2E tests i.e. in browser testing via playwright and can't be testing in a node based test runner, it's explicitly removed from the coverage report in a clear way that communicates this.
As an aside, if the team is trying to minimise the amount of tests they are writing. Which is often the case around debates of coverage.. They are likely writing poor tests that don't allow you to change your code easily in the future. If that is the case, coverage is the least of your concerns.
Start by agreeing and figuring out how tests improve your velocity.
Waksu@reddit
You should have 100% test coverage on code that matters.
SideburnsOfDoom@reddit
How we do it:
Test features not lines. Cover all features, including error cases.
Test "from the outside in" as a first choice. As that you can refactor without any tests breaking or even changing.
Mocks are allowed, but are de-emphasised.
FlipperBumperKickout@reddit
I don't care to m much about coverage percent.
Using locks makes it kind of pointless. How am I gonna test the behaviour of the system if I replace part of it 😅
Goodos@reddit
Whatever the coverage ends up being after testing all defined behaviour, is the correct percentage.
Andreas_Moeller@reddit
Dont measure coverage. Having 100% code coverage does not mean your application works, and it becomes a target.
Instead measure error rate (what % of changes are bug fixes) and try to improve that
anyOtherBusiness@reddit
The 21st stupid CRUD operation might not need full coverage.
A complex tree traversal operation might need more than just 100% coverage. You might want to test with different datasets, to cover different scenarios including edge cases.
No_Bowl_6218@reddit
Code coverage is a good metrics for non tech management. For tech people, without mutation testing, coverage mean nothing.
flavius-as@reddit
100% of the domain model.
If any of it cannot be covered by exercising the code in front of it, then that code can be removed as dead code, getting it back up to 100%.
Adorable_Tadpole_726@reddit
A lot of the time struggling to get 100% test coverage really means you need to rethink how your code is written.
fabulous-nico@reddit
"Program testing can be used to show the presence of bugs, but never to show their absence!" - EWD
CanIhazCooKIenOw@reddit
We test the most outcomes we can.
Aiming for 100% lines leads to useless tests.
Wise_Tie_9050@reddit
the number of times I’ve found an extremely subtle bug in the last untested line of code in an MR…is not insignificant
nomoreplsthx@reddit
My entire metric boils down to "would I be comfortable if every PR auto-deployed immediately to prod if the tests passed'
eeltech@reddit
first of all, I've had the misfortune of inheriting projects with 0 coverage :/ So just the fact there is some coverage, and an actual attempt to define what is adequate coverage, is IMHO already a win, and a team that has this on their mind, is probably doing an ok job so I don't stress too much about hitting an actual figure, but more about each PR includes adequate testing for any new branches/business logic that it adds
No specific tests needed for DTO's or simple getters/setters - those should have coverage through the units/classes that use them. If you try to hit 100% at some point you're basically testing that your runtime/language can access variables correctly - is this a correct use of your time?
Esseratecades@reddit
I don't care what the % is but
All of the other things I care about in testing are not things that coverage % can really tell you.
TacoTacoBheno@reddit
You pass enterprise requirement. They don't care if tests actually test anything.
There's also probably a coverage exclusion file
rayfrankenstein@reddit
On a legacy codebase that never had tests written for it it’s “0%, bur a great thing to keep in mind for the rewrite”.
woodnoob76@reddit
A very Lean/Toyota answer: when a bug appears, cover it before fixing. Or reformulating by coverage thinking: when a piece of code tends to break or regress often, cover it (and for real refactor this).
Another approach that you can automate, I asked AI coding to keep the ratio to max 1:1 in test:code total lines of code ratio (a generally approved rule of thumb), and respect the pyramid (unit/integration/end to end). It tends to rationalize which tests are bringing the most value, proof of working software and criticality of the feature. Looking at test coverage by zone you can also eyeball it (have an automated report on these numbers, plus coverage grouped by sections
james-prodopen@reddit
Mind adding a bit more context on what the automated report tells you?
woodnoob76@reddit
Mine is not automated, but I have in my code reviews command prompt to also check the code to test volume ratio, and the test pyramid ratio (from memory: 70% unit tests, covers the logic, 20% integration tests, and 10% end to end). The prompt refers to the general definition, so interpret it in context (prototype project, or actual thing meant to be released for others, enterprise grade product, etc).
Manually I would put this in a CI/CD most probably, these remains rough scripts, effective line counting and comparison
BadLuckProphet@reddit
Its the right amount of coverage when you feel that there is a 90% chance that every new feature and bug fix won't cause a regression that makes it to prod.
Testing language features for the sake of a % is useless and will motivate your team to make stupid tests just for the sake of having tests instead of thinking about what kinds of tests they need to have to smartly ensure the code works as expected.
Tests should be something you make for yourself and your team to save yourself headaches, not something you make for your boss to prove what a good and productive resource you are.
That said, my metric is bad for inexperienced or arrogant teams.
mattk404@reddit
Cover the stuff that will get you called in the middle of the night. Let incidents guide where those areas are. New features require tests first or near first and require review of areas 'nearby' because that is what will wake you up in the middle of the night.
reboog711@reddit
For libraries I push for 100%. We're usually higher than 98%.
For "real" projects; we shoot for 90%.
These are pretty subjective numbers, that someone probably pulled out of the air at some point. No real thought went into them.
However, it is not difficult to create a test suite that has 100% coverage without testing anything, so you need to be diligent about how you write tests, and how they are PR reviewed.
Most of the time, these DTOs are just classes w/ no functionality. There is nothing to test. However, Sometimes our UI (typescript) Objects will implement interfaces, that use getters. I will test the getters.
diablo1128@reddit
I worked on safety critical medical devices for years and we needed 100% code coverage in statement, decision, and Boolean. We got in via a combination of automated unit and integration testing.
zaitsman@reddit
Investors. What looks good on paper during due diligence, which we sort of aim at 75%+
TrapdoorThunder@reddit
We don’t! It’s arbitrarily set by an org wide policy that the devs have no control over! Welcome to hell!!
__scan__@reddit
100%
SamWest98@reddit
zero percent on my team LOL
BrofessorOfLogic@reddit
In principle there is nothing wrong with aiming for 100% coverage. I would even say that 100% is something to always strive for, with the understanding that it may not always be realistic.
The key is understanding when it's realistic and when it's just a waste of time. It depends on the type of project and language.
If I'm building Yet Another CRUD API in Python with a high level framework, then doing 100% coverage can be very realistic. The language and framework covers so much ground work and plumbing, that basically every line in my code is value-adding business logic.
But the most important thing is to understand that while coverage is A Good Thing, it's also just step one on a larger journey. And if you are only looking at coverage, you are definitely not doing testing correctly.
The real value in testing comes from stuff like: asserting just the right things at the right level of abstraction; using realistic test data so you're not just testing the happy case; integration- and end-to-end testing to ensure flows are working across components and services, etc, etc.
There's a lot more that goes into it, more than can be written in one Reddit comment. There are entire books on the topic. But again: do care about coverage, but don't get stuck on just coverage.
CallinCthulhu@reddit
Whatever management says is the right amount for whatever metrics they track.
Does that mean I write a lot of useless tests? Yes, it does.
But with AI I don’t really care because it’s not like I have to write them myself. It’s only the core logic that I really pay attention to for tests.
drahgon@reddit
Line coverage is just an objective way to ensure tests are getting added. It speaks nothing to the quality of said tests or their usefulness.
In practice I find you can get about 60 to 80% with mostly just happy path testing. Another 10-15% requires careful error path testing. In the last 5-10% is things that probably are going to add zero value and you're just doing it for coverage at that point.
So once you get up to a number you like and you think that it provides value you shouldn't let any Mrs lower that number. That ensures that new code that may be vital as well will also have coverage at minimum and it might require a little bit extra depending on how much code is added.
cstopher89@reddit
The right amount is easy. Any business logic should have a suite of tests around it.
secretBuffetHero@reddit (OP)
one test? what about branching logic? what about edge conditions?
cstopher89@reddit
Did I say one test?
secretBuffetHero@reddit (OP)
how do you decide "right"? vibes? what my vibes are different than your vibes
zouxlol@reddit
it depends how critical the code is. we use 80% as a blanket statement but we cover everything on the backend so you know if you are going to damage data that damages everything else for example
ryanbrownstar@reddit
I write great tests to cover core functionality, then I write crap tests until sonarqube let's me merge
secretBuffetHero@reddit (OP)
I believe spiderman said the same.
"with great tests comes great core functionality "
loumf@reddit
Check out diff_cover https://github.com/Bachmann1234/diff_cover which can give you coverage for just the PR, or just your release branch diff.
I have a higher threshold for this code than the whole codebase
failsafe-author@reddit
I lead by example. I find if I write a lot of tests, other people around me will do it too.
I use TDD, and it gets high test coverage. And absolutely use mocks. I really don’t know how people test without them or why people dislike them.
I usually don’t test application initialization, and I’m not a fan of testing visual styling. Though I haven’t written a UI professionally in ages.
ivancea@reddit
We usually don't cover by lines, but by cases and paths. If you do TDD, coverage is also weird.
There are many different apps where different ways of testing may make sense. But in general, you test to say "this outcome is now written with blood". So you know that, whatever you do, that will happen. Do it for every potential case, and you have your feature tested. No need to check coverage, even if it will usually be near 100%, ignoring edge cases like maybe interrupted threads or such edge exceptions
w-lfpup@reddit
There's no serious number or fast rules for coverage but I do think _where_ you focus test coverage is important.
I think a great starting point is 100% test coverage of a forward facing public API. It's an achievable goal.
But too many tests behind a public API in a team environment can "freeze" development or create an "add only" expectation that leads to bloat. Once tests are written it becomes really difficult to justify removing them.
So if you change some data flows behind a public API and those changes involve removing tests in the process? That's going to be a tougher sell to your team unless everyone gets real cool about it.
Not over-testing gives your team space to pivot and make stuff more efficient later
tomqmasters@reddit
It's more about priorities. Make sure your core functionality is covered first and then go from there.
w-lfpup@reddit
There's no serious number or fast rules for coverage but I do think _where_ you are focus test coverage is important.
I think 100% test coverage of your forward facing public APIs is a great starting point and an achievable goal.
However too many tests behind a public API can "freeze" development or create an "add only" expectation that leads to bloat. Once tests are written it becomes really difficult to justify removing them.
So if you change some data flows behind a public API and those changes involve removing tests in the process? That's going to be a tougher sell to your team unless everyone gets real cool about it.
Not over-testing gives me some space to pivot and make stuff more efficient later
F0tNMC@reddit
As is usual, the answer is it depends. But to summarize my rules of thumb:
IsleOfOne@reddit
We do not use coverage as a metric whatsoever.
Abject-Kitchen3198@reddit
100% line coverage with zero assertions.
Seriously, I try to create the least number of meaningful tests. That can result in a metric of maybe 80-90%. Than, analyze the line by line coverage to find uncovered important cases or obsolete code. Add tests where needed, remove obsolete code, refactor where feasible. Will probably still have some lines/branches not covered and I will leave it at that.
Sevii@reddit
Typically there is a corporate mandate of some percentage. Otherwise it's a constant argument that boils down to whoever controls the config that fails builds.
gregdonald@reddit
Implementation code that isn't important enough to be under test should be deleted.
High code coverage isn't just about coverage, only inexperienced developers think that. High code coverage is about communication, documentation, and trust.
I write my failing tests first, so I only ever have 100% coverage.
https://tddbook.com/
publicclassobject@reddit
You sound like you are in bikeshedding hell
ancientweasel@reddit
I like to generate a report that show coverage by color coding the lines and then we read that report.
secretBuffetHero@reddit (OP)
yellow everywhere. good enough?
Oswald_Bene@reddit
"Percentage per line"
Meanwhile, 19 deploys were made based on faith
GeekRunner1@reddit
LOL coverage. I wish.
mkg11@reddit
Excluding everything but services/controllers we have to reach 90%
GrumpsMcYankee@reddit
...coverage?
NoPrinterJust_Fax@reddit
We do 100%. Annoying sometimes but it’s really nice other times. No ambiguity around what “should” be tested when everything is tested
retirement_savings@reddit
My experience on teams that do 100% coverage is that you end up with a lot of tests that are just assertions/mocks of your code and not actually useful test.
NoPrinterJust_Fax@reddit
Yeah we use a lot of mocks
serial_crusher@reddit
I favor 100% coverage of backend code as an automated requirement. If you’re not there it sounds daunting and time consuming, but can be mitigated by implementing a tool like diff-cover, where you’re only required to have 100% coverage on the lines touched in your PR.
That said, quality of the tests isn’t something you can automate. You have to pay attention in code review to make sure people aren’t over-mocking. I think spending more time on integration tests than strict unit tests where all the inputs and outputs are mocked, is a good thing. You want your tests to map to actual functional requirements of your product, wherever possible, moreso than the specifics of some internal tool.
Ie if your public facing API needs to only be accessible to certain user tiers, a test that creates two users, calls the API, and verifies that the correct one of them gets the “unauthorized” response, is better than one that just mocks the
current_user.isAdmin()method or whatever.bigkahuna1uk@reddit
Only cover items with behaviour. If you follow that rationale, then the coverage of data objects naturally falls out instead of just being a metric created artificially to meet some arbitrary percentage.
I've worked in teams who were obsessive in trying to get 100% code coverage and it really is an exercise in diminishing returns. It does not add the value that it purports to.
ZukowskiHardware@reddit
I like having everything 100% covered if users are interacting with it. By that I mean, explicitly ignore files or sections that are not worth it. When you make a change and don’t mean to not cover it you get bonked. For internal stuff like 29%
_Atomfinger_@reddit
In my teams we don't have the concept of "right amount of coverage". We use coverage as a tool to identify blind spots in our testing and to identify that a PR has a bunch of code that is not covered by a test, but we don't measure ourselves by it.
In other words, for us, it is a tool we use to identify certain issues rather than a goalpost we try to reach.
Not static, but I find mutation testing to be much more useful.
I usually let the teams decide that, but given the opportunity, I tend to ban them. Hot take: If you have to use mocks to test your system efficiently with unit tests, then you need better architecture.
Golandia@reddit
I love using code with 100%. I don’t love writing code when it needs to be 100% because tests aren’t fun to write.
smidgie82@reddit
On my team, though, we don't track a particular number, we just watch the trend over time. If it's trending down, we (team leadership) pay attention to make sure that we (the team) are being disciplined about testing. If it's flat or trending up, we leave well enough alone. And of course we're always looking out for whether meaningful changes are covered in any single change.