Code coverage tells you what you didn't test — not whether your tests are good
Posted by indy2kro@reddit | programming | View on Reddit | 21 comments
A look at how coverage metrics work in CI/CD, why they're misunderstood, and how to enforce them without incentivizing shallow tests.
ganja_and_code@reddit
"Quality and Quantity are Two Different Metrics"
No shit. Is your next article entitled "Green and Orange are Two Different Colors"?
PerkyPangolin@reddit
Big if true.
AvidCoco@reddit
The condition of
if (true)is always meat so you can simply this line to just “Big”.xavia91@reddit
true in this case is a pointer to the statement, just a very shitty variable nameing choice.
xavia91@reddit
Writing a lot of articles doesn't mean they are good articles 🤯
BaNyaaNyaa@reddit
I'm sure you'll like my next article about apples and oranges. I have some insights you might have never considered!
RapunzelLooksNice@reddit
Unless you are colorblind...
ecafyelims@reddit
"code coverage is a measure of code coverage, not test quality"
🤯
NullField@reddit
It's obvious to many, but many also don't understand the implications of what it takes to get to and maintain 100% coverage or even what coverage even means.
It's largely a company environment issue and not up to individual devs. When given little time to test and high coverage requirements, meaningless tests to hit coverage targets is inevitable.
ecafyelims@reddit
This is a programming sub, which is why I said it like that. Programmers know. Business is the one pushing for 100% code coverage without allotting sufficient time for proper testing.
100% coverage with poor tests is detrimental, imo, because it takes time to write and maintain and primarily because later updates break the tests so frequently and so numerously, that engineers become blind to the test failures and start reflexively "fixing" the tests to match the new results so that they pass after update.
Which, of course, defeats the purpose of the tests, and bugs are introduced with tests that reinforce the bugs.
I've sung this song many times. It's business forcing it along with a handful of newer engineers who just haven't seen it happen and believe it can be done with perfect engineering. Sure, but the company doesn't want to pay for perfect engineering. They want fast engineering that's good enough, and they want to check a box that says 100% code coverage so they can include it in marketing.
It was just funnier to make the joke 🤣
agumonkey@reddit
time to invent code coverity
mnilailt@reddit
Big if true
jer1uc@reddit
Been a while since I audibly laughed at a Reddit comment. Thank you stranger.
CherryLongjump1989@reddit
It doesn't tell you what you did not test. It only tells you some of what you did not test.
AvidCoco@reddit
This is the important caveat with code coverage. It just tells you what percentage of LoC were called during the tests, not whether the logic implemented by those lines is sound.
You can have 100% coverage without testing any logic at all.
teknikly-correct@reddit
lol don't tell my manager - oh never mind, you can tell him, he'll still tell his manager and his manager's manager etc about how 100% "code coverage" with his chest all puffed out.
zmose@reddit
Mutation test suites 👍👍👍
stewsters@reddit
When you really need to test everything.
Uncaffeinated@reddit
I developed a comprehensive mutation test suite for my project, and then deleted part of the code and challenged Codex to reimplement it. It eventually managed to produce something that passed all the tests, but still had fundamental bugs.
echocage@reddit
Sounds like the tests weren’t robust enough then
ireddit_didu@reddit
Duh.