Slow running integration tests in CI/CD pipeline?

Posted by WorldTraveller101@reddit | ExperiencedDevs | View on Reddit | 3 comments

We have a Jenkins CD pipeline for deploying a fairly large Java API. The jenkins server pulls the code for git, runs unit tests, builds the artifacts for deployment and deploys to the kubernetes cluster. We have many deployments in a day, sometimes every \~30 mins or so.

Also we have a set of integration and regression that take a long time to complete and is not feasible to add them to the CD pipeline (and thus making CD a proper CI/CD pipeline). Our QAs manually run integration and regression tests from time to time to catch bugs.

Since integration and regression tests are not run on every CD, sometimes bug creep in to production causing havoc and we have to roll back the code.

So my question is, how do/would you add integration and regression tests to your frequently running CD pipeline?

[-]

ccb621@reddit

There was a very similar post to this one yesterday: https://www.reddit.com/r/ExperiencedDevs/comments/14stil9/can_you_effectively_do_true_cicd_with_many/.

Questions

How long do the regression tests take?
How long do you want them to take?
What have you done to profile the tests and decrease the runtime?

Ultimately, your question is really, "how do I make the long-running tests faster?" The answer is either to either profile them and improve the code being run (e.g., remove extraneous calls), or shard the test suite to run more tests in parallel.

StagCodeHoarder@reddit

2 hours and 30 minutes
Less than 5 minutes
What takes time is that each individual test dumps, recreates and reseeds the entire database. Its a small database.

BookkeeperAccurate19@reddit

We kept regressions down by grouping slowest integration tests outside main pipeline, only running them before prod deploy. How do you decide which to fast-path vs batch?