Implement Test Session Retries Similar to Monitoring Checks
Hello Checkly Team,
I recently reached out regarding the possibility of implementing retries for test sessions in a manner similar to what we can do with monitoring checks. This feature would be incredibly beneficial for users who need to ensure reliability and effectiveness in their test strategies, especially in environments where transient issues might cause test failures that don't necessarily indicate a problem with the tested application itself.
The ability to configure retries for test sessions would not only add a layer of robustness to testing strategies but also align the capabilities of test sessions more closely with the monitoring checks feature. This consistency across Checkly's platform would enhance the user experience, making test management more intuitive and efficient.
Implementing a retry mechanism could include customizable options such as:
The number of retry attempts before a test is marked as failed.
The interval between retries.
Conditions under which a retry should be triggered (e.g., specific error types or failure scenarios).
Such a feature would be a significant improvement for teams looking to minimize the impact of flaky tests and transient errors, ensuring more accurate monitoring and alerting based on the true state of their applications and services.
I believe this enhancement would greatly benefit many Checkly users, providing them with more control over their testing processes and improving the reliability of their test outcomes. I encourage other users who see the value in this feature to upvote this request, bringing it to the attention of the Product Team.
Thank you for considering this suggestion. I look forward to any updates on its potential implementation and am happy to provide further input or participate in beta testing if needed.
Log in to comment and vote
Comments11
Tim Nolet
Apr 17, 2024
Hey @Jiang Liu thanks for mentioning this. For test session, we currently rely on whatever retry options your CI platform gives you. We explicitly do not want to mix the retry behaviour of monitoring checks with the testing phase of things. Having said that, I think there is good use case for enabling retry settings on the CLI level, e.g.
npx checkly test --retries 2or something similar.This will give you quicker feedback and you can 100% control the behaviour of runs via the CLI without relying on "hidden" setting on the UI.
Jiang Liu
Apr 19, 2024
@Tim Nolet Thanks for the reply. Yes, I don't mind the retries option in either CLI or UI, as long as it resolves this issue. Ty!
Brian Begy
May 6, 2024
We need a retry button right on the results page that queues up an immediate retry.
User Story: As a Backend Developer making an important change to the back end code, I want to clear the last CI hurdle and merge my code, thus finishing my sprint commitments an hour before the sprint ends.
But when CI runs, I see a failing test in Checkly that cannot possibly (or can it?) be related to my changes.
I'm now completely blocked until I can understand if I broke something, or if something is flaky with our test.
I press the thoughtfully provided "retry test" button, wait 90 seconds and see that the problem is indeed a "wait for " operation that depends on a flaky upstream dependency. I relax and merge my code.
Shashank
May 16, 2024
This feature is really important for us on a daily basis. Upvoting.
Alvar Lagerlöf
Jun 7, 2024
We're using check groups to configure our test sessions, it would be nice if was configurable from there.
María de Antón
Jul 29, 2024
Hi all,
Test sessions retries through the CLI are now available.
You can either run
npx checkly test --retries= 2ornpx checkly trigger -- retries=3Please see more details in the changelog: Checkly CLI 4.8.0. - Test session retries.If you’d like us to extend this functionality even further, please open a new feedback note. Thank you so much!