Alert Renotification Functionality
Currently, we can set up an alert that fires on failure, along with a reminder that is limited to 30 minutes.
I would like to request the ability to renotify/re-alert if the failed check is still failing after a user-defined period, such as X hours or X days.
This feature will help with:
Continued Visibility: Renotifications ensure that test failures do not go unnoticed, especially if the initial alert is missed or not promptly acted upon.
Priority Management: Persistent issues can be escalated more effectively within the team, promoting quicker resolution.
Log in to comment and vote
Comments2
Nick McBurney
Jun 21, 2024
Hi Rikard,
There is a few scenarios:
1. We make a change which breaks test but doesnt include update to resolve the test with release (usually this would be resolved quickly but has been missed due to other high priority work taking precedent) causing test to be in 'failure' state for day or two.
2. Test fails at 7pm on a Friday, a bunch of other alerts happen over the weekend - creating noise in our alerting channel and original failure gets missed.
The specific reason im raising is we had a wide outage on one of our test groups, at the same time a test in another group had started failing (unrelated to outage). We missed the original alert due to larger outage and only noticed issue a few days later after someone had reviewed dashboard and see test had been failing for a number of days (we have since enforced a 'daily sense check' to review dashboard to reduce chance of this but re-alerting would also help.
Rikard Nilsson
Jun 19, 2024
Hi Nick,
Thanks for the feedback, is it common that you have checks failing for a long time, and if so, do you know why? I can see the usefulness of this feature, but I'm also interested in understanding why checks would be failing for so long before being addressed.
Cheers,
Rikard