Skip to main content

Custom Metrics on Dashboard

Problem:
I have a a Playwright test for an onboarding flow currently assesses LLM-generated content using an LLM-as-a-judge method with a binary pass/fail outcome (score > 60). While functional, this approach lacks the ability to track score trends over time, which would provide more valuable insights.

Suggested solution:
Implement a custom metrics system that:

  1. Captures the actual LLM evaluation scores from tests

  1. Exposes these scores to Checkly's dashboard

  1. Enables visualization of individual scores per run and aggregated metrics (7-day and 14-day averages)

  1. Creates an extensible framework for any future custom metrics beyond just LLM evaluation scores

Log in to comment and vote

No comments yet

Be the first to share your thoughts.