In machine learning monitoring and observability, the evaluation window is a plot of the period or duration of time against the metric being calculated. For instance, the previous 30 days. Any evaluation metric that can be represented as a duration of time can be visualized as an evaluation window. The window defines which inference events count toward today’s accuracy, drift score, or latency percentile. Change the window and you change the story the dashboard tells.
Key takeaways
- An evaluation window bounds the timestamps included when aggregating a metric.
- Fixed windows (last 7 days) and rolling windows (trailing 24 hours) answer different operational questions.
- Short windows react fast but swing with noise; long windows smooth noise but hide fresh regressions.
- Align evaluation windows with label arrival latency so metrics are not computed on incomplete ground truth.
- Compare current windows to baseline windows of the same length for fair drift detection.
How windows appear in dashboards
Time-series charts plot metric value on the y-axis and window end time on the x-axis. Each point might represent “accuracy over the prior 24 hours ending at midnight UTC.” Tables may show a single active window: “production accuracy, last 30 days.”
Some systems support multiple simultaneous windows: a 1-hour alert window and a 28-day trend window on the same model. The short window pages on-call; the long window feeds quarterly reviews.
Fixed versus rolling windows
A fixed calendar window resets on schedule (daily, weekly). It is easy to communicate (“March accuracy”) but can mix deploy events awkwardly at boundaries.
A rolling window always looks back N hours or days from now. It is standard for alerts because every point covers the same duration. Rolling windows overlap, so consecutive points correlate; use that when interpreting small wiggles.
Session or batch windows appear in LLM eval pipelines: score all traces ingested between job start and stop. Treat those as evaluation windows even when the UI calls them “runs” or “experiments.”
Choosing window length
Match the decision latency you support. Fraud models may need hourly windows. Monthly forecasting models may use 30-day aggregates. Consider:
Traffic volume. Low-volume models need longer windows for stable rate estimates. A 100-request hour makes accuracy meaningless.
Label delay. If ground truth arrives three days late, a “last 24 hours” accuracy chart lies until labels backfill. Either delay publication or show provisional versus final metrics.
Seasonality. Retail spikes can make a 7-day window unrepresentative. Compare to the same window last year or use baselines that expect cyclic shifts.
Deploy cadence. If you ship daily, windows shorter than a typical bake period confuse deploy attribution.
Windows and drift detection
Drift compares a current window distribution to a reference window (training snapshot, prior month, or pre-deploy baseline). The comparison is only valid when windows are defined consistently: same length, same filters, same feature encoding version.
Population stability index, KL divergence, and embedding distance monitors all depend on window pairing. A common mistake is comparing last 24 hours to a training set collected two years ago without adjusting for known seasonality.
Windows in eval harnesses and CI
Offline evals use implicit windows: the dataset snapshot date. Online, explicitly document "metric X, trailing 7d, UTC, slice=US" in runbooks so incident bridges share vocabulary.
An evaluation harness may replay production traffic from a stored window into CI when investigating regressions. Export rows from the same window you saw degrade online to avoid training-serving skew in the debug set.
LLM and agent evaluation platforms let you align monitor windows with experiment time ranges so promoted models are judged on comparable traffic periods.
Failure modes
Mixed time zones. UTC versus local midnight shifts window boundaries and breaks cross-team comparisons.
Partial windows at deploy. A model live for six hours still gets a “7-day” chart with mostly predecessor data unless you filter by version.
Stale baselines. Reference windows that never refresh make benign drift look permanent.
Alert fatigue. One-hour windows on noisy metrics flood pages. Tune window and threshold together.
Practical guidelines
Document default windows per model tier in onboarding docs. Provide drill-down from aggregate window to trace list for the worst slice inside that window. When reporting externally, state window length, timezone, and whether labels are complete.
Automate backfill jobs so when labels arrive late, historical window aggregates update rather than freezing incorrect values. Writing LLM evals as tests pairs well with fixed golden datasets for CI while production uses rolling windows for continuous checks.
FAQ
What is a good default window for classification monitors?
Trailing seven days for aggregate accuracy plus a one-day window for alerts is a workable starting point if daily volume exceeds a few thousand labeled rows. Adjust down only when labels are fast and volume is high.
Can evaluation windows overlap?
Rolling windows always overlap by design. Fixed calendar windows can overlap in comparisons (e.g., March versus February) but each aggregate point should draw from disjoint events unless you intend cumulative metrics.
How do windows relate to evaluation store retention?
Retention must exceed your longest analysis window plus label delay. If you analyze 90-day trends with 5-day label lag, keep at least 95 days of joinable rows.
Should LLM eval windows match user session length?
Not necessarily. Session-level metrics may use per-session windows while operational dashboards use ingestion-time rolling windows. Name them differently in UI labels to avoid confusion.
Why did my metric jump at midnight?
Calendar-bound windows reset denominators at boundaries. Rolling windows can still shift as old high-error events age out. Check both window type and recent deploys before chasing model bugs.
