BlueprintLabs logo
Blog

Infrastructure//2 min read

VPS AI engineer: why alert quality matters more than alert volume, and how we are fixing it

February work on the VPS AI engineer was about reducing noise and anchoring AI suggestions to runbook logic. The goal is a system operators can trust, not one they have to second-guess.

blueprint labs

One of the most common failure modes when adding AI to an operations workflow is that it makes the noise problem worse. The system surfaces more information faster, which sounds like progress, but if the information does not help the operator make a better decision, speed is not an improvement. It is just faster noise.

The VPS AI engineer had this problem. Alert volume was high, a significant proportion of it was duplicate or redundant signals, and the suggestions coming out of the AI layer were not consistently tied to anything the operator could act on in a predictable way. The system was fluent but not reliable, and fluent-but-not-reliable is often harder to work with than a simpler system that is wrong less often.

February was focused on two things: reducing the noise and grounding the output.

On the noise side, we introduced tighter grouping logic for incoming signals. Alerts that represent the same underlying condition from different monitoring sources now consolidate rather than firing separately. This sounds straightforward and took longer than expected to get right, because the conditions that make two alerts the same underlying issue are context-dependent in ways that are not obvious to encode. The result is meaningfully lower alert fatigue for the people running production systems through the tool.

On the grounding side, we aligned AI suggestions to internal runbook patterns. This is the more interesting architectural decision. An AI system that generates freeform suggestions for every incident is useful up to a point. But operators in production environments are often working under time pressure and cognitive load. A suggestion that fits a known pattern they can act on immediately is more valuable than a novel suggestion that requires evaluation time they do not have. By anchoring the AI output to runbook logic, the suggestions become more consistent and more immediately actionable, even if they are occasionally less creative.

The tradeoff is that runbook-anchored suggestions will not handle genuinely novel incidents well. That is acceptable. Novel incidents require human judgment anyway. The AI's job in those cases is to surface relevant context, not to prescribe a response.

The system is getting more trustworthy because it is anchored to operational discipline. That is the direction we are continuing in.