How we maintain monitoring accuracy across 18 regions at scale.
Running 18 monitoring nodes across six continents sounds straightforward. Here's what it actually takes to keep every check accurate and every alert trustworthy.
Sarah Mitchell
Co-founder & CTO
The reliability of the monitor matters as much as the service being monitored
There's an uncomfortable irony in the monitoring business: your monitoring tool has to be more reliable than the things it monitors. If Sola has a false positive rate of even 0.1%, an engineering team running 500 monitors would see dozens of spurious alerts every day. That's not acceptable.
Here's how we maintain accuracy across our 18-region fleet at scale.
Node health scoring
Every monitoring node has a health score that factors in multiple dimensions. Nodes with degraded health scores are automatically weighted lower in consensus calculations, and nodes that fall below our minimum threshold are pulled from rotation entirely.
Factor | Weight | Threshold for removal |
|---|---|---|
Check success rate | 40% | < 99.5% |
Clock accuracy (NTP drift) | 25% | > 50ms drift |
Network stability | 20% | > 2% packet loss |
Response time consistency | 15% | > 3σ from baseline |
Clock synchronization across nodes
Every Sola monitoring node synchronizes its clock against multiple NTP sources before each check cycle. This matters because we calculate latency to the millisecond, and clock drift between nodes would distort our response time measurements and make our aggregated data unreliable.
We audit clock skew across all 18 nodes every 60 seconds. If any node drifts beyond our threshold, it's temporarily pulled from rotation until it resynchronizes.
Network path isolation
Each regional node is connected to multiple upstream providers. If a node's primary network path experiences degraded performance, our routing layer automatically shifts traffic to the secondary path. This prevents a single ISP issue from generating false positives for your endpoints.
We track BGP routing changes in real time across all our upstream providers. When we detect a significant routing change, we cross-reference it against our alert data to filter out any failures that are attributable to our own network rather than your service.
Consensus-based result validation
Before any alert fires, our aggregation layer runs a consensus check across regional results. A single outlier result — even from a healthy region — is weighted against the median of all other results. This catches the case where a single node has a local problem but the service is actually healthy.
This self-healing architecture means that even if 2 of our 18 nodes experience problems simultaneously, the other 16 continue to provide accurate monitoring data without any manual intervention.
