Cisco IT reduced endpoint telemetry volume by 75% while building a clearer view of device, collaboration, and network health. As we extend our observability transformation to the endpoint, the team is focusing on a practical question: how much telemetry is enough to improve troubleshooting without overwhelming systems or budgets? Learn how we improved troubleshooting and digital resilience while keeping the data footprint sustainable, and the lessons we learned along the way.
The challenge: Full-Stack visibility without overcollection
In today’s distributed enterprise, digital resilience begins at the employee’s endpoint. When an employee’s experience is disrupted, the issue may start on the physical device but show up across collaboration tools, software-as-a-service (SaaS) applications, or the network-path. For example, a choppy Webex session could be affected by central processing unit (CPU) pressure, poor Wi-Fi quality, virtual private network (VPN) state, or a network-path change.
At Cisco, that challenge plays out across a global environment, spanning 90,000+ employees, and is constantly changing. We support macOS, Windows, and Linux devices, multiple network states, collaboration applications, security agents, VPN, proxy paths, SaaS destinations, and users working from many locations.
When any layer of that stack degrades undetected, employee experience suffers and digital resilience weakens. That is why endpoint telemetry matters, but it also makes telemetry design more important. One metric can quickly multiply into many metric time series (MTS) when dimensions such as host, operating system, disk, process, state, direction, target, or hop are added. Across a fleet, and at high collection frequency, the cost curve can rise quickly.
Endpoint observability has a practical challenge: collecting every metric from every endpoint at high frequency can create more cost, more data noise, and more endpoint overhead than actual business value. We needed a different approach.
A different approach: right-sizing telemetry
We focused our endpoint observability strategy on a single question:
What is the right set of telemetry to understand endpoint performance and speed issue resolution without creating unnecessary cost, noise, or overhead?
The answer was a right-sized telemetry model implemented in Splunk Observability Cloud. This approach is defined by its ability to correlate signals across traditionally siloed domains – endpoint health, collaboration experience, and network-path behavior. By analyzing these in a unified view, support teams can move beyond generic application complaints. For example, a reported Webex issue is now immediately contextualized; telemetry can distinguish between local device pressure (CPU or memory) and network-path failures (DNS, VPN, or proxy reachability).
To implement this, we had to move from abstract debates to concrete engineering. We began by cataloging every metric produced by the Splunk OpenTelemetry Collector across our macOS, Windows, and Linux environments. This inventory allowed us to identify gaps in our default data set and pinpoint where custom metrics were required.
We then evaluated each candidate metric through a rigorous filter:
- Actionability: Does this signal trigger a specific support response?
- Utility: Does it duplicate existing data or offer a unique context?
- Efficiency: Should it be a continuous metric, or a log-based snapshot to avoid high cardinality?
- Scope: Does the entire fleet require this, or only specific cohorts?
This measurement-first approach transformed our strategy from an exercise in “collecting everything” into a precise, operational framework that treats every metric as an investment.
Eight strategies for right-sized endpoint telemetry
Based on our experience in adopting the Splunk OpenTelemetry Collector across the corporate landscape, we identified eight strategies that are key to a successful rollout of endpoint telemetry collection.
- Tune frequency to the use case. A 60-second collection interval is often enough for employee experience signals and, in our modeling, can cut datapoint volume by up to 75% compared to the Collector default 10-second interval.
- Keep the core, remove the redundant. The default metric set focuses on broadly actionable signals: CPU, memory, disk utilization, and input/output (I/O). Everything else has to earn its place.
- Treat process metrics as high-value and high-cost. Process monitoring is scoped to business-relevant applications like Webex, Duo, and Cisco Collab Host, rather than capturing every running process.
- Use fleet profiles instead of one global default. Telemetry depth is tiered to match operational need, with richer collection reserved for troubleshooting, high-risk devices, or incident response.
- Make collection adaptive. Healthy devices run lean. Devices under stress can retain the process and performance detail needed for troubleshooting, so the observability system becomes more attentive when conditions deteriorate.
- Extend observability to the network path. Failure-domain-based checks help distinguish whether problems originate at the device, local network, VPN, proxy, or SaaS edge, not just whether an endpoint can reach a destination.
- Control cardinality before it controls you. Stable rollups stay as metrics. High-cardinality route snapshots and per-sample details move to logs, where they provide troubleshooting context without driving metric time series churn.
- Deploy with guardrails. Conservative intervals, hard limits on probe targets and route snapshots, Collector self-health monitoring, and feedback loops with support teams help keep the strategy scalable without creating new cost or reliability problems.
What we are learning
Our biggest lesson is that endpoint observability is not a choice between collecting everything and accepting visibility gaps. There is a better middle path: capturing actionable signals when devices need attention while keeping telemetry lean across the rest of the fleet.
By applying these strategies, we achieved a 75% reduction in endpoint telemetry data across our endpoints, which lowers Splunk Observability Cloud licensing costs. This approach is a key extension of Cisco IT’s unified observability strategy which spans the network, platforms, and service operations.
By bringing endpoint, collaboration, and network-path signals into Splunk Observability Cloud, we bring that same digital resilience foundation closer to the employee experience. Support teams can now narrow likely failure domains earlier without requiring every endpoint to send every possible signal all the time, giving us a stronger foundation for future automation.
At the endpoint level, digital resilience depends on collecting the right data, not all data. With selective telemetry, thoughtful guardrails, and the right failure-domain focus, teams can troubleshoot more efficiently, reduce the risk of undetected employee-impacting issues, and do so in a way that is practical and sustainable.
Related resources
- Cisco IT’s observability transformation: From fragmented data to unified insights
- Splunk Observability Cloud
- Splunk OpenTelemetry Collector
Explore more Cisco on Cisco stories to see how Cisco IT applies technology inside Cisco to improve resilience, operations, and employee experience.
