Cisco Blogs

Using Synthetic Transactions for End User Experience Metrics

June 21, 2010 - 0 Comments

One of the most important aspects of voice service management is the assurance of user satisfaction.  In Cisco IT, we utilize Unified Operations Manager, a Cisco product, to measure and report on End User Experience Metrics.  Specifically, we utilize UOM’s synthetic transaction capability to replicate user experience by making periodic test calls and off hook tests.  This provides statistics of service availability for basic dial tone as well as supporting applications like Unified Messaging. 

The synthetic tests serve to replicate user activity (receiving a dial tone, making end-to-end phone calls, leaving voice mail, and creating/joining conference calls). These tests can verify the functional availability of the supporting infrastructure and validate different configuration aspects such as route patterns, route lists, inter-cluster trunks, and gateway dial peers. 

What follows is how we use thse synthetic tests at Cisco, to measure the availability and quality of our voice network.

Synthetic Transactions are tests that can be used to measure the availability of voice applications in the network. These tests verify whether the voice application can service requests from a user, such as verifying that phones can register with a Cisco Unified CallManager (CUCM). Operations Manager supports synthetic testing for the following applications:

• Cisco CallManager and Cisco CallManager Express

• Cisco TFTP Server

• Cisco Emergency Responder

• Cisco Conference Connection

• Cisco Unity, Unity Connection and Cisco Unity Express

Seven different synthetic transactions, within Operations Manager, can be used to create a test of voice services. Multiple tests can be configured against a single CUCM. In most cases, the synthetic tests can use synthetic phones, simulated by Operations Manager, to measure the availability of voice applications by emulating your actions. For example, a synthetic test places a call between clusters and then checks to see if the call is successful. If a synthetic test fails, Operations Manager generates a critical event. Such events are displayed in the Alerts and Events display. To perform the synthetic transactions, the user configures the necessary number of simulated Cisco 7960 phones in the CUCM database.

Each synthetic transaction requires a unique phone number and MAC address. Using the Operations Manager graphical user interface (GUI), the user uses the simulated phones to configure the set of transactions to run for a test against the CUCM. Operations Manager then acts as a simulated IP phone using the SIP or Skinny Client Control Protocol (SCCP) signaling protocols to request the voice services.  For each test, Operations Manager expects a certain response. If an unexpected response or no response (20 second timeout) is received, the test fails. A failed test could indicate that the services are down, the network is slow or mis-configured, or the transactions themselves were mis-configured.

Cisco IT has, thus far, only utilized the Off-hook and End-To-End Call Tests.  There are several reasons for this. 

The first limiting factor is the desire to test, notify and report on directly service impacting issues as opposed to provisioning issues.  This eliminates the phone registration and TFTP download tests from periodic availability testing.  These tests may and should still be used on an ad-hoc basis to troubleshoot individual phone issues.

Also removed because of limited service impact and to reduce the sheer number of simultaneous synthetic transactions are the Conference Connection, MWI and Emergency Call Tests.  This leaves the Off-Hook Test and End-to-End Call Test.

The Off-Hook test is used to ensure that Call Manager Subscriber Nodes are able to get dial tone. 

The End-To-End Call Test is used to test network and application availability.  For the network, this includes intra-cluster and inter-cluster scenarios.  Application availability testing is performed against telephony components of Unity. 

Off-Hook Transaction Design Detail

The Off-Hook or Dial-Tone Test simulates an off-hook state to the CUCM and checks for receipt of a dial tone. The test passes if it receives a dial tone signal from the CUCM.  This transaction is an important portion of the overall availability of the IPT service.  The design is simple.  Every Call Manager Subscriber Node shall have an Off-Hook/Dial-Tone Test configured for it.  Failure of any Off-Hook transaction will generate a critical event and alert to the CUOM Service Level and Event views. 

End-To-End Call Transaction Design Detail

The End-to-End Call Test, in IT’s implementation, initiates a call to a real IP endpoint.  These endpoints can be dedicated test phones, IPIVR ports, IPCC ports or Unity Ports.  Each End-To-End test passes if it registers, goes off-hook, and places the call; and there is a ring indication; and the destination endpoint goes off-hook to accept the call.
IT uses the End-To-End Call Test to verify network and application availability.

Intra-Cluster Network Availability

Test Scenario:                    

Represents on-net Intra-Cluster calling over the LAN/MAN.  Small number of IP Phones distributed throughout the environment set to auto-answer synthetic calls placed from CUOM.  Test scenario verifies call signaling (source & destination) and call routing (source).  Total call flows = 12 (4 regions, 3 flows per region). 

Inter-Region Availability:

Test Scenario:                    

Represents region-to-region dialing over the wan/pstn aggregate.  The target endpoint will be the Unity Hunt Pilot Number.  Test scenario verifies call signaling (source & destination), call routing (source-to-destination), and available call path (source-to-destination).  Total call flows = 12 (4 regions, 3 flows per region)

Call Source:            CUOM Server in local DC

Call Destination:     Destination Unity Hunt Pilot Number

Call Pattern:            Full Mesh (Sites A, B, C, D):

                                                –  A->B, A->C, A->D

                                                –  B->A, B->C, B->D

                                                –  C->A, C->B, C->D

                                                –  D->A, D->B, D->C 


Failure of any End-To-End transaction will generate a critical event and alert to the CUOM Service Level and Event views. 

Unity Synthetic Transactions

End-to-End synthetic calls are created to test the availability of Unity applications.  For each Unity Pair, one synthetic test shall be created on that theater’s CUOM with the synthetic phone defined on a co-located CUCM subscriber node. 

For the Unity Pair, the test call shall terminate to the Hunt Pilot Number. 

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.