Cisco IOS Queue Wedges Explained

July 13, 2009 - 0 Comments

An interface queue wedge is a class of vulnerability in which certain packets are received and queued by a Cisco IOS router or switch, but due to a processing error, are never removed from the queue. This is a problem as there are a finite number of packets that may be queued on an individual interface. Should the queue become full, the device will be unable to receive new traffic via that interface.

Although this type of problem could affect any networked device, queue wedges are a classic security problem for Cisco IOS. There have been several instances of queue wedges on Cisco IOS that resulted in Cisco Security Advisories. Here are some examples:

In order to gain a better understanding of why these vulnerabilities exist and are considered severe, we need to examine things in a bit more detail.

Packets received by a Cisco IOS device that must be process switched are placed into an interface-specific input queue. There are many types of packets that must be process switched, such as packets destined to an IP address on the local device or transit packets that need additional processing by the device CPU, perhaps to be logged via Access Control List logging or due to the presence of an IP option.

However, strictly speaking, an interface input queue is not a queue at all. On Cisco IOS each interface has what appears to be a software queue into which received packets are stored. In reality, the input “queue” is simply a means to quantify and limit the number of packets stored for each interface. The packets are stored elsewhere in memory, and the interface input queue is just a counter that indicates the number of stored packets that happen to have been received on a particular interface. When a packet is received, the counter is incremented. When a received packet has been processed satisfactorily—even if that processing resulted in a negative response or handled error—the counter is decremented.

Therein lies the problem; it is only after a packet has been “processed satisfactorily” that the counter is decremented. If for any reason a packet is received but not completely processed, the counter might have been incremented but not decremented. Received packets are counted against the interface input “queue,” which is of a limited and relatively small size. For example, on most interface types on most platforms the default interface input queue is only 75 packets, a value that can be configured via the hold-queue value in interface configuration command. Once the input queue contains nothing but packets that, due to a bug, will never be dequeued, the queue is said to be wedged. More recently this condition has been termed as a blocked interface.

This can be seen on a Cisco IOS device when the input queue size is equal to or greater than (depending on the Cisco IOS version) the input queue max value, as shown below. In this example, the current “size” of the input queue is 75, which is equal to the “max” size of the input queue, which is also 75.

Router# show interface Ethernet 0/0
Ethernet0/0 is up, line protocol is up
  Hardware is AmdP2, address is 0001.0001.0001
  Internet address is  MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set  Keepalive set (10 sec)
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:20, output 00:00:05, output hang never
  Last clearing of "show interface" counters never
  Input queue: 75/75/44/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo  Output queue: 0/40 (size/max)
  5 minute input rate 4000 bits/sec, 9 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     2937 packets input, 182298 bytes, 0 no buffer
     Received 7 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     58 packets output, 6540 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

After an interface has been blocked, it is not possible for the Cisco IOS device to receive additional packets for local processing via that interface. However, the interface may still send traffic, or receive traffic that is to be forwarded as opposed to processed locally by the CPU. In addition to making it impossible to access the device via the blocked interface, this condition can cause ARP or the IGP to fail, effectively rendering the interface useless. (Yes, I know this excludes the esoteric Selective Packet Discard, but if you know that, you’re likely not learning anything from this post. 😉 Currently, the only way to recover from a blocked interface is to reload the device.

Cisco has provided a few detection mechanisms that may be used to identify a blocked interface on Cisco IOS. This includes an EEM script specifically for this purpose and a whitepaper describing how this condition can be detected using SNMP.

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.