Packet Loss: What are the Symptoms, Causes, and Resolutions in an IP Video-Surveillance System?

Issue

Symptoms include the following:

Part or all of a video image appears smeared. (See the example below.)

FIGURE 1
Video Smearing

The issue is usually intermittent. At other times, the picture may appear fine.

Pixelation, whereby blocks of the screen appear inconsistent with the rest of the picture:

FIGURE 2
Video Pixelation

 

In some cases, packet loss can also result in the following issues:

  • Devices being marked offline in a video-management system.
  • Missing recording segments on a video timeline (known as 'video gaps')

It is recommended that packet loss be kept under 0.5% of all video traffic. More than that amount can result in the above symptoms.

Product Line

Pelco Cameras, Pelco Video Management

Environment

Pelco IP cameras, encoders, recorders, and video management systems.

Cause

Packet loss can occur at the camera, at the switch, at the recorder, or at the viewing station. Possible causes include the following:

  1. A bad network cable
  2. A bad network port
  3. A duplex mismatch
  4. Excessive port utilization
  5. High CPU, RAM, or hard-drive utilization

Further, if any of the network links are wireless, RF interference can cause packet loss. For that reason, wireless links are not recommended in a surveillance system.

Resolution

First, determine where in the network the packets are being dropped. Video packets in a Pelco system are typically sent using UDP. Unlike TCP, UDP datagrams are not resent if they are dropped. Therefore, only one device (the switch, the camera, the recorder, or the viewing station) will have a record of the dropped packet.

Linux

Pelco IP cameras and Endura devices use Linux-based operating systems. The two most-common tools to measure packet loss on a Linux system are the ifconfig and ethtool commands. To view summary statistics for the primary interface on a Linux device, type ifconfig eth0 and press ENTER. Here is the typical output from that command:

eth0  Link encap:Ethernet  HWaddr 00:25:90:76:6B:82
          inet addr:192.168.5.159  Bcast:192.168.5.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4609645089 errors:62 dropped:0 overruns:0 frame:62
          TX packets:7513114 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5744158387593 (5478056.3 Mb)  TX bytes:3966165812 (3782.4 Mb)
          Interrupt:16 Memory:f7900000-f7920000

In the example above, we see several statistics related to packet loss. The letters RX stand for "receive" and the letters TX stand for "transmit."

The first statistic of note above is errors. An error is a malformed packet. The packet may be missing certain information in the header or it may have failed a cyclical-redundancy check (CRC). If your system shows errors to the right of RX, it means that it has received malformed packets from the switch. Common root causes of this are a bad cable or a bad interface on the switch or the receiving machine.

The second statistic of note is dropped. Dropped packets can be caused by any of the following:

  1. Softnet backlog full
  2. Bad or unintended VLAN tags
  3. Unknown or unregistered protocols
  4. IPv6 frames when the server is not configured for IPv6

Of the list above, the first is the most-common cause. If the destination host's network card, CPU, or hard drive(s) are too busy, or if the host is out of RAM, then packets may be dropped. Packets dropped for this reason can also appear to the right of the overruns counter. To diagnose this, it is recommended you verify that the network port is operating at the recommended speed and duplex by typing ethtool eth0 and pressing ENTER. Network cameras must operate at 100 mbps or higher and other devices, such as viewing stations or recorders require a link at 1 gbps or higher. Further, the link must be full duplex. If it is not at full duplex, it is recommended that you hard-code the switch port to be full duplex.

The third statistic is called frame. This denotes a type of error, specifically indicating that the frame received by the host failed a cyclical-redundancy check. So, typically, both the error and frame counters will increment together.

More information about the possible causes of the dropped packets can be obtained by typing the following command: ethtool -S eth0

Windows

To view packet-loss statistics on a Windows system, open a command prompt and type the following command: netstat -s

If packet loss is shown, check to make sure the network card's speed is correct by typing ncpa.cpl and right-clicking on your connection and choosing Status.

Cisco IOS

To view packet-loss statistics on a Cisco router or switch, type show interfaces and press ENTER. Here is the resulting output:

FastEthernet0/0 is administratively down, line protocol is down (disabled)
  Hardware is Lance, address is 0030.f235.2901 (bia 0030.f235.2901)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  ARP type: ARPA, ARP Timeout 04:00:00,
  Last input 00:00:08, output 00:00:05, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0 (size/max/drops); Total output drops: 0
  Queueing strategy: fifo
  Output queue :0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     0 input packets with dribble condition detected
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out