EnduraView in Depth

Issue

How exactly does EnduraView work?

Product Line

Pelco Video Management

Environment

  • Endura WS5000, all versions
  • Endura VCD5000, all versions
  • Endura NET5402R-HD, all versions

Cause

Not applicable.

Resolution

EnduraView is an algorithm used to select the best image quality and frame rate for a given networked video display and layout without affecting the system's recording rate.

Important considerations

The following are characteristics of audio and video sources within the Endura System:

  1. All live-video sources provide either a high-resolution and low-resolution pair of streams, or a single high-resolution stream for all video content. Unless the camera is connected using a UDI5000-CAM, it will provide a pair of live streams.
  2. All recorded video sources record all configured video content at the highest available resolution.
  3. All video sources, both live and recorded, provide an I-frame only streaming mode for all available video content.
  4. All display devices maintain a list of active streaming video sessions for display purposes. With each session, the display device maintains a least recently used (LRU) timestamp that allows the display device to determine which session among the set of active sessions was accessed least recently.

The importance of these characteristics will become apparent as this document proceeds.

Terminology

The following terminology is used throughout this document:

  • Video source. Any networked video source, be it live or recorded.
  • Video content. The image content of a networked video stream.
  • Dual source. A video source that provides a high-resolution and low-resolution pair of streams for all available video content.
  • Single source. A video source that provides a single high-resolution video stream for all available video content.
  • Primary stream. For a dual-stream source, the high-resolution stream. For a single source, the single supported stream.
  • Secondary stream. For dual-stream source, the low-resolution stream. For a single source, undefined.
  • I-frame mode. A streaming mode in which only the I-frames of a video stream are displayed by pruning all P-frames from the stream. This comes in two forms:
    • Hard I-frame mode. An I-frame mode in which P-frames are pruned at the transmitter. Only a recorder, such as an NSM5200, will transmit I-frames by themselves.
    • Soft I-frame mode. An I-frame mode in which P-frames are pruned at the receiver. This is the most-common method.
  • Streaming session. An active video session maintained by a video-display device.

How EnduraView Worked in Endura 1.0

First, the display device attempts to initiate a streaming session with the primary stream of the selected video source. Assuming that the resulting machine's performance is of an acceptable level, no further action is taken. Otherwise, the bit rate of some streams will need to be lowered in order to return to an acceptable level of performance.

Assume there is a single streaming session associated with display device. If the selected video source has two streams, the display device attempts to initialize a streaming session with the secondary stream of the video source. If this results in an acceptable level of performance, no further action is taken. Otherwise the display device initializes the primary stream in I-frame mode (either hard or soft, whichever is feasible).

Now assume that the current number of streaming sessions for the display device is more than one. We first find the session with the oldest LRU timestamp. If the source associated with that session has two streams, we reduce the bit rate on the session by initializing the secondary stream of the source. If the associated source has a single stream, we initialize the primary stream in I-frame mode (either hard or soft, whichever is feasible). If this results in an acceptable level of performance, no further action is taken. Otherwise, we find the streaming session with the next-oldest LRU timestamp, and repeat the process.

Optimizations

One drawback of the previously described algorithm is the potential number of connections that might need to be initiated in order to obtain the necessary level of performance. For instance, assuming that we are initiating a single stream for display within a performance-inhibited environment, the display device might be required to initiate the primary stream, then initiate the secondary stream, and finally initiate the primary stream again in I-frame mode. In some circumstance, the latency involved in initiating and tearing down all of these sessions may be unacceptable.

There are several optimizations that may be used to alleviate this situation:

  1. Always initiate the low-resolution stream if the number of streaming sessions being maintained exceeds some threshold. For instance, if the display device is maintaining more than four streams, and we want to reinitialize one of the sessions from a different video source that is a dual source, the display device will forego initiating the primary stream and instead initialize the secondary stream.
  2. Use only I-frame mode for bit rate reduction. That is, forego all use of the secondary stream for bit rate reduction and use I-frame mode on all currently active sessions, whether they use the primary or secondary stream.

Endura 2.0 Enhancements

Reason for modifications

EnduraView was modified for the version 2.0 of the WS5000 software, the VCD5202, and the NET5402R for the following reasons:

  • Use of the Media Processing Framework (MPF) for the streaming of video and audio data. The MPF is designed to use the graphics card resources for video rendering and CPU for decoding. By doing this, it allows the software to display more video at a higher resolution and frame rate simultaneously. This is different than older versions of the software where most of the video decoding and rendering was done by the CPU.
  • The minimum hardware specification has been increased for the WS5000 from a single-core processor to a quad-core processor. The amount of video-card memory has also been increased. The VCD5202 and NET5402R have been upgraded to quad-core processors and the video cards have 512 MB of memory.

Endura View is invoked in three different ways:

  • Video-layout stream preference. This approach is proactive, and gives the user the best possible resolution and frame rate based on the size of the video area, and the number of streams the layout can support. See the table below for the stream preference of each layout.
  • Video-card limitations. The video card can only render so many frames per second before it becomes over burdened. Therefore, we have set a limitation in the software that only 480 frames can be rendered per second. This means the user can view 16 streams at 30 FPS or any combination of lower frame rates simultaneously. The rest of the streams will display only I frames, using the soft I-Frame mode described earlier in this document.
  • CPU limitations. Since the CPU does all of the video decoding, the 2.0 software now monitors the CPU performance to help determine when to invoke EnduraView. When the CPU averages over 70% capacity in a PC running the WS5000 software, or 75% on a VCD5200 or NET5402R, for than 30 seconds, EnduraView will be invoked. Then, one of video streams will change based on the Most Recently Used List described below. If all of the video streams have been processed by EnduraView and the CPU still averages over 70% (WS5000) or 75% (VCD, NET5402R), then the user will receive a warning message stating that the machine has reached the limits of its processing power.

Most Recently Used List (MRUL)

The software now tracks and keeps a list of what video windows the user selects. This information allows the software to determine the order of which videos will be processed by EnduraView. If the machine reaches a CPU limitation or GPU limitation, the last video window in the list will be the first one to drop down to I-frame only. However there are two exceptions to this rule: The first exception is that recorded video will be the last to be processed by EnduraView. The assumption is that the user would prefer to see the recorded video at full frame rate as much as possible. The other exception is that all of the videos on the current workspace should be last to be processed by EnduraView. The assumption is that the user wants to see all of the video at full frame rate on the workspace that current video window is selected. Therefore, the application automatically puts all of the video windows on the current workspace at the top of the MRUL.

Stream Preference Order for Quad Core Machines
 
Screen Division Video Cell 1 2 3 4 5 6
1x1 Any H.264
2560x1920
30 FPS
MPEG-4
4CIF
30 FPS
MPEG-4
2CIF
30 FPS
MPEG-4
CIF
30 FPS
Not applicable Not applicable
2x2 Any H.264
2560x1920
15 FPS
H.264
720p
30 FPS
MPEG-4
4CIF
30 FPS
MPEG-4
2CIF
30 FPS
MPEG-4
CIF
30 FPS
Not applicable
3x3 Any H.264
640x512
30 FPS
MPEG-4
4CIF
30 FPS
MPEG-4
2CIF
30 FPS
MPEG-4
CIF
30 FPS
Not applicable Not applicable
3x2 Any H.264
640x512
30 FPS
MPEG-4
4CIF
30 FPS
MPEG-4
2CIF
30 FPS
MPEG-4
CIF
30 FPS
Not applicable Not applicable
4x3 Any H.264
640x512
30 FPS
MPEG-4
4CIF
30 FPS
MPEG-4
2CIF
30 FPS
MPEG-4
CIF
30 FPS
Not applicable Not applicable
4x4 Any H.264
640x512
15 FPS
H.264
640x480
30 FPS
MPEG-4
4CIF
30 FPS
MPEG-4
2CIF
30 FPS
MPEG-4
CIF
30 FPS
Not applicable
1+5 1 Large H.264
2560x1920
15 FPS
H.264
720p
30 FPS
MPEG-4
4CIF
30 FPS
MPEG-4
2CIF
30 FPS
MPEG-4
CIF
30 FPS
Not applicable
5 Small H.264
640x512
15 FPS
H.264
640x480
30 FPS
MPEG-4
4CIF
30 FPS
MPEG-4
2CIF
30 FPS
MPEG-4
CIF
30 FPS
Not applicable
1+12 1 Large H.264
2560x1920
15 FPS
H.264
720p
30 FPS
MPEG-4
4CIF
30 FPS
MPEG-4
2CIF
30 FPS
MPEG-4
CIF
30 FPS
Not applicable
12 Small H.264
800x608
15 FPS
H.264
640x480
30 FPS
MPEG-4
4CIF
30 FPS
MPEG-4
2CIF
30 FPS
MPEG-4
CIF
30 FPS
Not applicable
2+8 2 Large H.264
2560x1920
15 FPS
H.264
720p
30 FPS
MPEG-4
4CIF
30 FPS
MPEG-4
2CIF
30 FPS
MPEG-4
CIF
30 FPS
Not applicable
8 Small H.264
640x512
15 FPS
H.264
640x480
30 FPS
MPEG-4
4CIF
30 FPS
MPEG-4
2CIF
30 FPS
MPEG-4
CIF
30 FPS
Not applicable