"Keywords:
QoS, video coding
1、 Foreword
In the past 20 years, Internet, mobile communication and multimedia communication have achieved unprecedented development and great commercial success. The integration of mobile communication and multimedia technology is accelerating, such as network architecture, low-power integrated circuits, powerful digital signal processing chips, efficient compression algorithms and so on. Video image coding and transmission technology for wireless network and Internet has become the forefront of information science and technology.
In 2003, ISO / IEC's moving picture expert group (MPEG) and ITU-T's video coding expert group (VCEG) jointly formulated the latest third generation video coding standard h.264/avc. Its main purpose is to provide higher coding efficiency and better network adaptability. Under the same reconstructed image quality, compared with H.263 + and MPEG-4 ASP standards, it can save 50% of the code stream; The layered mode is adopted, and the video coding layer (VCL) and network extraction layer (NAL) are defined. The latter is specially designed for network transmission, which can adapt to video transmission in different networks and further improve the "affinity" of the network. H. 264 introduces IP packet oriented coding mechanism, which is conducive to packet transmission in the network and supports video streaming media transmission in the network; It has strong anti error characteristics, especially suitable for wireless video transmission with high packet loss rate and serious interference.
2、 Review of fault tolerant algorithms for video communication
At present, video coding and compression standards mainly include mpeg-x and H.26X series. These compression algorithms are based on macroblocks to improve coding efficiency from three aspects:
(1) Motion estimation / motion compensation (MP / MC) eliminates video time redundancy;
(2) Discrete cosine transform (DCT) of image difference eliminates spatial redundancy;
(3) Variable length coding (VLC) of quantization coefficients eliminates statistical redundancy.
Practice shows that through the above methods, the video coding standard obtains high compression efficiency. However, there are still some thorny problems in the transmission of compressed code streams on the Internet, especially on wireless channels. One of the more prominent points is: on the one hand, these compressed code streams are very sensitive to channel bit error; On the other hand, due to multipath reflection and fading, wireless channel introduces a large number of random and burst errors, which affects the normal transmission of code stream. Especially when the VLC scheme is adopted, the code stream is more vulnerable to the influence of bit error. As a result, the synchronization between the decoder and the encoder will be lost, resulting in the inability to decode the VLC codeword correctly before the next synchronization codeword is encountered; At the same time, predictive coding technology will spread the error to the whole video sequence, which greatly reduces the quality of the reconstructed image. Therefore, in order to achieve good quality video transmission, we must take some fault-tolerant measures combined with the transmission characteristics of the practical application channel.
According to the different positions in the video transmission system, fault-tolerant algorithms can be divided into encoder based fault-tolerant algorithms, decoder based fault-tolerant algorithms and feedback channel based fault-tolerant algorithms. Of which:
(1) The fault-tolerant algorithm based on encoder adds redundant information to the recoded bit stream, which is added to the source or channel encoder, which reduces the coding efficiency and increases the implementation complexity in exchange for the fault-tolerant performance of coding, including layered coding, multi description coding, independent segmented coding Resynchronization coding and forward error correction coding (FEC).
(2) The fault-tolerant algorithm based on decoder is to use the correlation between the damaged macroblock and its adjacent macroblock to complete the recovery work, which includes error detection and error recovery. For error detection, error detection for syntax and embedded data are generally used; For error recovery, error concealment methods in time domain and space domain can be used.
(3) The fault-tolerant algorithm based on feedback channel refers to a way to obtain error information by decoder and transmit it to encoder for error processing through feedback channel. It mainly includes: error tracking, conditional ARQ, intra / inter coding mode selection and reference image selection mode.
At the same time, in the source encoder, the research on its anti error performance from the structure of video stream has become a research hotspot in recent two years. H. As the latest video coding standard, 264 / AVC has taken a series of practical technical measures to improve the network adaptability and enhance the robustness of data anti error code, so as to ensure the QoS of compressed video after video transmission. Different from previous video coding standards, h.264/avc standard defines video coding layer (VCL) and network abstraction layer (NAL) from the system level. The video coding layer is independent of the network, mainly including the core compression engine and the syntax definitions of blocks, macroblocks and slices. By introducing a series of new features, not only the coding and compression efficiency of H.264 is nearly doubled, but also a variety of error recovery tools enhance the robustness of video stream. The main function of the network extraction layer is to define the data encapsulation format and adapt the bit string generated by VCL to a variety of networks and multiple environments. Syntax definitions above slice level, including data representation required for independent slice decoding, similar to image and header sequence data in previous video compression standards; Anti competitive coding; Additional enhancement information and bit string of the encoding chip.
H. 264 separates nal and VCL from the framework structure for two main purposes: first, the interface between VCL video compression processing and nal network transmission mechanism can be defined, which allows the design of video coding layer VCL to be transplanted on different processor platforms, regardless of the data sealing format of nal layer; Second, both VCL and nal are designed to work in different transmission environments. Heterogeneous network environments do not need to reconstruct and recode VCL bit streams. The QoS of VCL and nal for video transmission is analyzed below.
3、 Error recovery of video coding layer of H.264 [1,4]
In H.261, H.263, MPEG-1, MPEG-2 and MPEG-4, many error recovery tools have been well applied: different forms of image segmentation (slice and block group), I-mode macroblock, slice and image interpolation, reference image selection (with and without feedback, image level, gob / slice or MB level), data segmentation, etc.
H. 264 standard inherits some excellent error recovery tools in the previous video coding standards, and also improves and innovates a variety of error recovery tools. This paper mainly introduces the error recovery tools of H.264, including parameter set, flexible macroblock sorting and redundant chip rs.
1. Parameter set
Parameter set is a new concept of H.264 standard. It is a method to enhance the ability of error recovery by improving the structure of video bitstream. H. The parameter set of 264 is divided into sequence parameter set and image parameter set. The sequence parameter set includes all information of an image sequence, that is, all image information between two IDR images. The image parameter set includes all relevant information of all segments of an image, including image type, serial number, etc. the loss of some serial numbers during decoding can be used to check whether the information packet is lost or not. A plurality of different sequence and image parameter sets are stored in the decoder. The encoder selects the appropriate parameter set according to the storage position of the head of each coding fragment. The image parameter set itself also includes the reference information of the sequence parameter set used.
As we all know, the loss of some key information bits (such as sequence and image header information) will cause serious negative effects of decoding. H.264 separates these key information and ensures correct transmission in error prone environment by virtue of the design of parameter set. The design of this code stream structure undoubtedly enhances the error recovery ability of code stream transmission.
The specific implementation methods of parameter set are also diversified: (1) through out of band transmission, which requires that the parameter set be transmitted to the decoder before the first chip coding arrives through a reliable protocol（ 2) Through in band transmission, it is necessary to provide a higher level of protection for the parameter set, such as sending replication packets to ensure that at least one reaches the target（ 3) The encoder and decoder use hardware to process the parameter set.
2. Film, film group and FMO
An image consists of several pieces, each containing a series of macroblocks (MB). MB can be arranged in raster scanning order or not. Each slice is decoded independently, and the macroblocks of different slices cannot be used as prediction reference in their own slice. Therefore, the chip setting will not cause bit error diffusion.
Flexible macroblock sorting FMO is a major feature of H.264 and is suitable for the application of basic level and extended level of H.264.
Image internal prediction mechanisms, such as intra prediction or motion vector prediction, allow only spatially adjacent macroblocks in the same slice group. FMO allocates each macroblock to slices not in the scanning order through macroblock allocation mapping technology. FMO mode divides images in a variety of modes, including checkerboard mode, rectangular mode, etc. Of course, FMO mode can also divide the macroblocks in a frame in order, so that the size of the segmented slice is smaller than the MTU size of the wireless network, and the image data segmented by FMO mode can be transmitted separately.
All MBS are divided into slice group 0 and slice group 1, which are represented by yellow and white respectively. When the white slice is lost, because the surrounding macroblocks belong to the macroblocks of other slices, using the neighborhood correlation, some weighting of the Yellow slice macroblock can be used to replace the corresponding macroblock of the white slice. This error concealment mechanism can significantly improve the anti error performance. Experiments show that in the video conference of CIF image, when the packet loss rate is up to 10%, the video distortion is low enough to require trained eyes to recognize. The cost of using FMO is to slightly reduce the coding efficiency (because it breaks the prediction between non neighbor MB) and have high delay in a highly optimized environment.
3. Data segmentation
Generally, the data of a macroblock is stored together to form a slice. Data division makes the macroblock data in a slice recombine, and the semantically related data of the macroblock form a division, which assembles the slice. H. The 264 video coding standard uses three different types of data segmentation.
(1) Type a segmentation
Type a segmentation is the division of header information, including macroblock type, quantization parameters and motion vector. This information is the most important.
(2) Type B segmentation
Type B segmentation is the division of intra frame information, including intra CBPs and intra coefficients. Intra frame information can prevent the propagation of errors. This type of data segmentation requires that the type a segmentation of a given slice is effective. Compared with inter frame information, intra frame information can better prevent drift effect, so it is more important than inter frame segmentation.
(3) C-type segmentation
C-type segmentation is the division of inter frame information, including inter frame CBPs and inter frame coefficients. Generally, it is the largest partition of coding slice. Inter frame segmentation is the least important, and its use requires type a segmentation to be effective.
When using data segmentation, the source encoder arranges different types of segmentation in three different buffers. At the same time, the size of the partition must be adjusted to ensure that it is less than the MTU length. Therefore, it is the encoder rather than nal to realize data segmentation. On the decoder, all segmentation is used for information reconstruction. In this way, if the intra frame or inter frame information is lost, the effective frame header information can still be used to improve the error concealment efficiency, that is, the effective macroblock type and motion vector retain the basic characteristics of the macroblock, so a fairly high information reconstruction quality can still be obtained, but only the detail information is lost.
4. Redundant chip method
H. The selection of reference image in H.264 is the same as that in H.263. In the feedback based system, when the decoder receives the lost or damaged image information, it selects the correct reference macroblock in the reference image sequence for error recovery; For the system without feedback, H.264 proposes redundant fragment coding.
Redundant slicing allows the encoder to add one or more redundant representations of the same MB to the same code stream. It should be noted that the coding parameters of these redundant chips are different from those of non redundant chips. For example, the main chip can be coded with low QP (high quality), and the redundant information can be coded with a high QP (low quality), so the quality is rough but the code rate is lower. When reconstructing, the decoder first uses the main chip, and discards the redundant chip if it is available; If the main slice is lost (for example, due to the loss of packets), the redundant slice can also be used for reconfiguration. Redundant chips are mainly used to support mobile environments with high bit error.
5. Intra coding
H. The intra coding in 264 is generally similar to the previous video coding standards, but important improvements have been made, mainly reflected in:
(1) The reference macroblock of the intra prediction macroblock in H.264 can be the inter coding macroblock. The intra prediction macroblock is not the same as the intra coding in H.263. The predicted intra coding has better coding efficiency than the non predicted intra coding, but reduces the resynchronization performance of the intra coding. This performance can be restored by setting the limited intra prediction flag.
(2) There are two kinds of slices containing only intra macroblocks: one is intra slice (I slice) and the other is immediate brush slice (IDR slice). The immediate refresh slice needs to exist in the immediate refresh image (IDR picture). Compared with the short-term reference image, the immediate refresh image has stronger resynchronization performance.
In order to be more suitable for applications in wireless IP network environment, H.264 improves the resynchronization performance of intra image by using rate distortion optimized coding and setting intra prediction flag.
4、 Error recovery of network extraction layer in H.264
Nal supports many packet based wired / wireless communication networks, such as H.320, MPEG-2 and RTP / IP. However, at present, the network protocol level adopted by most video applications is RTP / UDP / IP, so the following description is mainly based on this transmission framework. Next, we first analyze the basic processing unit Nalu of nal layer and its network encapsulation, segmentation and merging methods.
1. Nal unit
Each nal unit is a variable length byte string of a certain syntax element, including header information containing one byte (used to represent data type) and load data of several integer bytes. A nal unit can carry a coding chip, a / B / C data segmentation or a sequence or image parameter set.
NAL units are transmitted sequentially by RTP serial number. Where, t is the load data type, accounting for 5bit; R is the importance indicator bit, accounting for 2 bits; The last f is the forbidden bit, accounting for 1 bit. The details are as follows:
(1) Nalu type bit
It can represent 32 different types of features of Nalu. Types 1 ~ 12 are defined by H.264, and types 24 ~ 31 are used outside H.264. RTP load specification uses some of these values to define packet aggregation and splitting, others
Our other product:
