This article refers to the "New Generation Video Compression Coding Standard ----- H.264 / AVC", and the Lei Yushen Blog "Sound Code Decoding Technology Zero Base Learning Method".
Concept section:
H.264 encoding: Video codec technology has two sets of standards, ITS (ITU-T) standard H.261, H.263, H.263 +, etc., Etc., and ISO's MPEG standard MPEG1, MPEG2, MPEG4, etc. Wait. H.264 / AVC (also known as MPEG-4 Part 10) is the latest standard for the advantages of the two major organizations H.263 + and MPEG4, the most value part is undoubtedly a higher data compression ratio. Under the same image quality conditions, the data compression ratio of H.264 is 2 times higher than H.263, 1.5 times higher than MPEG-4.
Related noun explanation:
Improve line scan: Images can be seen as a two-dimensional array, and progressive scanning is starting from row at the 0th line when the image is acquired, and then the first line is sampled until the last line is sampled.
Interlaced: A pair image is divided into two parts, 0, 2, 4, 6, 8 ..., and 1, 3, 5, 7, 9 .... This is divided into two parts to acquire image data, then a pair image can therefore be divided into two parts, we become a field, an even-numbered line, and the field of odd lines makes a bottom.
Frame: The image data obtained by the progressive scan is named frame. Image of smaller activity or stationary use frame encoding.
Field: The image data obtained by the interlaced surface is known as the field. Moving images with large activity are generally coding.
In the BIT stream of the H.264 encoder, each bit is part of a syntax element. The syntax element is organized to have a hierarchical structure, which describes the information of each level, and the syntax elements are organized into sequences, images, sheets, macroblocks, and sub-macroblocks.
Sequence: Sequence will contain multiple images
Image: One frame or one is called an image.
Slice: An image contains at least one piece. (Some blog said it is a strip)
I: Package I macroblock, I macroblock utilizes a pixel decoded from the current piece as a reference for intra prediction (the decoded pixels in other slices are not taken as a reference for intra prediction).
P Slices: Packages P and I macroblocks, the P macroblock uses the previously encoded image as a reference image for intra prediction, and an intra-coded macroblock can further make macroblock split: 16 × 16,16 × 8, 8 × 16 or 8 × 8 brightness pixel block (and the included color pixel); if the 8 × 8 sub-macroblock is selected, it can be divided into various sub-macroblocks, and the size is 8 × 8 8 × 4, 4 × 8 or 4 × 4 brightness pixel blocks (and the included color pixels).
B: Capacity B and I macroblocks, B macroblocks use two-way reference to image (currently and came from encoded image frames) for intra prediction
Macroblock: A piece contains several macroblocks. The main carrier of video information, because it contains brightness and chrominance information of each pixel. The most important work of video decoding is to provide an efficient way to get a pixel array in a macroblock in the code stream. (Macroblock is also divided into i, p, b macroblock)
Macro block: A macroblock contains a number of sub-macroblocks.
Motion estimation: divide the moving image into several macroblocks, and try to search for the position of each macroblock in the adjacent frame image, and obtain the relative offset of the spatial position between the two, the relative offset obtained is usually The motion vector of the sport vector is referred to as a motion estimate. (H.264 involves numerous algorithms, the search algorithm here is no longer more
2. Structure:
H.264 Original code stream (also known as bare stream), is a NALU that is connected, and its function is divided into two layers: video coding layer (VCL, Video Coding Layer) and network extraction layer (NAL, Network Abstract Layer. The VCL data is the output of the encoded processing, which represents the compressed encoded video data sequence, the network extract layer (NAL, NETWORK ABSTRAction Layer) encapsulates the VCL to become a NALU.
After a frame passes the H.264 encoder, it is encoded as one or more slices, and the slice of the slice is loaded, which is NALU. A piece of data is divided into several macroblocks.
3. Coding process:
H.264 codec is mainly divided into five parts: inter-frame and intra prediction, transform and reverse transformation, quantization, and reverse quantization, loop filter (Entropy Coding) ).
In contrast, the input frame or field Fn is processed by the encoder in units of macroblocks, first in accordance with the method of intra or inter predicting encoding.
Predicting encoding: The video describes a collection of continuous images. After a large number of statistics show that there is a lot of data in the front and rear images, that is, there is redundant data, then use the current image to "subtract" to the previous image, get two "difference" . Then just need "difference" to get current image information from the previous image. This difference we call it. And this difference can be seen as a two-dimensional array to see as a two-dimensional matrix. Using some mathematical methods to transform a matrix to a certain compression purpose, generally usually used the mathematical method is DCT, also known as discrete cosine transformation, the result of the transformation we have sampling again, such as: the value in the DCT transform results range From 0, 1000, then we use 0, 10 to represent this range. This process is called sampling. For example, the result after discrete cosine transformation is {0, 200, 300, 800, 801, 900, 1000}, and we can replace {0, 2, 3, 8, 8, 9, 10} after re-sampling it. This can achieve the purpose of compression. Thereafter, the quantified sequence is laminated (entropy encoding).
The predicting code is the corresponding process of the uppermost line in the corresponding above, and the current frame Fn and the previous frame F'n-1 calculate the residual DN, passed through the conversion T, after quantifying Q, reordering, and then entropy encoding is compressed. Image data.
Differential pulse code DPCM: The current pixel is X, the left is adjacent to the pixel, on the top, near the pixel, and the upper left is near the pixel C. The closer the distance between the X, the stronger the correlation, the more the closer correlation is weaker. With P as the predicted value, the distance from the X is different, the weight of this pixel is reduced as the predicted value of the X, and the difference q is subjected to the difference Q, due to the correlation of the adjacent pixels. Strong, the Q value is very small to achieve the purpose of compression coding. The receiving end adds the difference Q to the predicted value to restore the original value X. This process is called differential pulse encoding DPCM. There is a difference between the actual value and the predicted value of the current pixel, and the residual quantization value is obtained. There is a quantization error due to quantization between the decoded output and the original signal.
Motion estimation: Since there is a certain correlation in the frame of the moving image, the image is divided into several macroblocks and searches each macroblock in the adjacent image. And get the relative offset of the macroblock. The resulting relative offset is called motion vectors. The process of obtaining motion vectors is called motion estimation. In the inter prediction encoding, since there is a certain correlation in the scene near the frame in the moving image, the moving image can be divided into a plurality of blocks or macroblocks, and the location of each block or macroblock is in the adjacent frame image. The relative offset of the control position between the lights is obtained, and the relative offset obtained is the motion vector referred to, and the process of obtaining the motion vector is called motion estimation. The motion vector and after the matching prediction error are sent to the decoding end, and the corresponding block or macroblock is found in the adjacent reference frame image according to the position of the motion vector, and the prediction error is added to the prediction error, and the prediction error is added from the adjacent reference frame image in the decoded end, and the prediction error is added to the prediction error, and the prediction error is added in the decoded reference frame image in the decoding end. Get the position of the block or macroblock in the current frame. The redundancy degree of the frame can be removed by motion estimation, so that the number of bits of video transmission is greatly reduced, so motion estimation is an important part of video compression processing.
Motion notation: Since multiple objects are generally different in the imaging scene, it is more difficult to divide images into complex areas directly according to different types of motions, and the most direct and unconstrained methods are The pixel specifies the motion vector, which is based on the pixel representation. For any type of image, it is necessary to take into account a large amount of unknown amount, and the solution is much incorrectly physically unless, unless it takes appropriate physical constraints during the process. It is impossible to implement the specific implementation, which is usually used to block the block-based object motion representation.
Block matching: Generally, for the scene containing a plurality of moving objects, the method in which the generally used method is to divide one image frame into multiple blocks, so that the motion in each region can be well used as a parameterized model. The image is divided into several NXN blocks (eg: 16x16 macroblocks), for each block, motion vector, and perform motion compensation prediction coding.
Sub-pixel position interpolation: Each block or sub-macroblock division area in the inter-frame encoded macroblock is predicted according to the area of the same size in the reference frame, and the relationship between them is indicated by motion vectors, due to nature Continuity of the object, the block motion vector between the adjacent two frames is not based on the whole pixel, which may be sub-pixels such as 1/4 or 1/8 pixels as sub-pixels.
Motion Vector Medium Value Prediction: Take the motion vector of the current block using the motion vector of the left block A, the upper side block B and the upper right C, which is adjacent to the current block E. Motion Vector Medium Value Prediction: Take the motion vector of the current block using the motion vector of the left block A, the upper side block B and the upper right C, which is adjacent to the current block E.
The upper block mode motion vector: H264 is provided with 16x16, 8x16, 16, 4x8, 8x8, 8x16, 4x8, 4x4, and their image divided regions are defined as search mode Model1-Model7.
Front Frame Passing Motion Vector Prediction: Motion Vector as the Sport Vector of the Block of the Same Coordinate Location of the Premnight Block as the current block
Time domain Near reference frame motion vector prediction: Due to continuity of video sequences, the motion vector in different reference frames of the current block also has a certain coherence. Assuming the time of the current frame is T, the current reference frame T 'is currently in the previous reference frame T'.
When the optimal matching block of the current block, the motion vector of the current block RF T 'can be estimated using the motion vector of the current block re-reference frame T' + 1.
Transform Coding: Most images have a common feature, a region of flat areas and slow change in content occupies most of the images, while the detail area and mutation areas occupy a small portion. That is, the low frequency flow in the image accounts for most, and the high frequency region occupies a small portion. This image transition to the frequency domain or so-called transform domain produces a small correlation, which can compress the encoding. (About Time Domain Frequency Domain Concepts, this article does not detailed, please refer to "Fourier Transform" and "Time Domain Frequency Domain" in Baidu Encyclopedia)
Link: https://www.jianshu.com/p/dd6694fe2d9a
Our other product:
