In the "H264 structure, the data encoded after the video image is called one frame, and one frame consists of a slice (Slice) or a plurality of sheets, and one piece consists of one or more macroblocks (MB).
Three different data forms in the H264 encoding process:
SODB data bit string ----> The most original code data, the VCL data; RBSP original byte sequence load ----> After the SODB is filled with end bits (RBSP Trailing Bits A bit "1") Several Bit "0", in order to be byte alignment; EBSP extended byte sequence load ---> Fill the imitation check byte (0x03) based on RBSP is: When NALU is added to Annexb, you need Add the start code startCodeprefix before each group of NALU, if the corresponding SLICE corresponding to the NALU is indicated by the 4-bit byte, otherwise the OX000001 (part of a frame) is indicated by the 3-bit byte (part of one frame). Further, in order to make the NALU body conflict with the start code, when encoding, each of the two bytes is continued to be 0, the 0x03 of one byte is inserted. Remove 0x03 when decoding. Also known as a shell operation.
Nal
NAL is full of NetWork Abstract Layer, ie network abstraction layers. In the H.264 / AVC video coding standard, the entire system framework is divided into two levels: video coding level (VCL) and network abstraction level (NAL). Among them, the former is responsible for effectively representing the content of video data, while the latter is responsible for formatting data and provides header information to ensure that data is suitable for transmission of various channels and storage media. The NAL unit is the basic grammar structure of the NAL, which includes one byte header information and a series of byte streams called the original byte sequence load (RBSP) from VCL.
Frame format
The H264 is transmitted in the network NALU, NALU structure is: NAL head + RBSP, the data flow in the actual transmission is shown in the figure:
Be
Be
Be
The H264 frame consists of NALU head and NALU body. The NALU head consists of one byte, and its syntax is as follows:
+------------- + | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | + - + + - + + - + + - + - + - + | F | NRI | TYPE | + --------------- +
F: 1 bits. Forbidden_Zero_bit. This bit must be 0 in the H.264 specification.
NRI: 2 bits. NAL_REF_IDC. Take 00 ~ 11, seem to indicate the importance of this NALU, such as 00 NALU decoder can discard it without affecting the playback of the image, the greater the value, indicating the current NAL It is important to be prioritized. If the current NAL is a film belonging to the reference frame, or when the sequence parameter set, or the image parameter set, this syntax element must be greater than 0.
TYPE: 5 bits.
Identify the RBSP data type in the NAL unit, where Nal_Unit_Type is 1, 2, 3, 4, 5 NAL units referred to as VCL NAL units, and other types of NAL units are non-VCL NAL units.
NAL_UNIT_TYPE. Type of this NALU unit, 1 to 12 are used by H.264, 24 ~ 31 is used by the application other than H.264, as is it:
0 No 1-23 NAL unit single NAL unit package 1 district, non-IDR image sheet 2 partition A 3 part partition B 4 partitions C 5 IDR image 2 Supplemental Enhancement Information Unit (SEI) 7 SPS 8 PPS 9 Sequence End 10 Sequence End 11 Code Distance Bottom 12 Filled 13-23 Reserved
24 STAP-A SLR Capsule 25 STAP-B Single Time Combination Pack 26 MTAP16 Multiple Time Combination Pack 27 MTAP24 Multiple Time Multiple Time Multiple Time Groups 28 Fu-A Split Unit 29 FU-B Split Unit 30-31 is not defined
Be
Be
Due to no length information, the actual transmission is not given in the syntax of the NAL, the storage system needs to increase additional headers to implement the boundaries of each NAL unit.
Among them, the AVI file and the MPEG TS broadcast stream take the word current flow syntax format, that is, the synchronization code of 0x00000001 before the NAL unit, one H.264 video frame read from the AVI file or the MPEG TS PES package is below The form of surface exists:
00 00 01 06 ... 00 00 00 01 67 ... 00 00 00 01 68 ... 00 00 00 01 65 ... SEI Information SPS PPS IDR SLICE
For the MP4 file, the NAL unit has no synchronization code, but there are several bytes of length code to indicate the length of the NAL unit, which is given by the number of bytes occupied by the MP4 file; in addition, read from MP4 The video frame does not contain PPS and SPS, which is in the MP4 file header, and the parser must get them when the file is opened. One H.264 frame read from the MP4 file is often the following form (assuming the length code is 2 bytes):
00 19 06 [... 25 bytes ... 24 AA 65 [... 9386 bytes ...] SEI information idR slice
Bracket
Be
H264 OVER RTP is basically divided into three types:
(1) Single Nal Unit Packet is also the actual NAL type, which can be understood as a package is a frame of H264 data, which is more in practice.
(2) AGGREGATION Packet One pack data contains multiple H264 frames.
The frames in the STAP-A package contain the same NALU-TIME, no DON
The frames in the STAP-B package contain the same NALU-TIME, with DON
Frames in the MTAP16 package include different NALU-TIME, TIMESTAMP OFFSET = 16
Frames in the MTAP24 package include different nalu-time, timestamp offset = 24
Packed NAL unit size in aggregation packet is 65535 bytes
(3) Fragmentation Unit One frame data is divided into multiple RTP packets, which is also common, especially for keyframes. There are two versions of FU-A, Fu-b.
Be
When the H264 package is transmitted, if the bag is too large, it will be divided into multiple pieces. The NALU head will be replaced by 2 below.
The FU INDCATOR OCTET HAS The Following Format:
+------------- + | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | + - + + - + + - + + - + - + - + | F | NRI | TYPE | + --------------- +
Don't be scared to this format is the RTP H264 load type mentioned above, TYPE is Fu-a
The Fu Header Has The Following Format:
+------------- + | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | + - + + - + + - + + - + - + - + | S | e | r | type | +------------- +
S bit is 1 represents the NAL of the fragmentation. When it is 1, E cannot end for 1 e bit 1, when it is 1, S cannot be 1
R bit reserved
Type is Type in the nalu head, takes 1-23 value
AU
The general document does not describe the AUD. In fact, this is a frame started. The byte order is: 00 00 00 01 09 F0 From the structure, there is START Code, so it is indeed a NALU, the type 09 is in the H264 definition It is AUD (splitter). Most players can play normally without AUD. Follow the AUD, usually the combination of SPS / PPS / SEI / IDR or simple is a slice, which is the beginning of a frame. Like a player like flash, each time you need a complete frame data, then the data between the 2 AUD is packaged to the player according to the format.
When the H.264 encodes, add the start code 0x000001 before each NAL, the decoder detects the start code in the code stream, and the current NAL ends. In order to prevent the 0x000001 data from 0x000001, H.264 also proposes a 'Preventing the Emulation Prevention "mechanism. When you encode one NAL, if there is a continuous two 0x00 bytes, an 0x03 is inserted later. When the decoder detects the 0x000003 data in the NAL, the 0x03 is discarded and the original data is restored. 0x000000 >>>>>> 0x00000300 0x000001 >>>>>> 0x00000301 0x000002 >>>>> 0x0000030 0X000003 >>>>>> 0x00000303
In general, there are two ways to pack the H264's code stream, a format for annex-b byte stream format, this is the default output format of the most encoder, which is 3 to 4 out of each frame. The byte is the start_code, 0x00000001 or 0x000001 of the H264. The other is the original NAL packaging format, which is the beginning of the number of bytes (1, 2, 4 bytes) is the length of NAL, not START_CODE, and it is necessary to obtain a global data to obtain the encoder's Profile, Level, PPS, SPS and other information can be decoded.
Analysis of SPS, PPS
SPS profile_idc and level_idc are all configuration and levels compliant by the Biocircraft.
constraint_set0_flag is equal to 1 means all the regulations in a certain section. constraint_set0_flag is equal to 0 means that the bit stream can follow all regulations in a certain section. When profile_idc is equal to 100, 110, 122 or 144, construct_set0_flag, construct_set1_flag, and constraint_set2_flag should be equal to 0.
The value of log2_max_frame_num_minus 4 should be within 0-12 (including 0 and 12), this syntax element is mainly for reading another syntax element frame_num service, Frame_num is one of the most important syntax elements, which identifies the decoding order of the object belonging. . This syntax element also specifies the maximum value that Frame_Num can achieve: maxframenum = 2 * Exp (log2_max_frame_num_minus4 + 4).
PIC_ORDER_CNT_TYPE is a counting method that decodes image sequence. The value of PIC_ORDER_CNT_TYPE is 0 to 2 (including 0 and 2).
LOG2_MAX_PIC_ORDER_CNT_LSB_MINUS4 indicates the value of the variable maxpicordercntlsb in the image sequence number decoding process for a certain design,
Num_ref_frames specifies short-term reference frames and long-term reference frames, complementary reference field pairs, complementary reference field pairs, and unspeakable reference field pairs, complementary reference fields, and unparalleled reference fields. Num_REF_FRAMES The value range should be in 0 to Maxdpbsize.
GAPS_IN_FRAME_NUM_VALUE_ALLOWED_FLAG indicates the permissible value of the frame_num given in a certain section and the decoding process in the case where there is a difference between the FRAME_NUM value given in some section.
PIC_WIDTH_IN_MBS_MINUS1 plus 1 refers to the width of each decoded image of the macroblock. The semantics of PIC_HEIGHT_IN_MAP_UNITS_MINUS1 depends on the variable frame_mbs_only_flag, which is as follows: - If FRAME_MBS_ONLY_FLAG is equal to 0,
PIC_HEIGHT_IN_MAP_UNITS_MINUS1 plus 1 indicates a high of a macroblock unitSpend. - Otherwise (frame_mbs_only_flag is equal to 1), PIC_HEIGHT_IN_MAP_UNITS_MINUS1 plus 1
The height of one frame of macroblock is. The variable FrameHeightinMBS is drawn from the following formula: frameheightinmbs = (2 - frame_mbs_only_flag) * PicheightInmapUnits.
MB_ADAPTIVE_FRAME_FIELD_FLAG equal to 0 indicates that there is no exchange between frame and field macroblock in an image. MB_ADAPTIVE_FRAME_FIELD_FLAG equal to 1 indicates that there may be exchange between the frame and the frame of the frame. When MB_ADAPTIVE_FRAME_FIELD_FLAG is not specified, the default value is 0.
Direct_8x8_inference_flag The method of using the B_SKIP, B_DIRECT_16X16 and B_SKIP, B_DIRECT_16X16, and B_Direct_8x8 brightness motion vector specified in a certain section. When Frame_MBS_ONLY_FLAG is equal to 0
Direct_8x8_inference_flag should be equal to 1.
Frame_cropping_flag is equal to 1 indicating the next value in which the frame shear offset parameter complies with the video sequence parameter set. Frame_cropping_flag is equal to 0 means that there is no frame shear offset parameter.
VUI_Parameters_Present_flag is equal to 1 means the VUI_Parameters () syntax structure that exists as mentioned by Appendix E. VUI_Parameters_Present_flag is equal to 0 indicates that there is no VUI_Parameters () syntax structure as mentioned by Appendix E.
PPS SEQ_PARAMETER_SET_ID refers to the sequence parameter set of the activity. The value of the variable SEQ_PARAMETER_SET_ID should be within 0 to 31 (including 0 and 31).
Entropy_coding_mode_flag The entropy encoding method for selecting syntax elements is represented by two identifiers in the syntax table. The specific method is as follows: if enpy_coding_mode_flag is equal to 0, then the method specified by the descriptor on the left side of the syntax table. PIC_ORDER_PRESENT_FLAG is equal to 1 representing the syntax elements associated with image sequence will appear in the top, PIC_ORDER_PRESENT_FLAG equal to 0 indicates that the syntax elements associated with the image sequence in the strip header.
Num_slice_groups_minus1 plus 1 represents the number of strip groups in an image. When num_slice_groups_minus1 is equal to 0, all strips in the image belong to the same strip group.
NUM_REF_IDX_L0_AMTIVE_MINUS1 indicates the maximum reference index number of the reference image list 0, which will use a list 0 to predict these strips of the Num_REF_IDX_ACTIVE_OVERRIDE_FLAG equal to 0, when using a list 0 to predict the strip of the image. When MbaffRameFlag is equal to 1, NUM_REF_IDX_L0_ACTIVE_MINUS1 is the maximum index number value of the frame macroblock decoding, and 2 * num_ref_idx_l0_active_minus1 + 1 is the maximum index number value decoded by the field macroblock decoding. Num_ref_idx_l0_Active_minus1 The value of 0 to 31 should be within 0 to 31 (including 0 and 31).
Weighted_pred_flag is equal to 0 indicates that weighted prediction is not applied to P and SP bands. Weighted_pred_flag is equal to 1 indicates that weighted forecasts should be used in P and SP strips.
Weighted_BipRed_idc is equal to 0 means B band should adopt the default weighting prediction. Weighted_BipRed_IDC is equal to 1 means that the B strip should adopt specific indicated weighted predictions. Weighted_BipRed_IDC is equal to 2 indicates that the B strip should adopt an implicit weighting prediction. Weighted_BipRed_idc's value should be between 0 and 2 (including 0 and 2).
PIC_INIT_QP_MINUS26 indicates that the SliceQPy initial value of each strip is reduced by 26. When a non-zero-value Slice_QP_DELTA is decoded, the initial value is further corrected when the strip layer is corrected, and is further corrected when the macroblock layer decodes the MB_QP_DELTA of the 0 value. The value of PIC_INIT_QP_MINUS26 should be between - (26 + QPBDOffSenstries) to +25 (including boundary values).
PIC_INIT_QS_MINUS26 indicates that the SliceQsy initial value of all macroblocks in the SP or Si strip is reduced by 26. When a non-zero-value SLICE_QS_DELTA is decoded, the initial value is corrected at the strip layer. The value of PIC_INIT_QS_MINUS26 should be between -26 to +25 (including boundary values).
Chroma_qp_index_offset is the offset of the parameter QPY and QSY to be added to the table in the table of the QPC value. The value of chroma_qp_index_offset should be in the range of -12 to +12 (including boundary values).
DEBLOCKING_FILTER_CONTROL_PRESENT_FLAG equal to 1 Indicates a set of syntax elements that control the feature of the block effect filter will appear in the top of the head. DEBLOCKING_FILTER_CONTROL_PRESENT_FLAG equal to 0 Indicates a set of syntax elements that control the feature of the block effect filter does not appear in the top, and their estimated values will take effect.
Constrained_intra_pred_flag is equal to 0 indicates that the intra prediction allows the use of residual data, and the prediction of macroblocks encoded using an intra macroblock prediction mode can use the decoded sample of the adjacent macroblock encoded by the inter-frame macroblock prediction mode. Constrained_intra_pred_flag is equal to 1 representing a restricted intra prediction, in which case the prediction of the macroblock encoded using the intra macroblock prediction mode is used only using the residual data and the decoded sample from the I or Si macroblock type.
Redundant_pic_cnt_present_flag is equal to 0 indicates that the redundant_pic_cnt syntax element does not indicate the header, the image parameter set indicates (directly or directly or associated with the corresponding data division block A). Redundant_PIC_CNT_PRESENT_FLAG equal to 1 indicating that the reduction block B and data split block C which indicates (directly or associated with the corresponding data division block A) indicated (directly or associated with the corresponding data division block A) in the bar head and image parameters.
SPS parsing is high
UINT UE (Byte * PBUFF, UINT NLEN, UINT & NSTARTBIT) {// Calculated 0BIT NZERONUM = 0; While (NStartbit
> (NStartBit) % 8)))))) // &: Bit and,% Take the {Break;} NZERONUM ++; NStartBit ++;} NStartBit ++; // Calculation result DWORD dwret = 0; for (uint i = 0; i > (NStartBit% 8))) {dwret + = 1;} nStartBit ++;} Return (1 << Nzeronum) - 1 + dwret;} int SE (Byte * PBUFF, UINT NLEN, UINT & NStartbit) {INT UEVAL = UE (PBUFF, NLEN, NSTART); Double K = UEVAL; INT NVALUE = CEIL (k / 2); // CEIL function: CEIL function is Seeking minimum integers that are not less than gived. CEIL (2) = CEIL (1.2) = CEI (1.5) = 2.00 IF (UEVAL% 2 == 0) nvalue = -nvalue; return nvalue;} DWORD u (uint bitcount, byte * buf, uint) {dWord DWRET = 0; for (uint i = 0; i > (NStartBit% 8))) {dwret + = 1;} nStartBit ++;} return dwRet;} bool h264_decode_seq_parameter_set (BYTE * buf, UINT nLen, int & Width, int & Height) {UINT StartBit = 0; int forbidden_zero_bit = u (1, buf, StartBit); int nal_ref_idc = u (2, buf, Startbit); int nal_unit_type = u (5, buf, startbit); if (nal_unit_type == 7) {INT profile_idc = u (8, buf, startbit); int connection_set0_flag = u (1, buf, startbit); // ( BUF [1] & 0x80) >> 7; int condition_set1_flag = u (1, buf, startbit); // (BUF [1] & 0x40) >> 6; int condition_set2_flag = u (1, buf, startbit); / / (BUF [1] & 0x20) >> 5; int condition_set3_flag = u (1, buf, startbit); // (BUF [1] & 0x10) >> 4; Int Reser VED_ZERO_4BITS = U (4, BUF, Startbit); INT level_idc = u (8, buf, startbit); int seq_parameter_set_id = ue (buf, nlen, startbit); if (profile_idc> = 100) {Int chroma_format_idc = UE (BUF, nLen, StartBit); if (chroma_format_idc == 3) {int residual_colour_transform_flag = u (1, buf, StartBit);} int bit_depth_luma_minus8 = Ue (buf, nLen, StartBit); int bit_depth_chroma_minus8 = Ue (buf, nLen, StartBit); INT QPPRIME_Y_ZERO_TRANSFORM_BYPASS_FLAG = U (1, BUF, STARTBIT); int seq_scaling_matrix_present_flag = u (1, buf, startbit); int SEQ_SCALING_LIst_present_flag [8]; if (seq_scaling_matrix_present_flag) {for (int i = 0; i <8; i ++) {seq_scaling_list_present_flag [i] = u (1, buf, StartBit);}}} int log2_max_frame_num_minus4 = Ue (buf, nLen, StartBit); int pic_order_cnt_type = Ue (buf, nLen, StartBit); if (pic_order_cnt_type == 0) {int log2_max_pic_order_cnt_lsb_minus4 = Ue (buf, nLen, StartBit);} else if (pic_order_cnt_type == 1) {int delta_pic_order_always_zero_flag = u ( 1, buf, StartBit); int offset_for_non_ref_pic = Se (buf, nLen, StartBit); int offset_for_top_to_bottom_field = Se (buf, nLen, StartBit); int num_ref_frames_in_pic_order_cnt_cycle = Ue (buf, nLen, StartBit); / * 0-255 * / INT * OFFSET_FOR_REF_FRAME = new int [NUM_REF_FRAMES_IN_PIC_ORDER_CNT_CYCLE]; for (int i = 0; i 
