"1, NAL" NetWork Abstract Layer, "in the network abstraction layer. In the H.264 / AVC video coding standard, the entire system framework is divided into two levels: video encoding level (VCL) and network abstraction level (NAL). The former is responsible for effectively representing the content of video data, while the latter is responsible for formatting data and providing header information to ensure that data is suitable for transmission on various channels and storage media. Therefore, we usually per frame data is a NAL unit (SPS and Except for PPS. In the actual H264 data frame, the front of the frame tend to have 00 00 00 00 00 00 00 00 00 00 00 01 separator, generally, the first frame data compared by the encoder is PPS and SPS, followed by I frame ...
As shown below:
2, how to determine the frame type (Is the image reference frame or I, P frame, etc.)?
The NALU type is our judged tool, from the official documentation as shown below:
We still follow the data corresponding to the above-mentioned code stream to layer analysis, and the next byte after 00 00 00 01 is the NALU type, and after converting it into binary data, the interpretation order is from left to right. , As follows: (1) The first bit is forbidden, the value is 1 indicates the syntax error (2) The second to 3 of the reference level (3) No. 4 ~ 8 is the NAL unit type
For example, there are 67,68 and 65 after 00000001 above
The binary code of 0x67 is: 0110 0111 4-8 is 00111, transfer to decimal 7, reference secondary: 7 correspondence sequence parameter set SPS
The binary code of 0x68 is: 0110 1000 4-8 is 01000, converted to decimal 8, refer to the second map: 8 correspondence image parameter set PPS
The binary code of 0x65 is: 0110 0101 4-8 is 00101, converted to decimal 5, refer to the second map: 5 corresponds to the sheet in the IDR image (I frame)
Be
Therefore, it is determined whether or not the algorithm is: (NALU type & 0001 1111) = 5 NAALU type & 31 = 5
For example, 0x65 & 31 = 5
//
With the development of wireless networks and smartphones, smart phones are increasingly close, entertainment, business applications, financial applications, and traffic travel, making people's life colorful and convenient. Also make it a part of people who are not replaced in people. Among them, multimedia is increasingly widely used due to their intuitiveness and real-time, and the video decoding and playback will also become a hot spot of research.
H.264
The standard technology is increasingly mature, using a unified VLC symbol code, high precision, multi-mode displacement estimate, based on 4 × 4-piece integer transformation, hierarchical encoding grammar, etc. These measures make the H.264 algorithm have high coding efficiency, and under the same reconstruction image quality, it is possible to save about 50% of H.263. Moreover, the H.264 code stream structure network is adaptable, and the error recovery capability is added. It is suitable for wireless networks that are limited by bandwidth and high errors.
This paper combines the decoding method in the FFMPEG open source code, using multi-threaded receiving data packets, multi-level buffer data, reception, and decoding parallel double thread operation, etc., to alleviate data blockage, decoding, decoding, decoding, decoding data due to large data volume and fast speed Error, sluggish video, delay, etc. The transmission speed of the H.264 video is fast and stable. Finally realize video transfer of the PC to the Android mobile phone, and the decoding playback on the Android phone.
This technique can be applied to video conferencing, video surveillance and other applications.
Be
First, the overall structure of the H.264 video transmission play system
H.264 video transmission playback system is divided into server-side and clients
Two parts, the server side is responsible for reading the H.264 video data, and packaged to the client in the RTP / RTCP format, and accepts feedback on the client, the transmission speed, etc. The Android phone client mainly completes the real-time code stream data from the server side, buffered, video data resolution, and then sends decoding, and finally display playback on the phone. The server side is implemented in C language, and the client mainly implements it in Java language.
Be
Second, key technology and its implementation
Based on
RTP protocol packaging and unpack
(
1) Single NAL package
H.264NALU unit is often
[NALU HEADER] [NALU HEADER] [NALU PAYLOAD] consists of three parts, where start code is used to mark the beginning of a NALU unit, must be "00000001" or "000001", pick up the start code, package other data to The RTP package can be.
(
2) Split packaging
due to
1500 bytes are the upper limit of the length of the IP datagram, remove 20 bytes of data newspaper, 1480 bytes are used to store UDP dibctions. So when the number of bytes in the frame exceeds this value, we must pack it. Moreover, the UDP is also overhead in the process of transmission, so the maximum number of bytes of the RTP package is 1400 bytes.
The package format that needs to be divided, first describing the format of the subsets:
The FU indicator is formatted:
+------------- +
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+ - + + + - + + + + - + - + + +
| F | NRI | TYPE |
+------------- +
FU indicates the type of byte
28, 29 denotes FU-A and Fu-B. The value of the NRI domain must be set according to the value of the NAL unit NRI to be divided.
The format of the fu head is as follows:
+------------- +
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+ - + + + - + + + + - + - + + +
| S | e | r | TYPE |
+------------- +
S: Start
(1bit) When set to 1, the start bit indicates the beginning of the slice NAL unit. The first fraction package is set to 1, and the other fractions set to 0.
E: End the position
(1bit), when set to 1, the end bit indicates the end of the slice NAL unit, ie the FU load is set to 1 when the final fragment is set, and at other times are set to 0.
R: Reserved
(1bit) must be set to 0.
TYPE:
5bit
(
3) Process analysis of packaging and unpacking:
Pack:
Detailed description when fragmentation:
1 first
The Fu Indicator for the Fu-a package is set up: f = nalu head in f, nri = nalu head in NRI, TYPE = 28 Fu Header: s = 1, E = 0, R = 0, Type = NALU head Type in
2 in the middle
FU-a package FU indeicator is set up: f = nalu head in f, nri = nalu head in NRI, Type = 28 Fu Header: s = 0, E = 0, R = 0, Type = NALU head Type in
3 tail
Fu Indicator for the Fu-a package is set up: f = nalu head in f, nri = nalu head in NRI, TYPE = 28 Fu Header: s = 0, E = 1, R = 0, TYPE = NALU head Type in.
Be
Unpacking:
Below we
The code implementation of the fragmentation of the fragmentation when the RTP is unpacking:
BYTE STARTBIT = (Byte) (RECBUF [13] & 0x80);
Byte endbit = (byte) (Recbuf [13] & 0x40);
1 If,
STARTBIT == - 128, this package is the first package of fragmentation.
NALBUF [4] = (BYTE) ((RECBUF [12] & 0xE0) + (RECBUF [13] & 0x1f); this sentence is used to reconstruct a combination
NAL unit type
2 if
(Startbit == 0) && (endbit == 0), this package is the middle part of the fragmentation.
3 if
Endbit == 64, this package is the tail of the slice.
When the classification is clear, the corresponding processing can be made, as analyzed in the figure.
2. Code stream management mechanism
(1) Reception of the code stream. In the case where the transmitting end code is sent very fast, since the receiving end must receive a stream, it is necessary to analyze, decode, this processing requires a longer process. If the receiving end sequentially executes this process, it will result in unable Receive the package of the sender, the packet, which brings a decoding error, and cannot play the video normally, and even the program is rushing. We have taken concurrent handling mechanism for this issue. One meaning of thread concurrent existence is to improve the speed running on a single processor. exist
In Java, we use the actuator (Executor) in the java.util.concurrent package to manage thread Thread objects. We created 20 threads, which is to submit 20 tasks to SingLethReadexecutor. These tasks will be ranked, and each task will run before the next task begins. Each task is in the order they are submitted, in the next task Before starting. This not only achieves fast reception but also ensures that the received package order is correct. After such processing, the reception and analysis decoding can be divided into two parts, we can temporarily store the received data in the buffer, and then then receive the next packet data, do not wait to analyze, decoding is completed Receive the next package of data. This greatly improves receiving efficiency while avoiding packet loss issues.
(2) Video data analysis and decoding. Due to the use of concurrent mechanisms, the received data is more than one pack, so how the received data should be processed reasonably, becoming our next difficult point. We need to be guaranteed to be the order of the packet, and only one pack can be handled each time, which involves a collaborative problem between a thread. We use consumer producers such thread collaboration model to do processing. We will put another buffer after the buffer from the buffer of storage data, and the notification decoder can perform data decoding from this buffer, then analyze the program of video data into the waiting. After the decoding is completed, the program notifies the analysis of the video data continues to perform video data analysis, and the decoding program is waiting. Two programs are alternately performed in execution and waiting.
(3) Multi-level buffer mechanism. Above we also mentioned several buffers, summarized as follows.
1 Saving of data after receiving, due to the constant real-time code stream of the server side, we can't process it immediately, so we must have a buffer.
2 Buffering between the receiving end and the processing end, because the network is unstable, the received data may sometimes be slow, which will directly cause the decoded instability and the discontinuous video playback, so set a buffer here. To a smooth, transitional role, this buffer is stored both to receive a large number of code streams, providing data for video data analysis, having a process of writing and reading, so we use the first queue
Queue container to make buffers.
3 The buffering between parsing and decoding procedure, because the amount of data in this process is not very large, and the speed of this acquisition data directly affects the speed of the decoding, so we have to use an efficient buffer to take The buffer effect at this time is due to
Stack is automatically assigned by the system, so the speed is faster, so we allocate an array on the stack to store it.
4 Decoding the buffer between playback, this buffer also except that the playback video is continuously stabilized, mainly to display the image, and can perform some processing, smooth, filtering, etc.
3. Realization of decoding and playback
H.264 decoding is transplanted
The H.264 decoded portion in FFMPEG is ANDROID, and depth deletion optimization. The interface part, file reception processing, and video display are made by Java, and the underlying video decoding portion uses C to do to meet speed requirements. H.264 Codestream Segmentation NAL (Accepting the recovery of video data) is done in the Java layer without being installed into C, because the data per selection will be limited, if the amount of data sent is large, the bottom layer may A few frames of video at a time, but the interface layer can only display one frame, resulting in a frame loss. If the amount of data per time is small, it will cause multiple underlying calls but did not perform substantial decoding, so although this is different, the speed is slow, but the comprehensive consideration is still putting data analysis work The Java layer is completed.
We use the video data after decoding
Bitmap
Displaying, the method of DRAW to SurfaceView is displayed on the mobile phone screen, because some mobile phones do not support RGB24 but almost all mobile phones support RGB565, so the decoding returns RGB565 data.
4.
Program process function architecture
The problem of slowing the data is slow, and the multi-thread concurrent mechanism is used to solve, facing a large number, and unstable packets, setting multi-stage buffering is set for the characteristics of each link. Make the video play more smooth and smooth. For the order of analysis and decoding, the idea of thread collaboration is used, using consumers, producer model, and ensure timing of video data. In addition, for the decoding portion, the platform transplantation is performed by the existing decoding method.
The division of the C layer and the Java layer and to achieve this complete function. In the case of a network condition, the Android mobile phone video playback is short, and the play is smooth and smooth. This article technology studies can use the various applications of video playback, have strong practical value.
Be
Wen / Nanjing University of Posts and Telecommunications Zhang Yongqi Gong Jianrong
Thank http://blog.sina.com.cn/s/blog_4ad7c2540101lep1.html
http://blog.csdn.net/jefry_xdz/Article/details/8461343 "
Be
Third, conclude
Complete design of this article and achieve it from
The H.264 video transfer and decoding playback function of the PC end to the Android phone. Detailed analysis of the technical points and difficulties in realization, analyze the RTP packaging, unpacking process, targeting
Our other product:
