The meaning of video coding
Large storage space for original video data, a 1080P 7 s video requires 817 MB
The original video data transmission occupies a large bandwidth, and it takes 11 minutes to transmit the above 7 s video with a bandwidth of 10 Mbps
After H.264 encoding and compression, the video size is only 708 k, and the 10 Mbps bandwidth only needs 500 ms, which can meet the needs of real-time transmission. Therefore, the original video collected from the video acquisition sensor must be video encoded.
Fundamental
So why can a huge original video be encoded into a very small video? What is the technology in this? Before talking about technology, we should first establish the concept of video that is continuous pictures.
The core idea is to remove redundant information:
Spatial redundancy: there is a strong correlation between adjacent pixels of a picture
Temporal redundancy: similar content between adjacent pictures in a video sequence
Coding redundancy: different pixel values have different probabilities
Visual redundancy: the human visual system is not sensitive to certain details
Knowledge redundancy: the structure of regularity can be obtained from prior knowledge and background knowledge
Video is essentially a series of pictures that are played continuously and quickly, so the easiest way to compress a video is to compress each frame of pictures. For example, the older MJPEG encoding is to compress each frame of pictures in the video. This encoding method There is only intra-frame coding, which uses spatial sample prediction to code. The image metaphor is to treat each frame as a picture, and use the JPEG encoding format to compress the picture. This kind of encoding only considers the compression of redundant information in a picture.
However, because of the time correlation between frames, some advanced encoders have been developed that can use inter-frame coding. Simply put, certain areas on the frame are selected through the search algorithm, and then the current frame is calculated It is a form of encoding with the vector difference between the front and rear reference frames. Through the following two consecutive frames in Figure 2, we can see that the skier is shifting forward, but in fact the snow scene is shifting backwards, and the P frame is referenced Frames (I or other P frames) can be encoded, the size after encoding is very small, and the compression ratio is very high.
Reference link about the frame http://mp.weixin.qq.com/s/ox6MsWx71b-GFsZihaOwww
Some students may be interested in how these two pictures came from. Here are two lines of FFmpeg commands to achieve. For more details on FFmpeg, please see the following chapters:
The first line generates a video with a moving vector
The second line outputs each frame as a picture
Use the command
ffmpeg -flags2 +export_mvs -i tutu.mp4 -vf codecview=mv=pf+bf+bb tutudebug2.mp4
ffmpeg -i tutudebug2.mp4'tutunormal-%03d.bmp'
In addition to spatial redundancy and temporal redundancy compression, there are mainly encoding compression and visual compression. The following is the main flow chart of an encoder:
Figure 3 and Figure 4 are two processes. Figure 3 is intra-frame coding, and Figure 4 is inter-frame coding. The main difference seen from the figure is that the first step is different. In fact, these two processes are also combined. Generally speaking, I frame and P frame use intra-frame coding and inter-frame coding respectively.
Encoder selection
I have sorted out the principle and basic process of the encoder. The encoder has experienced decades of development. It has evolved from only supporting intra-frame encoding to the new generation of encoders represented by H.265 and VP9 today. At present, some common encoders are analyzed, and we will take you to explore the world of encoders.
H.264
Introduction
The H.264/AVC project intends to create a video standard. Compared with the old standard, it can provide high-quality video at a lower bandwidth (in other words, only half the bandwidth of MPEG-2, H.263 or MPEG-4 Part 2 or less) without adding too much design complexity Makes it impossible to achieve or the cost of implementation is too high. Another purpose is to provide sufficient flexibility to be used in various applications, networks and systems, including high and low bandwidth, high and low video resolutions, broadcasting, DVD storage, RTP/IP networks, and ITU-T multimedia phones system.
H.264/AVC contains a series of new features, making it not only more efficient than previous codecs, but also can be used in applications in various network environments. This technical foundation makes H.264 become the main codec used by online video companies including YouTube, but using it is not a very easy task. In theory, using H.264 requires a lot of money. Patent fees.
Patent license
Like the first and second parts of MPEG-2 and the second part of MPEG-4, product manufacturers and service providers that use H.264/AVC need to pay patent license fees to patent holders. The main source of these patent licenses is a private organization called MPEG-LA LLC. This organization has nothing to do with the MPEG Standardization Organization, but this organization also manages the MPEG-2 Part One System, Part Two Video, and MPEG-4 Part One. Two-part video and other technology patent licenses.
Other patent licenses need to apply to another private organization called VIA Licensing, which also manages patent licenses for audio compression standards such as MPEG-2 AAC and MPEG-4 Audio.
Open source implementation of H.264
openh264 is an open source H.264 encoding program implemented by Cisco. Although H.264 requires a high patent fee, there is an annual limit on the patent fee. After Cisco pays the annual patent fee for OpenH264, OpenH264 is actually free Use it freely.
x264 is a video coding free software licensed under GPL. The main function of x264 is to perform H.264/MPEG-4 AVC video encoding, not as a decoder.
Excluding the cost issue for comparison:
The CPU usage of openh264 is much lower than that of x264
openh264 only supports baseline profile, x264 supports more profiles
HEVC/H.265
Introduction
High Efficiency Video Coding (HEVC) is a video compression standard (also called H.265), which is regarded as the successor of the ITU-T H.264/MPEG-4 AVC standard. In 2004, ISO/IEC Moving Picture Experts Group (MPEG) and ITU-T Video Coding Experts Group (VCEG) began to develop as ISO/IEC 23008-2 MPEG-H Part 2 or ITU-T H.265. The first version of the HEVC/H.265 video compression standard was accepted as the official standard of the International Telecommunication Union (ITU-T) on April 13, 2013. HEVC is considered not only to improve video quality, but also to achieve twice the compression rate of H.264/MPEG-4 AVC (equivalent to a 50% reduction in bit rate under the same picture quality), and can support 4K resolution and even ultra-high-definition TV (UHDTV), the highest resolution can reach 8192×4320 (8K resolution).
Patent license
HEVC requires all content manufacturers that use H.265 technology, including Apple, YouTube, Netflix, Facebook, and Amazon, to pay 0.5% of their content revenue as a technology usage fee. The entire streaming media market reaches about 100 billion U.S. dollars each year, and it continues to In the growth, the levy of 0.5% is definitely a huge fee. And they have not let go of equipment manufacturers, among which TV manufacturers need to pay 1.5 US dollars per unit and mobile device manufacturers 0.8 US dollars per unit in patent fees. They have not even let go of manufacturers such as Blu-ray device players, game consoles, and video recorders, which must pay $1.1 each.
Open source implementation of H.265/HEVC
libde265 HEVC is provided by struktur company under the open source license GNU Lesser General Public License (LGPL), and viewers can enjoy the highest quality images at slower internet speeds. Compared with previous decoders based on the H.264 standard, the libde265 HEVC decoder can bring your full HD content to up to twice the audience, or reduce the bandwidth required for streaming by 50%.
x265 is developed by MulticoreWare and is open sourced under the GPL agreement.
VP8
Introduction
VP8 is an open video compression format that was first developed by On2 Technologies and then released by Google. At the same time, Google also released the VP8 coded implementation library: libvpx, which was released in the form of BSD license terms, and subsequently added the right to use the patent. After some arguments, the authorization of VP8 was finally confirmed as an open source authorization.
Currently, the web browsers that support VP8 are Opera, Firefox and Chrome.
Patent license
In March 2013, Google reached an agreement with MPEG LA and 11 patent holders to allow Google to obtain VP8 and its previous VPx and other encodings that may be infringed on patents. At the same time, Google can also re-authorize related patents to VP8 users free of charge. , This agreement is also suitable for the next generation of VPx encoding. So far, MPEG LA has given up the establishment of the VP8 patent centralized licensing alliance, and VP8 users will be able to determine to use this code free of charge without worrying about possible patent infringement royalties.
Open source implementation of VP8
Libvpx is the only open source implementation of VP8. It was developed by On2 Technologies. After Google acquired it, it opened its source code. The license is very loose and can be used freely.
VP9
Introduction
The development of VP9 began in the third quarter of 2011. The goal is to reduce the file size by 50% compared to VP8 encoding under the same image quality. Another goal is to surpass HEVC encoding in encoding efficiency.
On December 13, 2012, the Chromium browser added support for VP9 encoding. Chrome browser started to support VP9 encoded video playback on February 21, 2013.
Google announced that it will complete the development of the VP9 code on June 17, 2013, when the Chrome browser will guide the VP9 code by default. On March 18, 2014, Mozilla added VP9 support to the Firefox browser.
On April 3, 2015, Google released libvpx1.4.0, which added support for 10-bit and 12-bit bit depth, 4:2:2 and 4:4:4 chroma sampling, and VP9 multi-core encoding/decoding.
Patent license
VP9 is an open format, royalty-free video encoding format.
Open source implementation of VP9
libvpx is the only open source implementation of VP9, developed and maintained by Google. Some of the codes are shared by VP8 and VP9, and the rest are the codec implementations of VP8 and VP9 respectively.
Comparison of VP9 and H.264 and HEVC
Comparison of HEVC and H.264 at different resolutions
Compared with H.264/MPEG-4, the average bit rate reduction of HEVC is:
It can be seen that the bit rate has dropped by more than 60%
HEVC (H.265) has a greater advantage in bit rate saving for VP9 and H.264, saving 48.3% and 75.8% respectively under the same PSNR
H.264 has a huge advantage in encoding time. Compared with VP9 and HEVC (H.265), HEVC is 6 times that of VP9, and VP9 is nearly 40 times that of H.264.
