"Let's first understand what links video transmission needs to go through on the Internet. It must first capture audio and video using a microphone and camera. Then, the original data must be compressed (encoded) into the codec, broadcast through the Internet connection (using the transmission protocol), sent to a server-side solution (usually CDN or a cloud based cluster, such as red5 Pro), and then decompressed (decoded) for users to watch the video.
At present, a considerable number of codecs are being used, including VP8 / 9, H.264 (AVC), h.265 (hevc) and AV1. Today's article will focus on VP8 / VP9. We classify VP8 and VP9 into one category because they are similar in terms of licensing, and VP9 is an evolution of VP8.
Although this article focuses on VP9 and h.265, the most important question is what kind of encoder is best. Finally, we will explain why H.264 is a more effective choice for low latency live streaming.
VP9
VP9 codec is a free open source video coding standard developed by Google. It is a follow-up to VP8 and was originally used to compress UHD content on YouTube because it improves the coding efficiency of its previous generation. The original VPX coder came from on2 technology, which was acquired by Google in 2010. Google then the codec source.
H.265
H. 265 codec, or efficient video coding (hevc), is jointly developed by video coding expert group (VCEG) and moving picture expert group (MPEG). It was approved as the official follow-up product of H.264 (also known as advanced video coding (AVC)) in April 2013. It improves the compression efficiency of H.264 and reduces the size of video by about 50%.
H.264
As mentioned above, H.264 or AVC is the most widely used video editing decoder at present. As of September 2019, 91% of video industry developers use it. Like h.265, H.264 was also developed by the moving picture expert group (MPEG) as an improvement of the previous standard, aiming to effectively transmit compressed high-quality video over the Internet.
H. 264 is protected by a number of patents and authorized by mpeg-la organization. However, Cisco system provided the public with a widely used free open source encoder and openh264 decoder in 2013. In other words, Cisco pays for the patent licenses we all use. This in turn created the widespread adoption of H.264 codecs. Openh264 appears in all web browsers.
We have introduced the codec above. Let's compare the differences between them. We listed six key factors to evaluate each codec.
Coding quality
VP9 and h.265 are not very different in this category. No matter which codec you use, the video looks good. However, when the bit rate is high, the performance of h.265 is slightly better than VP9, and vice versa.
In order to judge the image quality, we can use SSIM (structure standard index measurement), as shown in the figure below. When broadcasting a stream on the Internet, the process of compressing and expanding (encoding and decoding) the visual data contained in the stream may cause slight distortions because the decoder extrapolates the data to display it. Therefore, SSIM essentially measures the accuracy of the transmitted image after encoding and decoding.
Figure 1: quality / bit rate graphs comparing libvpx (VP9), x264 (H.264), and x265 (hevc)（ Image source: https://blogs.gnome.org/rbultje/2015/09/28/vp9-encodingdecoding-performance-vs-hevch-264/ ）
However, it is a little different from H.264.
VP9 and h.265 can improve compression in part by using larger macroblocks. Macroblocks are image or video processing units that contain image pixels to be displayed. H. 264 use 16 × 16 macroblocks, while pv9 and h.265 use 64 × 64 macroblocks. After a series of calculations called "intra prediction directions", these macroblocks are reconstructed into the same original image, but the details in non critical areas are slightly reduced. This enables VP9 and h.265 to improve efficiency because less detailed areas (such as the sky or the blurred background of the image) will not be decomposed into smaller units. The missing details in these areas do not significantly reduce the overall quality of the image because important parts are rendered in more detail. It should also be noted that as the bit rate increases, the quality difference between AVC (H.264) and the other two codecs will become smaller.
Image source: https://www.macxdvd.com/mac-dvd-video-converter-how-to/h265-vs-h264.htm
H. 264 will produce poor images, especially at low bit rates. When comparing images running at the same bit rate, VP9 and h.265 are both more detailed and clearer than images generated with H.264. In other words, in order to produce VP9 or h.265 images of the same quality, H.264 needs to run at a higher bit rate. However, although the difference in quality can be detected, it is not necessarily a direct problem. To measure this more objectively, we can look at SSIM, which shows that the results of H.264 are very close to VP9 and h.265. Therefore, although H.264 may not be as good as H.264 in image quality, this difference is not enough to overcome the big tradeoff described in detail in the next section.
We should also point out that other factors, such as improved sub-pixel interpolation and motion vector reference selection (motion estimation), also improve the image quality. This is because they help predict what the next frame in the movie will look like. These are quite complex concepts that are worth discussing in their own articles, so let's stop here.
Winner: Draw
Encoding Time
In order to achieve higher compression ratio, VP9 and h.265 need to perform more processing. All this extra processing means that they take longer to encode video. This will hurt your delay because all the extra processing time will delay the playback of the video. Among other factors, latency is important to ensure that your real-time video stream provides an interactive experience.
Figure 2: compare libvpx (VP9), x264 (AVC) and x265 (hevc) pictures by taking encoding time as a factor for bit rate improvement: https://blogs.gnome.org/rbultje/2015/09/28/vp9-encodingdecoding-performance-vs-hevch-264/
So what exactly does the chart above mean?
This figure shows the encoding time in seconds on the horizontal axis. The vertical axis shows the improvement of bit rate, which sets the combination and of SSIM and bit rate to x265@veryslow The reference points are compared. The reference point is why x264 does not exceed 0%.
What does this picture tell us?
VP9 and h.265 (as advertised) are 50% better than H.264. But they are also 10 to 20 times slower. If you follow the blue line of x264 (AVC), you will see that it is always lower than the other two lines for most bit rate benchmarks. Moreover, the green (h.265) and orange (VP9) lines intersect H.264 very early in their curves. This means that the number of seconds per frame rate will begin to increase sharply and really reduce your stream performance. Therefore, although VP9 and h.265 show better compression rate, it costs very high coding time, which will greatly increase the delay. This study by Waterloo University provides a more in-depth analysis of coding time and codec comparison（ reference material: https://ece.uwaterloo.ca/ ~w238liu/2019comparative/paper/iciar2019cc.pdf）
Winner: H.264
CPU consumption
As mentioned in the previous section, both VP9 and h.265 must run more compression algorithms than H.264, which will increase their CPU utilization. Even in the case of complete optimization, real-time streaming media is a CPU intensive process, so improving the already high utilization will be a problem. However, there are some things that can alleviate this: hardware support. Dedicated chipsets will reduce CPU consumption.
At present, h.265 enjoys more hardware support, including windows10 (downloadable or through inter Kaby lake or newer processor), apple (IOS 11) and Android (Android 5.0) devices. Although most mobile devices support VP9, most other systems do not. Without direct hardware support, VP9 coding process will limit CPU, consume a lot of resources, shorten battery life, and may increase latency.
In the next section, we will introduce that H.264 enjoys extensive support and does not consume CPU like VP9 or h.265.
Winners: H.264, followed by h.265.
Adoption and Browser Implementation
In order to use codec, hardware support or software encoder is required. H. The adoption rate of 265 is very low, which is largely due to patent licensing. H. 265 there are four related patent pools: heve advance, MPEG La, velos media and technolor. This makes it more expensive and hinders its wide application, which limits it to specific hardware encoders and mobile chipsets. Only edge, Internet Explorer, and safari support h.265 hardware encoding. Even then, devices running browsers still need to support h.265 hardware coding. Even if h.265 is supported in properly implemented browsers, webrtc often does not work properly. Without the support of webrtc, it will be difficult to realize real-time delay.
VP9 is tax-free and open source, which clears the way for its wider adoption. This feature is supported by major browsers chrome, Firefox, edge and operating systems windows10, Android 5.0, IOS 14 and MacOS bigsur. Because webrtc supports VP9, it can also work directly in the browser. There are also rumors that Safari browser support is coming.
Although H.264 has a patent related to it, as we mentioned earlier, Cisco opened source the implementation of H.264 in 2013 and released it in the form of free binary download. This is a great impetus to the wide application of H.264. Therefore, H.264 is supported by all browsers, laptops and mobile devices.
Winner: the gap between H.264 and VP9 has been narrowed.
Bandwidth savings
The biggest benefit of improving compression and reducing file size is that it consumes less bandwidth when broadcasting. This means that slower users can still watch higher quality video streams.
So, which codec can produce better compression efficiency to create a smaller video?
According to a Netfix test, the performance of h.265 is about 20% higher than that of VP9. Although other tests produced different results, they all concluded that h.265 created a smaller file size. According to the objective metric used, h.265 saves 0.6% to 38.2% bit rate compared with VP9.
However, while it is useful to consume less bandwidth, there are other factors to consider. The average upload speed of global fixed broadband connection is 42.63 Mbps, which means that although H.264 requires higher connection speed, 4K streaming media can be supported in most places. Although the average upload speed of mobile devices is much lower, at 10.93 Mbps, they can still support 1080p streams.
This chart from boxcast shows that the global average connection speed can certainly meet the upload speed requirements of all resolution levels. Note: we cannot find a chart comparing all three codecs, but VP9 should be between H.264 and h.265.
Image source: https://www.boxcast.com/blog/hevc-h.265-vs.-h.264-avc-whats-the-difference
In addition, there are some ways to configure your streaming application to meet users in slower countries. You can do this by adding ABR and transcoding support. ABR (adaptive bit rate) will modify the bit rate to provide the best experience. Transcoding divides the broadcast into multiple qualities so that clients can request the best quality based on the available bandwidth.
You might think, "if the mobile device is stuck in 2 or
Our other product:
