Abstract: An RTP-based audio and video synchronization method is designed for the H.323 video conferencing system. Under the premise of strict compliance with the RTP protocol, connect audio and video data through the same media channel, which reaches lip tone synchronization. Purpose. Experiments show that the method has a small performance of audio and video with a small impact on the image quality, and has a wide range of practicability.
In the H.323 video conferencing system, the audio and video data collected at the same time in the sender can be played simultaneously at the receiving end, and the lip tone is considered to be synchronized. The audio and video data collected by the terminal is definitely synchronized. To ensure simultaneous playback, it is necessary to ensure that the audio and video consumes the same time during the acquisition and playback processing. The characteristics of IP networks determine the time consumed by audio and video data transmission through different channels, and lip synchronization is a big problem in a video conferencing system. If the number of audio and video data playing time sampling is within [-80 ms, + 80ms], the user basically does not feel disagree, once [-160ms, + 160ms], the user can obviously feel that the middle part is critical scope.
1 Introduction
1.1 Article Arrangement
In Section 2, the drawback of the existing audio and video synchronization scheme is analyzed. The implementation process of this article design is described in detail in Section 3. Section 4 gives experimental data and analysis results. Section 5 gives a conclusion.
1.2 Basic introduction
In the H.323 video conferencing system, the cause of the audio and video disagreement phenomenon is outside the network environment, and there is a separate transmission of audio and video. Although H.323 recommends audio and video through different channels, the RTP [2, 3] protocol of actual transmission data and its underlying UDP protocol do not specify a pair of connections that can only transmit audio or video, through the same The channel transmission audio and video is completely possible, and this can minimize the audio and video caused by the network cause, this article gives this idea implementation, and verifies.
2 existing solutions
The most commonly used lip synchronization method can be divided into two categories from ideas:
I'm thinking about the RTP package to send to each RTP package to record their sampling time. The receiving end ensures simultaneous amount of data simultaneously played simultaneously by increasing delay, etc. This type of method requires a neutral third party reference clock, which requires the participation of the RTCP protocol of SR [2,3]. If these two conditions do not have, the synchronization has lost the basis.
Idea II. The lip is not synchronized in nature is caused by Separation and processing of the sound video in the H.323 video conferencing system. If a method is associated with the audio and video information, it can effectively avoid disabled phenomena. One implementation is that the audio is embedded in the video to the video, and the receiving end extracts audio data from the video, thereby reaching the purpose of the lip synchronization [4]. The program achieves more complex, and uses non-standard RTP implementations that will bring interoperability between different manufacturers H.323 products.
3 new audio and video synchronization method
The basic idea of this method is to collect, encode, package, unpack, and ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, The time decoded is basically fixed, and the four processes of transmission, network transmission, reception, and network abnormality are transmitted, network transmission, reception, and network abnormalities due to network environment differences. The processing time will process the network performance. Different have a large difference, thereby causing a different step of playing the audio and video. Therefore, the focus of lip synchronous processing is to ensure that the four processes of the audio video in the four processes in the four processes are guaranteed to transmit, network transmission, reception, and network exceptions, that is, the part of the synchronization to the group frame synchronization in Fig. 1.
Be
Figure 1 Lip synchronization implementation
The time difference caused by other processing processes, as long as the system is fixed to the audio after the system is stabilized, because the time spectacles spended by the audio processing is small than the video processing, the specific difference can be obtained more experimental statistics.
The RTP protocol regulates the payload type (PT) in each RTP package is unique, but if the audio video is transmitted through the same channel, and guarantees that the audio and video frame collected by the same time is transmitted sequentially, it can Guaranteed the synchronization of the audio and video in the transmission and complied with the RTP protocol. The amount of audio data is small, one RTP package can carry one frame, one of the video frames require multiple RTP packet bearer, the frame end flag uses the Mark field in the RTP header, which means that the current package is one frame. End package.
According to the above thoughts, the specific implementation process design is as follows:
(1) The transmitting end separately samples, group frames, and packages, and then put it in the respective buffer queues, waiting to send (2) data transmission module from send buffer, 1) from the audio buffer queue Take a package (one frame); 2) Take the data from the video buffer queue, each to determine if the RTP Baotou's Mark field is 1, if it is 1, indicating that the current video frame has been taken, turn 1), If the Mark field is 0, it means that the current video frame has not been completed, turn 2); (3) The audio and video data is sent to the network through the same channel; (4) The receiving end receives the data, according to the PT field according to the PT field in the header Video, put it in the respective reception buff queue to request a packet loss retrans, chamfered rearrangement and other network exception processing [5, 6], then enter the group frame buffer waiting decoder to remove the data, and enter the group frame buffer data Dictorial packages and heavy inclusions, occasional packets; (5) audio and video respective unpacking group frames, implementation process as shown in Figure 2:
Be
Figure 2 Frame synchronization implementation schematic diagram.
(6) The audio and video sequence data sequentially in the respective decoding buffer queues, and simults to synchronize after calibrating the local timestamp plus local time stamps plus video data during group frame. Technology area
Tektron supports Amazon (AWS) media service, providing quality assurance for end-to-end video
IMEC is about to shock the first short-wave infrared (SWIR) band hyperspectral imaging camera
4K super high-definition home theater projector brings HD experience, full of fun
Video display system design based on unified calculation architecture technology
Apple TV 4K dismantling report: familiar modular components
Our other product:
