Lyrics synchronization implementation process-Android

---

1 Function realization flowchart

The content of the most widely used lyrics file LRC file is as follows.

 [00:02.37] ZEGO KTV
 [00:03.12]
 [00:03.80] Lyrics: ZEGO
 [00:04.55] Composing: ZEGO
 [00:05.18] Singing: ZEGO
 [00:06.43] Produced by: Shenzhen ZEGO Technology

This format file is small, easy to parse, and the accuracy of the lyrics is controlled to one line.

When the lyric file exists for the audience, chorus, and vocalist, you only need to know the current music progress timestamp at this time to achieve lyrics synchronization.

Instant SDK supports injecting non-media information into the media stream. Lyric information and audio and video information are transmitted in the same media channel, so that synchronization of lyrics and audio and video can be guaranteed. The progress of the lyrics is sent by the vocal end. After the audience and chorus end receive the timestamp, the key lyrics lines are highlighted according to the timestamp. To achieve the effect of synchronous display of lyrics.

Please refer to the specific implementation flowchart:

It is sufficient to internally agree on the communication format of the lyrics synchronization information between the lead singer, the audience and the chorus.

2 Receive lyrics time stamp data

The instant SDK pulls the stream and decodes to extract the Media Enhanced Supplementary Information (SEI), which will be called back to the application developer through onPlayerRecvSEI. The chorus and the audience need to pay attention to this callback interface.

2.1 Set callback, receive SEI

ZEGO SDK provides related interfaces for setting callbacks and receiving SEI.

/**
* Set callbacks to receive secondary media information
*/
mEngine.setEventHander(new IZegoEventHandler() {
    @Override
    public void onPlayerRecvSEI(String streamID, byte[] data) {
// Process SEI
    }
});

Among them, when the client receives the SEI, the SDK will call it.

2.2 Implement a callback function to handle media enhancement supplementary information (SEI)

SEI will be passed in as the input parameter of onPlayerRecvSEI. Developers need to process SEI in this function according to business requirements.

The data format sent by the lead singer to the audience and chorus is a time stamp. The sample code for handling secondary media information in KTV App is as follows:

/**
 * Set callback, receive SEI
 */
mEngine.setZegoEventHandler(new IZegoEventHandler() {
    @Override
    public void onPlayerRecvSEI(String streamID, byte[] data) {
        if (data.length == 0) {
            return;
        }

        int timeStamp = (data[0] & 0xff) | ((data[1] & 0xff) << 8) |
        (data[2] & 0xff) << 16) | (data[3] & 0xff) << 24);
        // Get the time stamp data here timeStamp
        // After getting timeStamp, display lyrics information according to the value of the timestamp
    }
});

When you get the timeStamp, you can parse the lyric data and display the lyric data of timeStamp.

3 Lead singer sends lyrics information

3.1 Send media enhancement supplementary information (SEI)

The Instant SDK provides related interfaces for sending SEI. In the KTV scene, the lyrics information is injected into the media stream as SEI, and sent to the streaming end along with the audio and video data. After the audience, chorus, and other streaming terminals receive the secondary media information, they get the application progress timestamp.

After the stream is successfully pushed, the following interfaces can be used to send SEI.

ZegoExpressEngine
    /**
     * Send media enhancement supplementary information
     *
     * This interface can send stream media enhancement supplementary information to synchronize some other additional information at the same time that the developer pushes and transmits audio and video stream data.
     * Generally, for scenes such as synchronizing music lyrics or precise video layout, you can choose to send SEI.
     * After the pusher sends the SEI, the puller can obtain the SEI content by monitoring the callback of [onPlayerRecvSEI].
     * Since the SEI information follows the video frame or audio frame, there may be frame loss due to network problems, so the SEI information may also be lost. To solve this situation, you should send it several times within the limited frequency.
     * Limit frequency: Do not exceed 30 times per second.
     * The SEI data length is limited to 4096 bytes.
     *
     * @param data SEI content
     */
    public void sendSEI(byte[] data);

3.2 How to send information using sending SEI

The SEI will be passed into the SDK as a parameter of data, and the developer will call this function and pass in the SEI according to business requirements.

The sample code for sending SEI in the Demo Demo is as follows:

int timeStamp = 0;// Get the playback progress timestamp
byte[] seiData = new byte[4];
seiData[0] = (byte) (timeStamp & 0xff); // 8 bits of the lower (right)
seiData[1] = (byte) ((timeStamp >> 8) & 0xff); //The second 8 bit
seiData[2] = (byte) ((timeStamp >> 16) & 0xff); //The third 8 bit
seiData[3] = (byte) ((timeStamp >> 24) & 0xff); //The fourth 8-bit bit

mEngine.sendSEI(seiData);

timeStamp is the progress timestamp of the current playback. You need to use the function of the player SDK MediaPlayer to obtain the progress timestamp of the accompaniment playback.

For a detailed description of the media secondary information function, please refer to the document: Video Advanced-Supplementary Media Enhancement Information (SEI)

3.3 Use MediaPlayer player to get the current playing timestamp data

Know how to send the secondary media information, then use the function provided by the ZEGO SDK MediaPlayer to get the timestamp

The sample code is as follows:

ZegoMediaPlayer player = mEngine.createMediaPlayer();
// Mix the audio played by the media player into the push stream
player.enableAux(true);
// Get the time stamp of the playing accompaniment progress through this function
int timeStamp = player.getCurrentDuration();

After getting the timeStamp, the vocalist needs to make a timer loop to get the timestamp and send it out using SDK SEI.

Take it in a loop every second, until the song is finished playing, so that the audience and chorus lyrics display reaches a synchronized state.