Customize how the video renders

---

Introduction

When the ZEGOCLOUD SDK's built-in video rendering module cannot meet your app's requirement, the SDK allows you to customize the video rendering process. With custom video rendering enabled, the SDK will send out the video frame data captured locally or received from a remote stream so that you can use the data to perform the rendering by yourself.

Listed below are some scenarios where enabling custom video rendering is recommended:

1. Your app uses a cross-platform framework that requires a complex UI hierarchy (e.g., QT) or game engines like Unity, Unreal Engine and Cocos.
2. Your app requires special processing of the video frame data for rendering.

Prerequisite

• ZEGO Express SDK has been integrated into the project to implement basic real-time audio and video functions. For details, please refer to Quick start .
• A project has been created in ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information .

Implementation process

The process of custom video rendering is as follows:

1. Create a ZegoExpressEngine instance.
2. Enable the feature of custom video rendering.
3. Set up the event handler for custom video rendering callbacks, and implement the callback handler methods according to your app's requirements.
4. Log in to a room and start previewing the locally captured video or playing a remote stream. The SDK will then start sending out video frame data of the local or remote stream through the corresponding callback for custom video rendering.

Refer to the API call sequence diagram below to implement custom video rendering in your project:

Enable the Feature of Custom Video Rendering

First, create a ZegoCustomVideoRenderConfig object and configure its attributes according to your actual rendering requirements.

• Set the bufferType (ZegoVideoBufferType) attribute to specify the video frame data type for custom rendering.

• Set the frameFormatSeries (ZegoVideoFrameFormatSeries) attribute to specify the color space (RGB or YUV) of the video frame data for custom rendering.

• Set the enableEngineRender attribute to determine whether the SDK's built-in renderer is also enabled. If it is set to true, the SDK's built-in renderer will render the stream to the View passed to startPreview or startPlayingStream. If it is set to false, the SDK's built-in renderer will not render the stream.

Then, call enableCustomVideoRender to enable custom video rendering.

ZegoCustomVideoRenderConfig renderConfig;
renderConfig.bufferType = ZEGO_VIDEO_BUFFER_TYPE_RAWDATA;
renderConfig.frameFormatSeries = ZEGO_VIDEO_FRAME_FORMAT_SERIES_RGB;
renderConfig.enableEngineRender = false;

engine->enableCustomVideoRender(true, &engineConfig);

Set up the Custom Video Rendering Callback Handler

Call setCustomVideoRenderHandler to set up an event handler (IZegoCustomVideoRenderHandler) to listen for and handle the callbacks related to custom video rendering.

Implement the handler method for the callback onCapturedVideoFrameRawData. The SDK sends out the video frame data captured locally through this callback for the custom rendering of the local preview.

Implement the handler method for the callback onRemoteVideoFrameRawData. The SDK sends out the video frame data received from a remote stream through this callback for the custom rendering of the remote stream playback.

// Create a custom video rendering callback handler from the class IZegoCustomVideoRenderHandler
class MyCustomVideoRenderHandler: public IZegoCustomVideoRenderHandler{

    virtual void onCapturedVideoFrameRawData(unsigned char** /*data*/, unsigned int* /*dataLength*/, ZegoVideoFrameParam /*param*/, ZegoVideoFlipMode /*flipMode*/, ZegoPublishChannel /*channel*/) {
      // Use the data received from this callback to perform the custom video rendering for local preview. 
      ...;

    }

    virtual void onRemoteVideoFrameRawData(unsigned char** /*data*/, unsigned int* /*dataLength*/, ZegoVideoFrameParam /*param*/, const std::string& /*streamID*/) {
      // Use the data received from this callback to perform the custom video rendering for remote stream playback. 
      ...;

    }
}

// Set up the custom video rendering callback handler for the engine.
engine->setCustomVideoRenderHandler(std::make_shared<MyCustomVideoRenderHandler>());

The flipMode (ZegoVideoFlipMode) parameter of the callback onCapturedVideoFrameRawData indicates whether you need to flip the video so that the preview matches the video mirroring effect of the mirrorMode (ZegoVideoMirrorMode) setting of setVideoMirrorMode.

The param (ZegoVideoFrameParam) parameter of both callbacks above describes the attributes of the video frames, which are defined as below:

struct ZegoVideoFrameParam
{
    /** The pixel format (e.g., I420, NV12, NV21, RGBA32, etc.) */
    ZegoVideoFrameFormat format;

    /** The stride values of each color plane (e.g., for RGBA format, only the value of strides[0] needs to be used; while for I420, the values of strides[0,1,2] need to be used).*/
    int strides[4];

    /** The width of the video image */
    int width;

    /** The height of the video image */
    int height;

    /** The rotation angle of the video */
    int rotation;
};

The relationship between the stride and the image width is illustrated in the picture below:

Trigger the Custom Video Rendering Callback

After logging in to a room, you can start the preview of the local stream or the playback of a remote stream to trigger the custom video rendering callback.

• Custom Video Rendering for Local Preview

  With custom video rendering enabled, calling startPreview will trigger the callback onCapturedVideoFrameRawData through which the SDK sends out the video frame data captured locally. You can then use the video data obtained from this callback to render the local preview by yourself.

  If the SDK's built-in renderer is enabled, you need to pass in a View as the canvas parameter when calling startPreview; otherwise, set the canvas parameter to nullptr.

  After the preview is started, you can then start publishing the stream.

• Custom Video Rendering for Remote Stream Playback

  With custom video rendering enabled, calling startPlayingStream will trigger the callback onRemoteVideoFrameRawData through which the SDK sends out the video frame data received from a remote stream. You can then use the video data obtained from this callback to render the remote stream by yourself.

  If the SDK's built-in renderer is enabled, you need to pass in a View as the canvas parameter when calling startPlayingStream; otherwise, set the canvas parameter to nullptr.

FAQ

Q1：What's the difference between custom video rendering for local preview and custom video rendering for remote stream playback?

Answer: Custom video rendering for local preview is performed on the stream publishing end to give the streamer a preview of the local stream with special rendering effects. Custom video rendering for remote stream playback is performed on the stream receiving end to provide the viewers with special rendering effects.

Q2：If the enableEngineRender attribute of ZegoCustomVideoRenderConfig is set to false, what value should be passed to the canvas parameter when calling startPreview and startPlayingStream?

Answer: If enableEngineRender is set to false, the engine's built-in renderer will not do any video rendering, so the canvas parameter can be set to nullptr.

Q3：When custom video rendering for local preview is enabled, will the processed video data for preview also get published out?

Answer: No. The custom rendering process only affects the local preview and does not affect the video data that are streamed out.

Q4：What is the width and height of the video frames for local preview custom rendering?

Answer: The width and height of the video frames for local preview custom rendering can be obtained from the param (ZegoVideoFrameParam) parameter of the custom video rendering callback, which should be the same as the video capture definition specified to call setVideoConfig or the engine's default video capture definition.

Q5：What is the pixel format of the video frame data for custom video rendering? Is the YUV format supported?

Answer: Both YUV and RGB are supported. As mentioned in step 4.1 above, you need to set the frameFormatSeries (ZegoVideoFrameFormatSeries) attribute to specify the color space (RGB or YUV) of the video frame data when calling enableCustomVideoRender to enable custom video rendering. The specific pixel format of the video frame data can be obtained from the format (ZegoVideoFrameFormat) attribute of the param (ZegoVideoFrameParam) parameter of the custom video rendering callbacks.

The supported pixel formats defined in ZegoVideoFrameFormat are listed below for your reference.

Q6：What is the frequency of the callbacks for custom video rendering? Is there anything I should pay attention to?

Answer: In general, the frequency of the callback for local preview custom rendering is the same as the frame rate set for stream publishing. But it also changes according to the frame rate set for traffic control (if such settings are enabled). The frequency of the callback for remote stream playback custom rendering changes according to the actual frame rate of the video data being received. For example, it may change due to any frame rate change on the stream publishing end for traffic control or any network stuttering on the stream receiving end.

Q7：How to get the data of the first frame for custom video rendering?

Answer: The data received from the first custom video rendering callback is the data for the first frame.

Q8：Is it possible to enable custom video rendering for local preview only and still let the ZEGO SDK handle the rendering for remote stream playback?

Answer: Yes. If the enableEngineRender attribute of ZegoCustomVideoRenderConfig is set to true, the SDK's built-in renderer will also render the video data to the view specified in the canvas parameter of startPreview and startPlayingStream while sending out the video data for custom rendering.

Q9：Custom video rendering is enabled, but no video data callback is received for the local preview rendering. How to solve the problem?

Answer: Before you start publishing the stream, you need to call startPreviewto trigger the custom rendering callback.

Q10：For local preview custom rendering, why is horizontal mirroring not done by default for the video frame data captured from the front camera?

Answer: For custom video rendering, you need to implement video mirroring by yourself. You can use the flipMode ([ZegoVideoFlipMode|_blank]@-ZegoVideoFlipMode)) parameter of the custom rendering callback to determine whether mirroring is required.