- Documentation
- Video Call
- Upgrade using advanced features
- Distincitve features
- Customize the video and audio
- Pre-process the video
Pre-process the video
Introduction
Custom video pre-processing refers to processing the captured video with the AI Effects SDK for features such as face beautification, and stickers, this might be needed when the Video Call SDK does not meet your development requirements.
Compared to the custom video capture feature,
the custom video pre-processing does not require you to manage the device input sources, you only need to manipulate the raw data thrown out by the ZegoExpress-Video SDK
, and then send it back to the ZegoExpress-Video SDK
.
For more advanced features such as layer blending, we recommend you refer to the Customize how the video captures.
Prerequisites
Before you begin, make sure you complete the following steps:
- ZEGO Express SDK has been integrated into the project to implement basic real-time audio and video functions. For details, please refer to Quick start .
- A project has been created in ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - Project Information .
Implementation process
The following diagram shows the API call sequence of the custom video pre-processing:
Enable the custom video pre-processing feature
- Create a
ZegoCustomVideoProcessConfig
object, and set thebufferType
property for providing video frame data type to the SDK.
Currently, the SDK only supports the CVPixelBuffer
type of video data (when the value of the bufferType
is ZegoVideoBufferTypeCVPixelBuffer
). Setting other enumeration values will not take effect.
- To enable the custom video pre-processing, call the
enableCustomVideoProcessing
method before starting the local video preview and the stream publishing starts.
ZegoCustomVideoProcessConfig *processConfig = [[ZegoCustomVideoProcessConfig alloc] init];
// Select the [CVPixelBuffer] type of video frame data.
processConfig.bufferType = ZegoVideoBufferTypeCVPixelBuffer;
[[ZegoExpressEngine sharedEngine] enableCustomVideoProcessing:YES config:processConfig channel:ZegoPublishChannelMain];
Define a event handler object
Define a custom video pre-processing event handler object as
ViewController
, and conforms to theZegoCustomVideoProcessHandler
protocol.@interface ViewController () <ZegoEventHandler, ZegoCustomVideoProcessHandler> ... @end
Call the
setCustomVideoProcessHandler
method to listen for and handle the custom video pre-processing event callbacks.// Set [self] as the event callback object for custom video pre-processing. [[ZegoExpressEngine sharedEngine] setCustomVideoProcessHandler:self];
Get and process the video raw data
To get the video raw data (the data of the
ZegoVideoBufferTypeCVPixelBuffer
type), call theonCapturedUnprocessedCVPixelBuffer
method.After processed the raw data using the
ZegoEffects SDK
, call thesendCustomVideoProcessedCVPixelBuffer
method to send the processed data.
- (void)onCapturedUnprocessedCVPixelBuffer:(CVPixelBufferRef)buffer timestamp:(CMTime)timestamp channel:(ZegoPublishChannel)channel {
// Custom pre-processing: use the Effects SDK here.
[self.effects processImageBuffer:buffer];
// Send the processed buffer.
[[ZegoExpressEngine sharedEngine] sendCustomVideoProcessedCVPixelBuffer:output timestamp:timestamp channel:channel];
}