Using Avatars when publishing streams

---

Instructions

Introduction

The ZEGO Avatar SDK supports exporting avatar textures and allows users to output their avatar renderings. Developers can make custom post-processing effects for the renderings and then use them for RTC stream publishing.

SDKs

• ZEGO Express SDK: A real-time audio and video SDK developed by ZEGOCLOUD, which provides basic real-time audio and video features, including live stream publishing and pulling and live co-hosting. It is referred to as Express SDK for short.
• ZEGO Avatar SDK: A virtual avatar SDK developed by ZEGOCLOUD, which allows users to customize their avatars by choosing the default avatars or creating their personalized avatars. It provides facial expression mirroring, speech simulation, and other features to enable real-time interactions with real persons. It is referred to as Avatar SDK for short.

Prerequisites

• Create a project in the ZEGOCLOUD Admin Console and get a valid AppID and AppSign for the project.
• Integrate the Express SDK into the project to implement basic real-time audio and video features. For more information, see Getting started - Integrate the SDK and Getting started - Implement a basic video call.
• Integrate the Avatar SDK into the project. For more information, see Getting started - Integrate the SDK.
• Contact ZEGOCLOUD technical support and submitted the Bundle ID of your project to enable related permissions.

Procedures

1. Initialize ZegoAvatarService

Refer to "1. Apply for authentication" and "2. Initialize ZegoAvatarService" in Create a virtual avatar.

2. Set data collection type for stream publishing

2.1 Initialize the Express SDK and log in to a room

Refer to "3.1 Initialize" (It is recommended that developers initialize the SDK when the app starts) and "3.2 Log in to a room" in Getting started - Implement a basic video call.

2.2 Set data collection type for stream publishing and enable custom video capture

1. Customize the video capture settings of the Express SDK by using ZegoCustomVideoCaptureConfig and call the enableCustomVideoCapture API to enable the custom video capture feature.
2. Call the setCustomVideoCaptureHandler API to set the custom video capture callback.
3. Call the setVideoConfig API to set video configurations.

// Custom video capture.
ZegoCustomVideoCaptureConfig *captureConfig = [[ZegoCustomVideoCaptureConfig alloc] init];
// Select video frame data of the CVPixelBuffer type.
captureConfig.bufferType = ZegoVideoBufferTypeCVPixelBuffer;
// Enable custom video capture.
[[ZegoExpressEngine sharedEngine] enableCustomVideoCapture:YES config:captureConfig channel:ZegoPublishChannelMain];

// Listen for events of starting and stopping custom capture.
[[ZegoExpressEngine sharedEngine] setCustomVideoCaptureHandler:self];

// Set video configurations and keep the configurations consistent with the avatar output size.
ZegoVideoConfig *videoConfig = [ZegoVideoConfig configWithPreset:ZegoVideoConfigPreset720P];
videoConfig.encodeResolution = CGSizeMake(300, 300);//Display the width and height.
[[ZegoExpressEngine sharedEngine] setVideoConfig:videoConfig];

3. Create a virtual avatar and start facial expression detection

1. After the ZegoAvatarService SDK is initialized, create a ZegoCharacterHelper object, introduce the appearance data, such as the face, cloth, and makeup, and set view parameters, such as the width, height, and position, to create a virtual avatar.
2. After the basic virtual avatar is created, call the startDetectExpression API, set the drive mode to ZegoExpressionDetectModeCamera, and use the front camera to detect facial expressions. Then, use the setExpression API of ZegoCharacterHelper to set facial expressions and drive facial expression changes of the virtual avatar.

- (void) initAvatar{
    // Basic resource path.
    NSString *assetBundlesPath = [[[NSBundle mainBundle] bundlePath] stringByAppendingString:@"/assets/human.bundle"];
    // Create a ZegoCharacterHelper class.
    _characterHelper = [[ZegoCharacterHelper alloc] init:assetBundlesPath];

    // Set the resource package address.
    NSString *packagesPath = [[[NSBundle mainBundle] bundlePath] stringByAppendingString:@"/assets/Packages"];
    [_characterHelper setExtendPackagesPath: packagesPath];
    // Use the default avatar. The male role is used as an example.
    [_characterHelper setDefaultAvatar: MODEL_ID_MALE];

    // Important: AvatarCapture will be created later. Do not display the avatar on the screen here, or you cannot obtain the avatar texture for stream publishing.
    // [_characterHelper setCharacterView:_avatarView];

    //Enable facial expression mirroring.
    [self startDetect];
}
// Enable facial expression detection.
- (void ) startDetect{
    __weak typeof(self) weakSelf = self;
    //Enable facial expression mirroring.
    [[_service getInteractEngine] startDetectExpression: ZegoExpressionDetectModeCamera callback:^(ZegoExpression *expression) {
        __strong typeof(self) he = weakSelf;
        [he->_characterHelper setExpression: expression];
    }]; 
}

4. Export avatar texture

After the basic avatar is created, use the startCaptureAvatar API of ZegoCharacterHelper to introduce the parameters of AvatarCaptureConfig, such as the width and height of the exported texture and start to export the metal texture.

// Start to export texture.
// Set the width (captureWidth) and height (captureHeight) of the returned avatar in the callback as required.
AvatarCaptureConfig* config = [[AvatarCaptureConfig alloc] initWithWidth:captureWidth height:captureHeight];
@weakify(self);//Solve self circular reference.
[self.characterHelper startCaptureAvatar:config callback:^(id<MTLTexture> srcTexture,int width,int height) {    
    // Metal texture length and width.
}];

(Optional) Custom post-processing

After you export the metal texture data by using startCaptureAvatar, you can register the onAvatarTextureAvailable callback of onAvatarCaptureCallback to listen for the result of exporting texture, including the length and width of the metal texture.

You can make custom post-processing effects for the texture data. For example, add a background image (the default avatar background image is transparent). Then, call the sendCustomVideoCapturePixelBuffer API to publish streams and send video frame data to the SDK.

To set the background color, you can refer to the sample source code obtained in Download.

// Perform the following operations after you obtain the avatar texture.
// Configure the metal texture of the avatar the same as that configured in Express.
AvatarCaptureConfig *avconfig = [[AvatarCaptureConfig alloc] initWithWidth:300 height:300];
__weak __typeof(self) weakSelf = self;
_isStop = NO;
// Enable avatar capture.
[self.characterHelper startCaptureAvatar:avconfig callback:^(id<MTLTexture> srcTexture,int width,int height){
    // The metal texture is obtained here.
    __strong __typeof(self) strongSelf = weakSelf;
    if(strongSelf.isStop){ //Stop stream publishing if avatar capture has stopped.
        return;
    }
    @autoreleasepool {

        // Demonstrate how to add background color for the avatar texture. Use the API of metal for metal textures.
        // The color is different from the background color of AvatarView. The texture is *transparent*.
        [[MetalTools sharedInstance] setupBgColorWithTexture:srcTexture bgColor:strongSelf.avatarBackgroundColor callback:^(id<MTLTexture>  _Nonnull newTexture) {
            //The format is BGRA. Format conversion is required.
            [strongSelf getPixelBufferFromBGRAMTLTexture:newTexture result:^(CVPixelBufferRef pixelBuffer){

                // Create a timestamp.
                CMTime time = CMTimeMakeWithSeconds([[NSDate date] timeIntervalSince1970], 1000);
                CMSampleTimingInfo timingInfo = { kCMTimeInvalid, time, time };
                CMVideoFormatDescriptionRef desc;
                CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, pixelBuffer, &desc);

                CMSampleBufferRef sampleBuffer;
                // Convert the format to pixelBuffer.
                CMSampleBufferCreateReadyWithImageBuffer(kCFAllocatorDefault, (CVImageBufferRef)pixelBuffer, desc, &timingInfo, &sampleBuffer);

                // Use the sendCustomVideoCapturePixelBuffer of the Express SDK for stream publishing.
                [[ZegoExpressEngine sharedEngine] sendCustomVideoCapturePixelBuffer:CMSampleBufferGetImageBuffer(sampleBuffer) timestamp:CMSampleBufferGetPresentationTimeStamp(sampleBuffer)];

                CFRelease(sampleBuffer);
                CFRelease(desc);
            }];

        }];
    }// endof @autoreleasepool
}];

5. Preview and publish streams

After the loginRoom API is called, you can call the startPublishingStream API to introduce the streamID (unique and generated based on your service) and publish the avatar texture video streams to ZEGOCLOUD. You can listen for the onPublisherStateUpdate callback to check whether the stream publishing is successful.

ZegoCanvas *Canvas = [ZegoCanvas canvasWithView:self.expressPreView];
Canvas.viewMode = ZegoViewModeAspectFill;
[[ZegoExpressEngine sharedEngine] startPreview:Canvas];

NSString *publishStreamID = self.publishStreamIDTextField.text;
//Start to publish streams.
[[ZegoExpressEngine sharedEngine] startPublishingStream:publishStreamID];

Features

1. Stop facial expression detection

Run the application in the backend or exit the current page. Call the stopDetectExpression API to stop facial expression detection.

// Stop facial expression detection.
[[[ZegoAvatarService sharedInstance] getInteractEngine] stopDetectExpression];

2. Stop exporting texture

Call the stopCaptureAvatar API to stop exporting the avatar texture.

// Stop exporting texture.
[self.helper stopCaptureAvatar]

3. Stop publishing streams, stop previewing, and log out of the room

Call the stopPublishingStream API to stop sending local audio and video streams to remote users.

If local preview is enabled, call the stopPreview API to stop the preview.

Call the logoutRoom API to log out of the room.

[[ZegoExpressEngine sharedEngine] stopPreview]; // Stop previewing.
[[ZegoExpressEngine sharedEngine] stopPublishingStream]; // Stop publishing streams.
[[ZegoExpressEngine sharedEngine] logoutRoom]; // Log out of the room.

4. Destroy engine

If users no longer use the audio and video features, call the destroyEngine API to destroy the engine and release the resources, including the microphone, camera, memory, and CPU.

[ZegoExpressEngine destroyEngine:nil];