Unity3D下实现Linux平台RTMP推流(以采集Unity窗体和声音为例)

技术背景

随着物联网等行业的崛起,越来越多的传统行业如虚拟仿真、航天工业、工业仿真、城市规划等,对Linux下的生态构建,有了更大的期望,Linux平台下,可选的直播推拉流解决方案相对Windows和移动端,非常少,基于Unity的Linux推送方案,更是几无参考。本文以Unity3d环境下Linux平台推送Unity窗体和Unity采集的音频,然后编码推送到RTMP服务器为例,大概说下实现过程。

技术实现

本文以采集Unity窗体数据为例,如果需要对接摄像头和屏幕亦可。简单来说,高效率的获取到原始的窗体Texture,拿到对应的RGB数据,传给现有的RTMP推送模块,音频的话,只要获取到AudioClip数据,实时读取然后传PCM数据。如需2路PCM数据混音,可以采集两路AudioClip数据,分别调用原生的Linux推送RTMP模块的PCM数据投递接口,编码打包推送即可,无图无真相:

下图以Linux平台下,unity3d环境采集窗体(为方便测试,加了个实时时间显示)和unity声音为例,整体延迟可达毫秒级。

音频的话,可以循环采集播放、也可以播放完毕后,加实时静音。

AudioClip数据采集,如需audiosource数据播放完毕后,填充静音帧,可调用FillPcmMuteData()接口。

    /// <summary>/// 获取AudioClip数据/// </summary>private void PostUnityAudioClipData(){//重新初始化TimerManager.UnRegister(this);audio_clip_info_ = new AudioClipInfo();audio_clip_info_.audio_source_ = GameObject.Find("Canvas/Panel/AudioSource").GetComponent<AudioSource>();if (audio_clip_info_.audio_source_ == null){Debug.LogError("audio source is null..");return;}if (audio_clip_info_.audio_source_.clip == null){Debug.LogError("audio clip is null..");return;}audio_clip_info_.audio_clip_ = audio_clip_info_.audio_source_.clip;audio_clip_info_.audio_clip_offset_ = 0;audio_clip_info_.audio_clip_length_ = audio_clip_info_.audio_clip_.samples * audio_clip_info_.audio_clip_.channels;pcm_mute_data_ = FillPcmMuteData(audio_clip_info_.audio_clip_);TimerManager.Register(this, 0.019999f, (PostPCMData), false);}

采集后的数据,定时调用原生接口,发送到原生接口:

var sample_length = sizeof(float) * pcm_sample.Length;pcm_data.data_ = Marshal.AllocHGlobal(sample_length);
Marshal.Copy(pcm_sample, 0, pcm_data.data_, pcm_sample.Length);
pcm_data.size_ = (uint)sample_length;publisher_wrapper_.OnPostAudioPCMFloatData(pcm_data.data_,pcm_data.size_,pcm_time_stamp_,pcm_data.sample_rate_,pcm_data.channels_,pcm_data.per_channel_sample_number_);Marshal.FreeHGlobal(pcm_data.data_);
pcm_data.data_ = IntPtr.Zero;
pcm_data = null;
pcm_time_stamp_ += 10;  //时间戳自增10毫秒

视频窗体数据采集:

        if (video_push_type_ != (uint)NTSmartPublisherDefine.NT_PB_E_VIDEO_OPTION.NT_PB_E_VIDEO_OPTION_LAYER)return;if (texture_ == null || video_width_ != Screen.width || video_height_ != Screen.height){Debug.Log("OnPostRender screen changed++ scr_width: " + Screen.width + " scr_height: " + Screen.height);if (screen_image_ != IntPtr.Zero){Marshal.FreeHGlobal(screen_image_);screen_image_ = IntPtr.Zero;}if (texture_ != null){UnityEngine.Object.Destroy(texture_);texture_ = null;}video_width_ = Screen.width;video_height_ = Screen.height;texture_ = new Texture2D(video_width_, video_height_, TextureFormat.BGRA32, false);screen_image_ = Marshal.AllocHGlobal(video_width_ * 4 * video_height_);Debug.Log("OnPostRender screen changed--");return;}

从Texture拿到数据,然后调用OnPostRGBAData()数据传递到原生接口即可。

原生接口模块初始化:

        /** nt_publisher_wrapper.cs* Github:https://github.com/daniulive/SmarterStreaming** InitSDK主要完成原生模块的初始化工作 */private bool InitSDK(){if (!is_pusher_sdk_init_){// 设置日志路径(请确保目录存在)Debug.LogError("NT_SL_SetPath++");String log_path = "./";NTSmartLog.NT_SL_SetPath(log_path);UInt32 isInited = NTSmartPublisherSDK.NT_PB_Init(0, IntPtr.Zero);if (isInited != 0){Debug.Log("调用NT_PB_Init失败..");return false;}is_pusher_sdk_init_ = true;}return true;}

RTMP推送模块基础参数配置:

        private void SetCommonOptionToPublisherSDK(){if (!IsPublisherHandleAvailable()){Debug.Log("SetCommonOptionToPublisherSDK, publisher handle with null..");return;}NTSmartPublisherSDK.NT_PB_ClearLayersConfig(publisher_handle_, 0,0, IntPtr.Zero);if (video_option_ == (uint)NTSmartPublisherDefine.NT_PB_E_VIDEO_OPTION.NT_PB_E_VIDEO_OPTION_LAYER){// 第0层填充RGBA矩形, 目的是保证帧率, 颜色就填充全黑int red = 0;int green = 0;int blue = 0;int alpha = 255;NT_PB_RGBARectangleLayerConfig rgba_layer_c0 = new NT_PB_RGBARectangleLayerConfig();rgba_layer_c0.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_RGBA_RECTANGLE;rgba_layer_c0.base_.index_ = 0;rgba_layer_c0.base_.enable_ = 1;rgba_layer_c0.base_.region_.x_ = 0;rgba_layer_c0.base_.region_.y_ = 0;rgba_layer_c0.base_.region_.width_ = video_width_;rgba_layer_c0.base_.region_.height_ = video_height_;rgba_layer_c0.base_.offset_ = Marshal.OffsetOf(rgba_layer_c0.GetType(), "base_").ToInt32();rgba_layer_c0.base_.cb_size_ = (uint)Marshal.SizeOf(rgba_layer_c0);rgba_layer_c0.red_ = System.BitConverter.GetBytes(red)[0];rgba_layer_c0.green_ = System.BitConverter.GetBytes(green)[0];rgba_layer_c0.blue_ = System.BitConverter.GetBytes(blue)[0];rgba_layer_c0.alpha_ = System.BitConverter.GetBytes(alpha)[0];IntPtr rgba_conf = Marshal.AllocHGlobal(Marshal.SizeOf(rgba_layer_c0));Marshal.StructureToPtr(rgba_layer_c0, rgba_conf, true);UInt32 rgba_r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,rgba_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_RGBA_RECTANGLE,0, IntPtr.Zero);Marshal.FreeHGlobal(rgba_conf);NT_PB_ExternalVideoFrameLayerConfig external_layer_c1 = new NT_PB_ExternalVideoFrameLayerConfig();external_layer_c1.base_.type_ = (Int32)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_EXTERNAL_VIDEO_FRAME;external_layer_c1.base_.index_ = 1;external_layer_c1.base_.enable_ = 1;external_layer_c1.base_.region_.x_ = 0;external_layer_c1.base_.region_.y_ = 0;external_layer_c1.base_.region_.width_ = video_width_;external_layer_c1.base_.region_.height_ = video_height_;external_layer_c1.base_.offset_ = Marshal.OffsetOf(external_layer_c1.GetType(), "base_").ToInt32();external_layer_c1.base_.cb_size_ = (uint)Marshal.SizeOf(external_layer_c1);IntPtr external_layer_conf = Marshal.AllocHGlobal(Marshal.SizeOf(external_layer_c1));Marshal.StructureToPtr(external_layer_c1, external_layer_conf, true);UInt32 external_r = NTSmartPublisherSDK.NT_PB_AddLayerConfig(publisher_handle_, 0,external_layer_conf, (int)NTSmartPublisherDefine.NT_PB_E_LAYER_TYPE.NT_PB_E_LAYER_TYPE_EXTERNAL_VIDEO_FRAME,0, IntPtr.Zero);Marshal.FreeHGlobal(external_layer_conf);}else if (video_option_ == (uint)NTSmartPublisherDefine.NT_PB_E_VIDEO_OPTION.NT_PB_E_VIDEO_OPTION_CAMERA){CameraInfo camera = cameras_[cur_sel_camera_index_];NT_PB_VideoCaptureCapability cap = camera.capabilities_[cur_sel_camera_resolutions_index_];SetVideoCaptureDeviceBaseParameter(camera.id_.ToString(), cap.width_, cap.height_);}SetFrameRate((uint)video_fps_);Int32 type = 0;   //软编码Int32 encoder_id = 1;UInt32 codec_id = (UInt32)NTCommonMediaDefine.NT_MEDIA_CODEC_ID.NT_MEDIA_CODEC_ID_H264;Int32 param1 = 0;SetVideoEncoder(type, encoder_id, codec_id, param1);SetVideoQuality(CalVideoQuality(video_width_, video_height_, is_h264_encoder_));SetVideoBitRate(CalBitRate(video_fps_, video_width_, video_height_));SetVideoMaxBitRate((CalMaxKBitRate(video_fps_, video_width_, video_height_, false)));SetVideoKeyFrameInterval((key_frame_interval_));if (is_h264_encoder_){SetVideoEncoderProfile(1);}SetVideoEncoderSpeed(CalVideoEncoderSpeed(video_width_, video_height_, is_h264_encoder_));// 音频相关设置SetAuidoInputDeviceId(0);SetPublisherAudioCodecType(1);SetPublisherMute(is_mute_);SetEchoCancellation(0, 0);SetNoiseSuppression(0);SetAGC(0);SetVAD(0);SetInputAudioVolume(Convert.ToSingle(audio_input_volume_));}

开始和停止预览:

        public bool StartPreview(){if(CheckPublisherHandleAvailable() == false)return false;video_preview_image_callback_ = new NT_PB_SDKVideoPreviewImageCallBack(SDKVideoPreviewImageCallBack);NTSmartPublisherSDK.NT_PB_SetVideoPreviewImageCallBack(publisher_handle_, (int)NTSmartPublisherDefine.NT_PB_E_IMAGE_FORMAT.NT_PB_E_IMAGE_FORMAT_RGB32, IntPtr.Zero, video_preview_image_callback_);if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_StartPreview(publisher_handle_, 0, IntPtr.Zero)){if (0 == publisher_handle_count_){NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);publisher_handle_ = IntPtr.Zero;}return false;}publisher_handle_count_++;is_previewing_ = true;return true;}public void StopPreview(){if (is_previewing_ == false) return;is_previewing_ = false;publisher_handle_count_--;NTSmartPublisherSDK.NT_PB_StopPreview(publisher_handle_);if (0 == publisher_handle_count_){NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);publisher_handle_ = IntPtr.Zero;}}

预览数据回调:

        //预览数据回调public void SDKVideoPreviewImageCallBack(IntPtr handle, IntPtr user_data, IntPtr image){NT_PB_Image pb_image = (NT_PB_Image)Marshal.PtrToStructure(image, typeof(NT_PB_Image));NT_VideoFrame pVideoFrame = new NT_VideoFrame();pVideoFrame.width_ = pb_image.width_;pVideoFrame.height_ = pb_image.height_;pVideoFrame.stride_ = pb_image.stride_[0];Int32 argb_size = pb_image.stride_[0] * pb_image.height_;pVideoFrame.plane_data_ = new byte[argb_size];if (argb_size > 0){Marshal.Copy(pb_image.plane_[0],pVideoFrame.plane_data_,0, argb_size);}{cur_image_ = pVideoFrame;}}        

开始推送和停止推送:

        public bool StartPublisher(String url){if (CheckPublisherHandleAvailable() == false) return false;if (publisher_handle_ == IntPtr.Zero){return false;}if (!String.IsNullOrEmpty(url)){NTSmartPublisherSDK.NT_PB_SetURL(publisher_handle_, url, IntPtr.Zero);}if (NTBaseCodeDefine.NT_ERC_OK != NTSmartPublisherSDK.NT_PB_StartPublisher(publisher_handle_, IntPtr.Zero)){if (0 == publisher_handle_count_){NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);publisher_handle_ = IntPtr.Zero;}is_publishing_ = false;return false;}publisher_handle_count_++;is_publishing_ = true;return true;}public void StopPublisher(){if (is_publishing_ == false) return;publisher_handle_count_--;NTSmartPublisherSDK.NT_PB_StopPublisher(publisher_handle_);if (0 == publisher_handle_count_){NTSmartPublisherSDK.NT_PB_Close(publisher_handle_);publisher_handle_ = IntPtr.Zero;}is_publishing_ = false;}

对应Unity层调用:

    public void btn_publish_Click(){if (publisher_wrapper_.IsPublishing()){return;}String url = rtmp_pusher_url.text;if (url.Length < 8){publisher_wrapper_.Close();Debug.Log("请输入RTMP推送地址");return;}publisher_wrapper_.SetVideoPushType(video_push_type_);publisher_wrapper_.SetAudioPushType(audio_push_type_);if (!publisher_wrapper_.StartPublisher(url)){Debug.LogError("调用StartPublisher失败..");return;}startPublishBtn.interactable = false;stopPublishBtn.interactable = !startPublishBtn.interactable;videoOptionSel.interactable = false;audioOptionSel.interactable = false;if (audio_push_type_ == (uint)NTSmartPublisherDefine.NT_PB_E_AUDIO_OPTION.NT_PB_E_AUDIO_OPTION_EXTERNAL_PCM_DATA|| audio_push_type_ == (uint)NTSmartPublisherDefine.NT_PB_E_AUDIO_OPTION.NT_PB_E_AUDIO_OPTION_TWO_EXTERNAL_PCM_MIXER){PostUnityAudioClipData();}}public void btn_stop_publish_Click(){if (!publisher_wrapper_.IsPublishing()){return;}StopAudioSource();if (texture_ != null){UnityEngine.Object.Destroy(texture_);texture_ = null;}if (screen_image_ != IntPtr.Zero){Marshal.FreeHGlobal(screen_image_);screen_image_ = IntPtr.Zero;}publisher_wrapper_.StopPublisher();videoOptionSel.interactable = true;audioOptionSel.interactable = true;startPublishBtn.interactable = true;stopPublishBtn.interactable = !startPublishBtn.interactable;}

总结

以上是Unity环境下,以采集Unity窗体和unity声音为例,对接Linux平台RTMP推送,整体延迟也可以达到毫秒级,基本满足常用场景了,感兴趣的开发者可酌情参考。

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.mzph.cn/news/552996.shtml

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

Android平台实现VR头显Unity下音视频数据RTMP推送

背景 随着技术发展的日新月异&#xff0c;虚拟现实产业已经从过去的探索期&#xff0c;自2020年起&#xff0c;慢慢过渡到高速发展期&#xff0c;随着5G时代的到来&#xff0c;大带宽高可靠低延迟网络环境&#xff0c;为虚拟现实产业提供了很好的网络保障&#xff0c;虚拟现实…

C++11特性之std:call_once介绍

std:call_once是C11引入的新特性&#xff0c;如需使用&#xff0c;只需要#include <mutex>即可&#xff0c;简单来说std:call_once的作用&#xff0c;确保函数或代码片段在多线程环境下&#xff0c;只需要执行一次&#xff0c;常用的场景如Init()操作或一些系统参数的获取…

2022年了,该学C++还是Java?

最近好多朋友私信我&#xff0c;C好不好学&#xff1f;学C好还是Java好&#xff1f; 我的回答是&#xff1a;C不好学&#xff0c;但你觉得C不好学的话&#xff0c;Java也不好学。因为C难是难在语言本身&#xff0c;java难是难在各种框架和库。 C学习进阶比较陡, 对新手不友好&…

Android平台音视频RTMP推送|GB28181对接之动态水印设计

技术背景 随着移动单兵、智能车载、智慧安防、智能家居、工业仿真、GB28281技术对接等行业的发展&#xff0c;现场已经不再限于采集到视频数据编码打包发送或对接到流媒体服务端&#xff0c;大多场景对视频水印的要求越来越高&#xff0c;从之前的固定位置静态文字水印、png水…

探究C++11智能指针之std::unique_ptr

背景 谈起C&#xff0c;它被公认为最难学的编程语言之一&#xff0c;不仅语法知识点广泛&#xff0c;细节内容之多&#xff0c;学习难度和学习周期也长&#xff0c;导致好多新入行的开发者对C“敬而远之”&#xff0c;甚至“从入门到放弃”。自C11开始&#xff0c;好多C程序员…

C++17新特性之try_emplace与insert_or_assign

由于std::map中&#xff0c;元素的key是唯一的&#xff0c;我们经常遇到这样的场景&#xff0c;向map中插入元素时&#xff0c;先检测map指定的key是否存在&#xff0c;不存在时才做插入操作&#xff0c;如果存在&#xff0c;直接取出来使用&#xff0c;或者key不存在时&#x…

Unity环境下实现Camera高帧率RTMP推送

Unity下RTMP直播背景方面不再赘述&#xff0c;今天主要讨论的是&#xff0c;Unity环境下&#xff0c;如何实现Camera高帧率RTMP推送&#xff0c;这里提到的高帧率&#xff0c;不再局限于常规环境下的30帧&#xff0c;以VR头显为例&#xff0c;更高的帧率&#xff08;比如50帧&a…

如何在Android平台GB28181接入终端实现语音广播和语音对讲

技术背景 在之前的blog&#xff0c;我们以Android平台国标接入终端为例&#xff0c;分别介绍了一些常规的功能&#xff0c;比如REGISTER、CATALOG、INVITE、Keepalive、SUBSCRIBE、NOTIFY等常规操作&#xff0c;今天主要介绍下语音广播和语音对讲这部分。 GB28181平台广播和对…

Android GB28181设备接入端语音广播和语音对讲技术实现探究

上篇文章提到Android端GB28181接入端的语音广播和语音对讲的实现&#xff0c;从spec角度大概介绍了下流程和简单的接口设计&#xff0c;好多开发者私信我&#xff0c;希望展开说一下。其实这块难度不大&#xff0c;只是广播和对讲涉及到双向实现&#xff0c;如果之前没有相关的…

Android native层实现MediaCodec编码H264/HEVC

Android平台在上层实现mediacodec的编码&#xff0c;资料泛滥&#xff0c;已经不再是难事&#xff0c;今天给大家介绍下&#xff0c;如何在Android native层实现MediaCodec编码H264/HEVC&#xff0c;网上千篇一律的接口说明&#xff0c;这里不再赘述&#xff0c;本文主要介绍下…

GB/T 28181联网系统通信协议结构和技术实现

技术回顾 在本文开头&#xff0c;我们先一起回顾下GB/T28181联网系统通信协议结构&#xff1a; 联网系统在进行视音频传输及控制时应建立两个传输通道&#xff1a;会话通道和媒体流通道。 会话通道用于在设备之间建立会话并传输系统控制命令&#xff1b;媒体流通道用于传输视…

Android平台GB28181设备接入端对接编码前后音视频源类型浅析

前言 今天主要对Android平台GB28181设备接入模块支持的接入数据类型&#xff0c;做个简单的汇总&#xff1a; 编码前数据&#xff08;目前支持的有YV12/NV21/NV12/I420/RGB24/RGBA32/RGB565等数据类型&#xff09;&#xff0c;其中&#xff0c;Android平台前后摄像头数据&…

C++学习之-析构函数必须为虚函数吗?

今天讨论个比较有意思的话题&#xff1a;析构函数是不是必须要为虚函数&#xff1f; 先说答案&#xff1a; 析构函数可以是虚函数&#xff0c;也可以不是虚函数。 再说原因&#xff1a; 析构函数为虚函数的情况&#xff1a;继承 当父类指针释放子类对象时&#xff0c;如果…

如何实现RTMP或RTSP播放端回调YUV/RGB数据?

今天某乎收到个问题推荐&#xff0c;如何实现RTSP回调YUV数据&#xff0c;用于二次处理&#xff1f; 正好前些年我们做RTSP和RTMP直播播放的时候&#xff0c;实现过相关的需求&#xff0c;本文就以Android为例&#xff0c;大概说说具体实现吧。 先说回调yuv或rgb这块意义吧&a…

GB28181控制、传输流程和协议接口之注册|注销和技术实现

注册和注销基本要求 SIP客户端、网关、SIP设备、联网系统等 SIP代理(SIP UA)使用IETFRFC3261中定义的方法 15 GB/T28181—2016Register进行注册和注销。 注册和注销时应进行认证&#xff0c;认证方式应支持数字摘要认证方式&#xff0c;高安全级别的宜支持数字证书的认证方式&…

GB28181状态信息报送解读及Android端国标设备接入技术实现

今天主要聊聊GB/T28181状态信息报送这块&#xff0c;先回顾下协议规范相关细节&#xff0c;然后再针对代码实现&#xff0c;做个简单的说明。 状态消息报送基本要求 当源设备(包括网关、SIP设备、SIP客户端或联网系统)发现工作异常时,应立即向本 SIP监控域 的SIP服务器发送状…

GB28181设备接入端如何实现校时?

在探讨这个问题之前&#xff0c;我们先看看GB/T28181-2016官方文档怎么说的&#xff0c;9.10.1章节校时基本要求提到&#xff1a; 联网内设备支持基于SIP方式或 NTP方式的网络校时功能&#xff0c;标准时间为北京时间。 SIP方式校时见本节具体描述&#xff1b;NTP(见IETFRFC2…

如何在Unity下采集音视频实现轻量级RTSP服务(类似于IPC)

好多开发者在做虚拟仿真、VR教育等场景的时候&#xff0c;遇到个问题&#xff0c;想把头显里面的画面在内网环境下低延迟的同步出来&#xff0c;又不想单独部署流媒体服务器。为此&#xff0c;我们在Unity下&#xff0c;添加了轻量级RTSP服务模块&#xff0c;通过头显端启动个轻…

【Datawhale 大模型基础】第十一章 环境影响

第十一章 环境影响 This blog is based on datawhale files and a paper. The initial consideration revolves around the potential positive or negative direct impact on the environment. Other transformative technological advancements, like the metaverse, are li…

Android GB28181接入端实时位置订阅和上报之-如何获取当前经纬度

我们在做Android平台GB28181的时候&#xff0c;其中实时位置(MobilePosition)订阅和上报这块&#xff0c;涉及到实时经纬度的获取&#xff0c;特别是执法记录、车载系统的那个等场景&#xff0c;几乎就是标配。 今天主要是分享一段实时获取位置的代码&#xff1a; /** Camera…