作为GB28181安卓客户端,实时视音频点播是必须支持的功能,对于历史视音频回放功能,不支持的话可以从设备上拷贝录像文件再播放,但有些场景没法拷贝,安卓支持回放还是需要的。
历史视音频的回放和实时视音频点播信令上很相似,音视频数据都是通过RTP传输,信令回放要处理SIP INFO消息,解析MANSRTSP协议,实现快进、慢放、暂停、停止、位置拖动等远程控制命令。
GB28181文档详细定义了历史视音频的回放的信令流程, 对于安卓设备端的实现, 需要实现媒体流发送者相关的信令:
1:SIP服务器向安卓设备发送Invite请求,请求中携带SDP信息,SDP中的s字段为“Playback”代表历史回放,u字段代表回放通道ID和回放类型,t字段代表回放时间段,增加y字段描述 SSRC 值,f字段描述媒体参数。
2:安卓设备收到SIP服务器的Invite请求后,回复200OK响应(回复200 OK等最终响应前,也可先回复一个临时响应,比如180 Ringing等),携带SDP消息体, SDP中描述了安卓设备发送媒体流的IP、端口、媒体格式、SSRC字段等内容。
3:SIP服务器收到安卓设备返回的200OK响应后,向安卓设备发送ACK请求,请求中不携带消息体,完成与安卓设备的Invite会话建立过程。
4:安卓设备按Invite SDP中给出的IP地址和端口等信息,发送音视频RTP包(推荐PS RTP包)到媒体服务器。
5:在回放过程中,播放端通过向SIP服务器发送会话内Info+MANSRTSP消息(SIP服务器再转发给安卓设备端)进行回放控制,包括视频的暂停、播放、快放、慢放、随机拖放播放等操作。
6:安卓设备端在文件回放结束后发送会话内Message消息,通知SIP服务器回放已结束(会话内消息请参考RFC3261-12.2 Requests within a Dialog)。
7:SIP服务器收到媒体通知消息后做相应的处理,之后SIP服务器向安卓端发送BYE消息。
8:安卓设备收到BYE消息后回复200OK响应,会话断开,释放相关资源。
为方便快速上手代码实现,下面给出信令的具体实例:
安卓设备收到的INVITE SDP:
v=0
o=64010000041310000137 0 0 IN IP4 192.168.0.193
s=Playback
u=64010000041310000137:0
c=IN IP4 192.168.0.193
t=1698218951 1698219270
m=video 20072 RTP/AVP 96a=recvonly
a=rtpmap:96 PS/90000
y=1900000005
s=Playback代表历史回放,SSRC是:1900000005(SSRC第1位为历史或实时媒体流的标识位,0为实时,1为历史).
安卓设备回复200 OK携带的SDP:
v=0
o=64010000041310000137 0 0 IN IP4 192.168.0.151
s=MyAndroidPlaybackTest
c=IN IP4 192.168.0.151
t=0 0
m=video 36010 RTP/AVP 96
a=rtpmap:96 PS/90000
a=sendonly
y=1900000005
回放过程中通过SIP INFO消息+MANSRTSP协议将正常播放速度变为4倍快进播放:
PLAY RTSP/1.0
CSeq: 110379
Scale: 4.000000
回放过程中使用SIP INFO消息+MANSRTSP协议把播放位置改为60.27秒的位置(考虑到安卓一般使用微秒或纳秒单位,转换到微秒是:60270000):
PLAY RTSP/1.0
CSeq: 110392
Range:npt=60.27-
文件回放结束后安卓设备端发送会话内Message消息:
<?xml version="1.0" encoding="GB2312"?>
<Notify>
<CmdType>MediaStatus</CmdType>
<SN>1738772385</SN>
<DeviceID>64010000041310000137</DeviceID>
<NotifyType>121</NotifyType>
</Notify>
通知事件类型是"121", 表示历史媒体文件发送结束。
相关实现代码:
/*
* Copyright (C) 1130758427@qq.com. All rights reserved.
*//**
* 部分信令接口
*/package com.gb.ntsignalling;public interface GBSIPAgent {void addPlaybackListener(GBSIPAgentPlaybackListener playbackListener);void removePlaybackListener(GBSIPAgentPlaybackListener playbackListener);/**响应Invite Playback 200 OK*/boolean respondPlaybackInviteOK(long id, String deviceId, String startTime, String stopTime, MediaSessionDescription localMediaDescription);/**响应Invite Playback 其他状态码*/boolean respondPlaybackInvite(int statusCode, long id, String deviceId);/** 媒体流发送者在回放结束后发Message消息通知SIP服务器回放文件已发送完成* notifyType 必须是"121"*/boolean notifyPlaybackMediaStatus(long id, String deviceId, String notifyType);/**终止Playback会话*/void terminatePlayback(long id, String deviceId, boolean isSendBYE);/**终止所有Playback会话*/void terminateAllPlaybacks(boolean isSendBYE);
}/**
* 信令Playback Listener
*/
package com.gb.ntsignalling;public interface GBSIPAgentPlaybackListener {/**收到s=Playback的历史回放Invite*/void ntsOnInvitePlayback(long id, String deviceId, SessionDescription sessionDescription);/**发送Playback invite response 异常*/void ntsOnPlaybackInviteResponseException(long id, String deviceId, int statusCode, String errorInfo);/** 收到CANCEL Playback INVITE请求*/void ntsOnCancelPlayback(long id, String deviceId);/** 收到Ack*/void ntsOnAckPlayback(long id, String deviceId);/** 播放命令*/void ntsOnPlaybackMANSRTSPPlayCommand(long id, String deviceId);/** 暂停命令*/void ntsOnPlaybackMANSRTSPPauseCommand(long id, String deviceId);/** 快进/慢进命令*/void ntsOnPlaybackMANSRTSPScaleCommand(long id, String deviceId, double scale);/** 随机拖动命令*/void ntsOnPlaybackMANSRTSPSeekCommand(long id, String deviceId, double position_sec);/** 停止命令*/void ntsOnPlaybackMANSRTSPTeardownCommand(long id, String deviceId);/** 收到Bye*/void ntsOnByePlayback(long id, String deviceId);/** 不是在收到BYE Message情况下, 终止Playback*/void ntsOnTerminatePlayback(long id, String deviceId);/** Playback会话对应的对话终止, 一般不会触发这个回调,目前只有在响应了200K, 但在64*T1时间后还没收到ACK,才可能会出发收到这个, 请做相关清理处理*/void ntsOnPlaybackDialogTerminated(long id, String deviceId);
}/**
* 部分JNI接口, rtp ps 打包发送等代码C++实现
*/public class SmartPublisherJniV2 {/*** Open publisher(启动推送实例)** @param ctx: get by this.getApplicationContext()* * @param audio_opt:* if 0: 不推送音频* if 1: 推送编码前音频(PCM)* if 2: 推送编码后音频(aac/pcma/pcmu/speex).* * @param video_opt:* if 0: 不推送视频* if 1: 推送编码前视频(NV12/I420/RGBA8888等格式)* if 2: 推送编码后视频(AVC/HEVC)* if 3: 层叠加模式** <pre>This function must be called firstly.</pre>** @return the handle of publisher instance*/public native long SmartPublisherOpen(Object ctx, int audio_opt, int video_opt, int width, int height);/*** 设置流类型* @param type: 0:表示 live 流, 1:表示 on-demand 流, SDK默认为0(live流)* 注意: 流类型设置当前仅对GB28181媒体流有效* @return {0} if successful*/public native int SetStreamType(long handle, int type);/*** 投递视频 on demand包, 当前只用于GB28181推送, 注意ByteBuffer对象必须是DirectBuffer** @param codec_id: 编码id, 当前支持H264和H265, 1:H264, 2:H265** @param packet: 视频数据, 包格式请参考H264/H265 Annex B Byte stream format, 例如:* 0x00000001 nal_unit 0x00000001 ...* H264 IDR: 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....* H265 IDR: 0x00000001 vps 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....** @param offset: 偏移量* @param size: packet size* @param pts_us: 时间戳, 单位微秒* @param is_pts_discontinuity: 是否时间戳间断,0:未间断,1:间断* @param is_key: 是否是关键帧, 0:非关键帧, 1:关键帧* @param codec_specific_data: 可选参数,可传null, 对于H264关键帧包, 如果packet不含sps和pps, 可传0x00000001 sps 0x00000001 pps* ,对于H265关键帧包, 如果packet不含vps,sps和pps, 可传0x00000001 vps 0x00000001 sps 0x00000001 pps* @param codec_specific_data_size: codec_specific_data size* @param width: 图像宽, 可传0* @param height: 图像高, 可传0** @return {0} if successful*/public native int PostVideoOnDemandPacketByteBuffer(long handle, int codec_id,ByteBuffer packet, int offset, int size, long pts_us, int is_pts_discontinuity, int is_key,byte[] codec_specific_data, int codec_specific_data_size,int width, int height);/*** 投递音频on demand包, 当前只用于GB28181推送, 注意ByteBuffer对象必须是DirectBuffer** @param codec_id: 编码id, 当前支持PCMA和AAC, 65536:PCMA, 65538:AAC* @param packet: 音频数据* @param offset:packet偏移量* @param size: packet size* @param pts_us: 时间戳, 单位微秒* @param is_pts_discontinuity: 是否时间戳间断,0:未间断,1:间断* @param codec_specific_data: 如果是AAC的话,需要传 Audio Specific Configuration* @param codec_specific_data_size: codec_specific_data size* @param sample_rate: 采样率* @param channels: 通道数** @return {0} if successful*/public native int PostAudioOnDemandPacketByteBuffer(long handle, int codec_id,ByteBuffer packet, int offset, int size, long pts_us, int is_pts_discontinuity,byte[] codec_specific_data, int codec_specific_data_size,int sample_rate, int channels);/*** on demand source完成seek后, 请调用* @return {0} if successful*/public native int OnSeekProcessed(long handle);/*** 启动 GB28181 媒体流** @return {0} if successful*/public native int StartGB28181MediaStream(long handle);/*** 停止 GB28181 媒体流** @return {0} if successful*/public native int StopGB28181MediaStream(long handle);/*** 关闭推送实例,结束时必须调用close接口释放资源** @return {0} if successful*/public native int SmartPublisherClose(long handle);}/**
* Listener部分实现代码
*/public class PlaybackListenerImpl implements com.gb.ntsignalling.GBSIPAgentPlaybackListener {/**收到s=Playback的文件下载Invite*/@Overridepublic void ntsOnInvitePlayback(long id, String deviceId, SessionDescription sdp) {if (!post_task(new PlaybackListenerImpl.OnInviteTask(this.context_, this.is_exit_, this.senders_map_, deviceId, sdp, id))) {Log.e(TAG, "ntsOnInvitePlayback post_task failed, " + RecordSender.make_print_tuple(id, deviceId, sdp.getTime().getStartTime(), sdp.getTime().getStopTime()));// 这里不发488, 等待事务超时也可以的GBSIPAgent agent = this.context_.get_agent();if (agent != null)agent.respondPlaybackInvite(488, id, deviceId);}}/**发送Playback invite response 异常*/@Overridepublic void ntsOnPlaybackInviteResponseException(long id, String deviceId, int statusCode, String errorInfo) {Log.i(TAG, "ntsOnPlaybackInviteResponseException, status_code:" + statusCode + ", "+ RecordSender.make_print_tuple(id, deviceId) + ", error_info:" + errorInfo);RecordSender sender = senders_map_.remove(id);if (null == sender)return;PlaybackListenerImpl.StopDisposeTask task = new PlaybackListenerImpl.StopDisposeTask(sender);if (!post_task(task))task.run();}/** 收到CANCEL Playback INVITE请求*/@Overridepublic void ntsOnCancelPlayback(long id, String deviceId) {Log.i(TAG, "ntsOnCancelPlayback, " + RecordSender.make_print_tuple(id, deviceId));RecordSender sender = senders_map_.remove(id);if (null == sender)return;PlaybackListenerImpl.StopDisposeTask task = new PlaybackListenerImpl.StopDisposeTask(sender);if (!post_task(task))task.run();}/** 收到Ack*/@Overridepublic void ntsOnAckPlayback(long id, String deviceId) {Log.i(TAG, "ntsOnAckPlayback, "+ RecordSender.make_print_tuple(id, deviceId));RecordSender sender = senders_map_.get(id);if (null == sender) {Log.e(TAG, "ntsOnAckPlayback get sender is null, " + RecordSender.make_print_tuple(id, deviceId));GBSIPAgent agent = this.context_.get_agent();if (agent != null)agent.terminatePlayback(id, deviceId, false);return;}PlaybackListenerImpl.StartTask task = new PlaybackListenerImpl.StartTask(sender, this.senders_map_);if (!post_task(task))task.run();}/** 收到Bye*/@Overridepublic void ntsOnByePlayback(long id, String deviceId) {Log.i(TAG, "ntsOnByePlayback, "+ RecordSender.make_print_tuple(id, deviceId));RecordSender sender = this.senders_map_.remove(id);if (null == sender)return;PlaybackListenerImpl.StopDisposeTask task = new PlaybackListenerImpl.StopDisposeTask(sender);if (!post_task(task))task.run();}/** 播放命令*/@Overridepublic void ntsOnPlaybackMANSRTSPPlayCommand(long id, String deviceId) {RecordSender sender = this.senders_map_.get(id);if (null == sender) {Log.e(TAG, "ntsOnPlaybackMANSRTSPPlayCommand can not get sender " + RecordSender.make_print_tuple(id, deviceId));return;}sender.post_play_command();Log.i(TAG, "ntsOnPlaybackMANSRTSPPlayCommand " + RecordSender.make_print_tuple(id, deviceId));}/** 暂停命令*/@Overridepublic void ntsOnPlaybackMANSRTSPPauseCommand(long id, String deviceId) {RecordSender sender = this.senders_map_.get(id);if (null == sender) {Log.e(TAG, "ntsOnPlaybackMANSRTSPPauseCommand can not get sender " + RecordSender.make_print_tuple(id, deviceId));return;}sender.post_pause_command();Log.i(TAG, "ntsOnPlaybackMANSRTSPPauseCommand " + RecordSender.make_print_tuple(id, deviceId));}/** 快进/慢进命令*/@Overridepublic void ntsOnPlaybackMANSRTSPScaleCommand(long id, String deviceId, double scale) {if (scale < 0.01) {Log.e(TAG, "ntsOnPlaybackMANSRTSPScaleCommand invalid scale:" + scale + " " + RecordSender.make_print_tuple(id, deviceId));return;}RecordSender sender = this.senders_map_.get(id);if (null == sender) {Log.e(TAG, "ntsOnPlaybackMANSRTSPScaleCommand can not get sender, scale:" + scale + " " + RecordSender.make_print_tuple(id, deviceId));return;}sender.post_scale_command(scale);Log.i(TAG, "ntsOnPlaybackMANSRTSPScaleCommand, scale:" + scale + " " + RecordSender.make_print_tuple(id, deviceId));}/** 随机拖动命令*/@Overridepublic void ntsOnPlaybackMANSRTSPSeekCommand(long id, String device_id, double position_sec) {if (position_sec < 0.0) {Log.e(TAG, "ntsOnPlaybackMANSRTSPSeekCommand invalid seek pos:" + position_sec + ", " + RecordSender.make_print_tuple(id, device_id));return;}RecordSender sender = this.senders_map_.get(id);if (null == sender) {Log.e(TAG, "ntsOnPlaybackMANSRTSPSeekCommand can not get sender " + RecordSender.make_print_tuple(id, device_id));return;}long offset_ms = sender.get_file_start_time_offset_ms();position_sec += (offset_ms/1000.0);sender.post_seek_command(position_sec);Log.i(TAG, "ntsOnPlaybackMANSRTSPSeekCommand seek pos:" + RecordSender.out_point_3(position_sec) + "s, " + RecordSender.make_print_tuple(id, device_id));}/** 停止命令*/@Overridepublic void ntsOnPlaybackMANSRTSPTeardownCommand(long id, String device_id) {CallTerminatePlaybackTask call_terminate_task = new CallTerminatePlaybackTask(this.context_, id, device_id, true);post_task(call_terminate_task);RecordSender sender = this.senders_map_.remove(id);if (null == sender) {Log.w(TAG, "ntsOnPlaybackMANSRTSPTeardownCommand can not remove sender " + RecordSender.make_print_tuple(id, device_id));return;}Log.i(TAG, "ntsOnPlaybackMANSRTSPTeardownCommand " + RecordSender.make_print_tuple(id, device_id));PlaybackListenerImpl.StopDisposeTask task = new PlaybackListenerImpl.StopDisposeTask(sender);if (!post_task(task))task.run();}/** 不是在收到BYE Message情况下, 终止Playback*/@Overridepublic void ntsOnTerminatePlayback(long id, String deviceId) {Log.i(TAG, "ntsOnTerminatePlayback, "+ RecordSender.make_print_tuple(id, deviceId));RecordSender sender = this.senders_map_.remove(id);if (null == sender)return;PlaybackListenerImpl.StopDisposeTask task = new PlaybackListenerImpl.StopDisposeTask(sender);if (!post_task(task))task.run();}/** Playback会话对应的对话终止, 一般不会触发这个回调,目前只有在响应了200K, 但在64*T1时间后还没收到ACK,才可能会出发收到这个, 请做相关清理处理*/@Overridepublic void ntsOnPlaybackDialogTerminated(long id, String deviceId) {Log.i(TAG, "ntsOnPlaybackDialogTerminated, "+ RecordSender.make_print_tuple(id, deviceId));RecordSender sender = this.senders_map_.remove(id);if (null == sender)return;PlaybackListenerImpl.StopDisposeTask task = new PlaybackListenerImpl.StopDisposeTask(sender);if (!post_task(task))task.run();}
}
安卓GB28181历史视音频的回放实现代码较多,大体上分为三块,信令部分,rtp打包发送部分,文件检索发送等逻辑, 有些用Java实现有些用C++, 这里为了方便说明信令流程和关键细节, 只给出部分接口定义和实现代码, 如有不清楚的地方请联系qq: 1130758427。
安卓设备录像文件远程回放实现代码细节较多, 处理起来也繁琐,建议能把录像文件从设备中拷出来播就拷吧。另外回放的音视频传输强烈推荐使用RTP over TCP(具体请参考RFC 4571)。