02 本地回环
本文是基于 Android WebRTC完整入门教程 这篇文章的实践过程记录，自己新增的内容主要体现在代码的注释中
这部分还可以参考大神PIASY的WebRTC native源码解析系列文章：https://blog.piasy.com/2017/08/30/WebRTC-P2P-part1/index.html
介绍WebRTC中最核心的概念PeerConnection , 给同一手机中的前后摄像头建立虚拟的连接, 相互传输画面
PeerConnection
PeerConnection也就是Peer-to-Peer connection(P2P), 就是两个"人"的连接. 双方分别创建PeerConnection对象, 然后向对方发送自己的网络状况ICE和多媒体编码格式SDP(因为这时候连接还没建立, 所以发送内容是通过服务器完成的). 当双方网络和编码格式协商好后, 连接就建立好了, 这时从PeerConnection中能获取到对方的MediaStream数据流, 也就能播放对方的音视频了
ICE
Interactive Connectivity Establishment, 交互式连接建立. 其实是一个整合STUN和TURN的框架, 给它提供STUN和TURN服务器地址, 它会自动选择优先级高的进行NAT穿透
SDP
Session Description Protocol: 会话描述协议. 发送方的叫Offer, 接受方的叫Answer, 除了名字外没有区别. 就是一些文本描述本地的音视频编码和网络地址等
主要流程
A(local)和B(remote)代表两个人, 初始化PeerConnectionFactory并分别创建PeerConnection , 并向PeerConnection 添加本地媒体流
- A创建Offer
- A保存Offer(set local description)
- A发送Offer给B
- B保存Offer(set remote description)
- B创建Answer
- B保存Answer(set local description)
- B发送Answer给A
- A保存Answer(set remote description)
- A发送ICE Candidates给B
- B发送ICE Candidates给A
- A, B收到对方的媒体流并播放
流程图如下
主要流程.png
准备步骤
主要是初始化PeerConnectionFactory和使用相机
public class MainActivity extends AppCompatActivity {
PeerConnectionFactory peerConnectionFactory;
PeerConnection peerConnectionLocal;
PeerConnection peerConnectionRemote;
SurfaceViewRenderer localView;
SurfaceViewRenderer remoteView;
// 在webrtc中，MediaStream代表一个媒体流，AudioTrack，VideoTrack代表音频”轨道”和视频“轨道”，如同一个MP4文件可以包含许多音轨和视频轨一样，一个MediaStream中可以包含多个AudioTrack和VideoTrack
MediaStream mediaStreamLocal;
MediaStream mediaStreamRemote;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
verifyStoragePermissions(this);
// 创建EglBase对象, WebRTC 把 EGL 的操作封装在了 EglBase 中，并针对 EGL10 和 EGL14 提供了不同的实现, 而 OpenGL 的绘制操作则封装在了 EglRenderer 中
// 获取EglBase对象的上下文
EglBase.Context eglBaseContext = EglBase.create().getEglBaseContext();
// create PeerConnectionFactory
// PeerConnectionFactory负责创建PeerConnection、VideoTrack、AudioTrack等重要对象
// 初始化PeerConnectionFactory
PeerConnectionFactory.initialize(PeerConnectionFactory.
InitializationOptions.
builder(this).
createInitializationOptions();
// PeerConnection的选项类
PeerConnectionFactory.Options options = new PeerConnectionFactory.Options();
// 视频编码工厂类，负责创建视频编码类
DefaultVideoEncoderFactory defaultVideoEncoderFactory = new DefaultVideoEncoderFactory(eglBaseContext,
true,
true);
// 视频解码工厂类
DefaultVideoDecoderFactory defaultVideoDecoderFactory = new DefaultVideoDecoderFactory(eglBaseContext);
// 设置PeerConnection工厂类，并创建PeerConnection工厂对象
peerConnectionFactory = PeerConnectionFactory.
builder().
setOptions(options).
setVideoEncoderFactory(defaultVideoEncoderFactory).
setVideoDecoderFactory(defaultVideoDecoderFactory).
createPeerConnectionFactory();
// SurfaceTextureHelper 负责创建 SurfaceTexture，接收 SurfaceTexture 数据，相机线程的管理
SurfaceTextureHelper localSurfaceTextureHelper = SurfaceTextureHelper.create("localCaptureThread", eglBaseContext);
// create VideoCapturer
// 获取前置摄像头
// WebRTC 视频采集的接口定义为 VideoCapturer，其中定义了初始化、启停、销毁等操作，以及接收启停事件、数据的回调
VideoCapturer localVideoCapturer = createCameraCapturer(true);
// peerConnectionFactory通过localVideoCapturer创建视频源
VideoSource localVideoSource = peerConnectionFactory.createVideoSource(localVideoCapturer.isScreencast());
localVideoCapturer.initialize(localSurfaceTextureHelper, getApplicationContext(), localVideoSource.getCapturerObserver());
localVideoCapturer.startCapture(480, 640, 30);//480, 640, 30分别是width, height, fps
//视频数据在 native 层处理完毕后会抛出到 VideoRenderer.Callbacks#renderFrame 回调中，在这里也就是 SurfaceViewRenderer#renderFrame，而 SurfaceViewRenderer 又会把数据交给 EglRenderer 进行渲染
localView = findViewById(R.id.localView);
localView.setMirror(true);
localView.init(eglBaseContext, null);
// create VideoTrack
// 创建视频轨，用于网络传输
VideoTrack localVideoTrack = peerConnectionFactory.createVideoTrack("100", localVideoSource);
// // display in localView
// localVideoTrack.addSink(localView);
SurfaceTextureHelper remoteSurfaceTextureHelper = SurfaceTextureHelper.create("remoteCaptureThread", eglBaseContext);
// create VideoCapturer
// 获取后置摄像头
VideoCapturer remoteVideoCapturer = createCameraCapturer(false);
VideoSource remoteVideoSource = peerConnectionFactory.createVideoSource(remoteVideoCapturer.isScreencast());
remoteVideoCapturer.initialize(remoteSurfaceTextureHelper, getApplicationContext(), remoteVideoSource.getCapturerObserver());
remoteVideoCapturer.startCapture(480, 640, 30);
remoteView = findViewById(R.id.remoteView);
remoteView.setMirror(true);
remoteView.init(eglBaseContext, null);
// create VideoTrack
// VideoTrack是webrtc中视频流最上层的接口，它内部其实是经过层层封装
VideoTrack remoteVideoTrack = peerConnectionFactory.createVideoTrack("100", remoteVideoSource);
// // display in remoteView
// remoteVideoTrack.addSink(remoteView);
// 向媒体流中添加视频轨
mediaStreamLocal = peerConnectionFactory.createLocalMediaStream("mediaStreamLocal");
mediaStreamLocal.addTrack(localVideoTrack);
mediaStreamRemote = peerConnectionFactory.createLocalMediaStream("mediaStreamRemote");
mediaStreamRemote.addTrack(remoteVideoTrack);
call(mediaStreamLocal, mediaStreamRemote);
}
...
}
使用相机
对createCameraCapturer()方法略作修改, 传入boolean参数就能分别获取前后摄像头, MainActivity.java中
// MainActivity.java中
// isFront==true 获取前置摄像头, 反之获取后置摄像头
private VideoCapturer createCameraCapturer(boolean isFront){
Camera1Enumerator enumerator = new Camera1Enumerator(false);
final String[] deviceNames = enumerator.getDeviceNames();
// First, try to find front facing cammera
for (String deviceName : deviceNames){
if (isFront ? enumerator.isFrontFacing(deviceName) : enumerator.isBackFacing(deviceName)){
VideoCapturer videoCapturer = enumerator.createCapturer(deviceName, null);
if (videoCapturer != null){
return videoCapturer;
}
}
}
return null;
}
拨打
建立连接的两人肯定有一个是拨打方, 另一个是接受方. 拨打方创建Offer发给接受方, 接收方收到后回复Answer。
// MainActivity.java中
private void call(MediaStream mediaStreamLocal, MediaStream mediaStreamRemote){
List<PeerConnection.IceServer> iceServers = new ArrayList<>();
// 创建本地的peerConnection
peerConnectionLocal = peerConnectionFactory.createPeerConnection(iceServers, new PeerConnectionAdapter("localConnection"){
@Override
// peerConnectionLocal发送收集到的iceCandidate
public void onIceCandidate(IceCandidate iceCandidate) {
super.onIceCandidate(iceCandidate);
// 远端添加iceCandidate
peerConnectionRemote.addIceCandidate(iceCandidate);
}
@Override
public void onAddStream(MediaStream mediaStream) {
super.onAddStream(mediaStream);
// 远端通过传输过来的媒体流将视频轨添加到localView进行渲染
VideoTrack remoteVideoTrack = mediaStream.videoTracks.get(0);
// “返回”到主线程，更新应用 UI
runOnUiThread(()->{
remoteVideoTrack.addSink(localView);
});
}
});
peerConnectionRemote = peerConnectionFactory.createPeerConnection(iceServers, new PeerConnectionAdapter("remoteConnection"){
@Override
public void onIceCandidate(IceCandidate iceCandidate) {
super.onIceCandidate(iceCandidate);
peerConnectionLocal.addIceCandidate(iceCandidate);
}
@Override
public void onAddStream(MediaStream mediaStream) {
super.onAddStream(mediaStream);
VideoTrack localVideoTrack = mediaStream.videoTracks.get(0);
runOnUiThread(()->{
localVideoTrack.addSink(remoteView);
});
}
});
// 添加本地的媒体流
peerConnectionLocal.addStream(mediaStreamLocal);
// 创建offer
peerConnectionLocal.createOffer(new SdpAdapter("local offer sdp"){
@Override
public void onCreateSuccess(SessionDescription sessionDescription) {
super.onCreateSuccess(sessionDescription);
peerConnectionLocal.setLocalDescription(new SdpAdapter("local set local"), sessionDescription);
peerConnectionRemote.addStream(mediaStreamRemote);
peerConnectionRemote.setRemoteDescription(new SdpAdapter("remote set remote"), sessionDescription);
peerConnectionRemote.createAnswer(new SdpAdapter("remote answer sdp"){
@Override
public void onCreateSuccess(SessionDescription sessionDescription) {
super.onCreateSuccess(sessionDescription);
peerConnectionRemote.setLocalDescription(new SdpAdapter("remote set local"), sessionDescription);
peerConnectionLocal.setRemoteDescription(new SdpAdapter("local set remote"), sessionDescription);
}
}, new MediaConstraints());
}
}, new MediaConstraints());
}
添加PeerConnectionAdapter类作为PeerConnection的观察者
public class PeerConnectionAdapter implements PeerConnection.Observer {
private String tag;
public PeerConnectionAdapter(String tag){
this.tag = "bo" + tag;
}
public void log(String str){
Log.d(this.tag, str);
}
@Override
public void onSignalingChange(PeerConnection.SignalingState signalingState) {
log("onSignalingChange" + signalingState);
}
@Override
public void onIceConnectionChange(PeerConnection.IceConnectionState iceConnectionState) {
log("onIceConnectionChange" + iceConnectionState);
}
@Override
public void onIceConnectionReceivingChange(boolean b) {
log("onIceConnectionReceivingChange" + b);
}
@Override
public void onIceGatheringChange(PeerConnection.IceGatheringState iceGatheringState) {
log("onIceGatheringChange" + iceGatheringState);
}
@Override
public void onIceCandidate(IceCandidate iceCandidate) {
log("onIceCandidate" + iceCandidate);
}
@Override
public void onIceCandidatesRemoved(IceCandidate[] iceCandidates) {
log("onIceCandidatesRemoved" + iceCandidates);
}
@Override
public void onAddStream(MediaStream mediaStream) {
log("onAddStream" + mediaStream);
}
@Override
public void onRemoveStream(MediaStream mediaStream) {
log("onRemoveStream" + mediaStream);
}
@Override
public void onDataChannel(DataChannel dataChannel) {
log("onDataChannel" + dataChannel);
}
@Override
public void onRenegotiationNeeded() {
log("onRenegotiationNeeded");
}
@Override
public void onAddTrack(RtpReceiver rtpReceiver, MediaStream[] mediaStreams) {
log("onAddTrack" + mediaStreams);
}
}
添加SdpAdapter(继承SdpObserver)作为Sdp的观察者
public class SdpAdapter implements SdpObserver {
private String tag;
public SdpAdapter(String tag){
this.tag = "bo" + tag;
}
public void log(String str){
Log.d(tag, str);
}
@Override
public void onCreateSuccess(SessionDescription sessionDescription) {
log("onCreateSuccess " + sessionDescription);
}
@Override
public void onSetSuccess() {
log("onSetSuccess ");
}
@Override
public void onCreateFailure(String s) {
log("onCreateFailure " + s);
}
@Override
public void onSetFailure(String s) {
log("onSetFailure " + s);
}
}
注意: 虽然这里没有真正使用到网络, 但是要添加网络权限
<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>
Android6.0及以上申请权限
private static final int REQUEST_ALL = 1;
private static String[] PERMISSIONS_ALL = {
"android.permission.CAMERA",
"android.permission.RECORD_AUDIO",
"android.permission.INTERNET",
"android.permission.ACCESS_NETWORK_STATE"
};
//然后通过一个函数来申请
public static void verifyStoragePermissions(Activity activity) {
try {
int permission = 0;
//检测所有需要的权限
for(String temp : PERMISSIONS_ALL){
permission = ActivityCompat.checkSelfPermission(activity, temp);
if (permission != PackageManager.PERMISSION_GRANTED){
break;
}
}
if (permission != PackageManager.PERMISSION_GRANTED) {
// 没有写的权限，去申请写的权限，会弹出对话框
ActivityCompat.requestPermissions(activity, PERMISSIONS_ALL,REQUEST_ALL);
}
} catch (Exception e) {
e.printStackTrace();
}
}
