Latest news about Bitcoin and all cryptocurrencies. Your daily crypto news habit.
Webrtc is a cross platform solution with RTC capabilities. One can stream his own video stream be it from camera or screen recording or any other video to any peer via webrtc. If one is developing a real time peer to peer game with real time data sharing between the peers, webrtc is one of the options. Let’s understand how video calling from one peer to another works in webrtc.
Getting started
Since September 2017 Google started to distributed precompiled versions of webrtc for android via maven. If you want to play with the source and compile it yourself you can do it easily as per these steps(Earlier you could compile it only on linux but now all the three major OS are supported). To use the pre compiled version simply add the following dependency.
compile 'org.webrtc:google-webrtc:1.0.22379'
How does webrtc work?
Before we start exchanging data between two peers via webrtc, we need to provide info to the peers about each other for media format negotiation and discovery. This is done via following protocols. Interactive Connectivity Establishment (ICE) is used for connecting peer to peer. Session Description Protocol (SDP) is used to provide the metadata of the media content like resolution, encoding, bitrate, etc. If the two peers are not on same network then we will need to provide a Session Traversal Utilities for NAT (STUN) server to provide the public address of the peers. If any of the network is firewall protected the we need to provide Traversal Using Relays around NAT (TURN) servers also. You can learn more about these protocols here. The mechanism to exchange ICE and SDP between the peers is called Signalling System and is usually done via websockets.
Webrtc android API
Most of our webrtc code will make use of PeerConnection and PeerConnectionFactory apis. PeerConnection is the equivalent of RTCPeerConnection in web world and is used to establish peer to peer connection. PeerConnectionFactory is used to create PeerConnection, MediaStream and MediaStreamTrack objects.
Signalling flow diagram for Webrtc (original photo by Mozilla Contributers is licensed under CC-BY-SA 2.5.)
Creating PeerConnectionFactoryBefore creating the factory object we need to initialize webrtc. As you can InitializationOptions here provides options for enabling/disabling hardware accelerations(I had to disable in on certain devices) and setting field trials. Field trials are for enabling experimental webrtc features.
val fieldTrials = (PeerConnectionFactory.VIDEO_FRAME_EMIT_TRIAL + "/" + PeerConnectionFactory.TRIAL_ENABLED + "/")val options = InitializationOptions.builder(application) .setFieldTrials(fieldTrials) .setEnableVideoHwAcceleration(videoAccelerationEnabled) .createInitializationOptions()PeerConnectionFactory.initialize(options)factory = PeerConnectionFactory(PeerConnectionFactory.Options())val rootEglBase = EglBase.create()factory?.setVideoHwAccelerationOptions(rootEglBase.eglBaseContext, rootEglBase.eglBaseContext)
Creating Media StreamsOnce we have the PeerConnectionFactory object we can now create a MediaStream object which has AudioTrack and VideoTrack associated with it.
val localMediaStream = factory.createLocalMediaStream(MEDIA_ID)val audioSource = factory.createAudioSource(MediaConstraints())val audioTrack = factory.createAudioTrack(AUDIO_ID, audioSource)localMediaStream.addTrack(audioTrack)videoCapturer = if (Build.VERSION.SDK_INT > Build.VERSION_CODES.LOLLIPOP) { createCameraCapturer(Camera2Enumerator(application))} else { createCameraCapturer(Camera1Enumerator(videoAccelerationEnabled))}val videoTrack = factory.createVideoTrack("VideoTrack", factory.createVideoSource(videoCapturer))localMediaStream.addTrack(videoTrack)
CameraVideoCapturerWebrtc provides us a very easy way to use Camera and Camera2 API depending on the support. On supported devices we can use either of the APIs
private fun createCameraCapturer(enumerator: CameraEnumerator): CameraVideoCapturer? { val deviceNames = enumerator.deviceNames for (deviceName in deviceNames) { if (enumerator.isFrontFacing(deviceName)) { val videoCapturer = enumerator.createCapturer(deviceName, null) if (videoCapturer != null) { return videoCapturer } } } for (deviceName in deviceNames) { if (!enumerator.isFrontFacing(deviceName)) { Timber.d("Creating other camera capturer.") val videoCapturer = enumerator.createCapturer(deviceName, null) if (videoCapturer != null) { return videoCapturer } } } return null}
SurfaceViewRendererSince we have the local MediaStream ready, we need to render it on a view so that its visible to the user. SurfaceViewRenderer is a View in webrtc libray that does the rendering of webrtc frames for us. We can directly add it in layout xml.
localViewRenderer.init(rootEglBase.getEglBaseContext(), null)localViewRenderer.setEnableHardwareScaler(true)localViewRenderer.setMirror(true)localViewRenderer.setScalingType(ScalingType.SCALE_ASPECT_FILL)val localVideoRenderer = VideoRenderer(localViewRenderer)videoTrack.addRenderer(localVideoRenderer)
PeerConnectionWe have everything ready to transmit now, its time to create the PeerConnection. We pass in a PeerConnection#Observer instance in the factory method, which is notified when a ICE candidate is generated (we need to send that to the other peer). The observer is also notified when the remote MediaStream is available. we will attach a renderer to it just like the local MediaStream. It also requires a list IceServer(STUN and TURN servers) which can be empty if testing on local network
val peerConnectionObserver = object : PeerConnection.Observer { override fun onIceCandidate(iceCandidate: IceCandidate) { localIceCandidatesSource.onNext(iceCandidate) } override fun onAddStream(mediaStream: MediaStream) { mediaStream.addRenderer(remoteRenderer) } ...}peerConnection = factory?.createPeerConnection(getIceServers(), peerConnectionObserver)
NOTE: In Observer’s onIceCandidate we shouldn’t add the IceCandidate to PeerConnection (peerConnection.addIceCandidate(iceCandidate) till we have the remote SDP
CreateOffer/CreateAnswerOnce the PeerConnection is created we need to initiate video call. The peer which initiates the call will createOffer and set its local SDP (peerConnection#setLocalSdp). When the local SDP is set it will send that SDP to the other peer which will set its remote SDP (peerConnection#setRemoteSdp). Once the remote SDP is set it will createAnswer with that sdp and set its local sdp when asnwer is created and send the local sdp to the peer which created the offer. The initiator peer will now set its remote SDP.
fun createOffer() { peerConnection.createOffer(object : SdpObserver { override fun onCreateSuccess(sdp: SessionDescription) { setLocalSdp(sdp) } ... }, getPeerConnectionConstraints())}fun setLocalSdp(sdp: SessionDescripton) { peerConnection.setLocalDescription(object : SdpObserver { override fun onSetSuccess() { api.sendSdp(peerConnection.localDescription) drainIceCandidates() } ... }, sdp)}
fun onOffer(sdp: SessionDescription) { peerConnection.setRemoteDescription(object : SdpObserver { override fun onSetSuccess() { createAnswer() } }, sdp)}
fun createAnswer() { peerConnection?.createAnswer(object : SdpObserver { override fun onCreateSuccess(sdp: SessionDescription) { setLocalSdp(sdp) } }, getPeerConnectionConstraints())}
Dispose/ Clean upWhen the peers decide to end the connection its pretty important to do clean up in particular order as the C layer does a reference count check and will crash with assertion failures if the objects are not properly disposed.
fun cleanUp() { peerConnection.dispose() viceoCapturer.dispose() videoSource.dispose() factory.dispose() localVideoRenderer.dispose() remoteRenderer.dispose() localViewRenderer.release() remoteViewRenderer.release() rootEglBase.release()}
Next UpBasic socket implementation for webrtc using ktor.io and video filters for webrtc.
References
- https://tech.appear.in/2015/05/25/Introduction-to-WebRTC-on-Android/
- https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Signaling_and_video_calling
Real time communication with Webrtc on Android was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.
Disclaimer
The views and opinions expressed in this article are solely those of the authors and do not reflect the views of Bitcoin Insider. Every investment and trading move involves risk - this is especially true for cryptocurrencies given their volatility. We strongly advise our readers to conduct their own research when making a decision.