web rtc

WebRTC Code and API‎ > ‎

WebRTC Native APIs

The WebRTC native APIs are implemented based on the following WebRTC spec.

The code that implements WebRTC native APIs (including the Stream and the PeerConnection APIs) are available inlibjingle. A sample client application is also provided there.

The target audience of this document are those who want to use WebRTC Native APIs to implement WebRTC javascript APIs or to develop native RTC applications.

Contents

1. 1 What is new
2. 2 Block diagram
3. 3 Calling sequences
   1. 3.1 Set up a call
   2. 3.2 Receive a call
   3. 3.3 Close down a call
4. 4 Threading model
5. 5 Stream APIs (mediastream.h)
   1. 5.1 Class MediaStreamTrackInterface
   2. 5.2 Class VideoTrackInterface
   3. 5.3 Class LocalVideoTrackInterface
   4. 5.4 Class AudioTrackInterface
   5. 5.5 Class LocalAudioTrackInterface
   6. 5.6 Class cricket::VideoRenderer, cricket::VideoCapturer
   7. 5.7 Class webrtc::AudioDeviceModule
   8. 5.8 Class MediaStreamInterface
   9. 5.9 Class LocalMediaStreamInterface
6. 6 PeerConnection APIs (peerconnection.h)
   1. 6.1 Class StreamCollectionInterface
   2. 6.2 Class PeerConnectionObserver
   3. 6.3 Class PortAllocatorFactoryInterface
   4. 6.4 Class PeerConnectionFactoryInterface
   5. 6.5 Function CreatePeerConnectionFactory
   6. 6.6 Function CreatePeerConnectionFactory
   7. 6.7 Class PeerConnectionInterface
7. 7 Reference

What is new

Compared to the previous version of WebRTC native APIs (code is availablehere as part of libjingle r115), the major difference is the new version contains the implementation of theStream APIs. With the Stream APIs, the audio/video I/O will be set to the MediaTrack instead of directly to the PeerConnection. Also PeerConnection now takes and returns the MediaStream object instead of just the media stream label as in the previous version.

In addition to this, the signaling protocol has also changed toROAP in this version, however, this should be transparent to the users of the APIs.

For those who have already developed the application based on the previous version, please check this patch as an example on how to migrate to the new APIs. The sample client application mentioned above can also be used as a reference.

---

Block diagram

---

Calling sequences

Set up a call

Receive a call

Close down a call

---

Threading model

WebRTC native APIs use two globally available threads: the signaling thread and the worker thread.Depending on how the PeerConnection factory is created,the application can eitherprovide those 2 threads or just let them be created internally.

The calls to the Stream APIs and the PeerConnection APIs will be proxied to the signaling thread which means that the application can call those APIs from whatever thread. 

All callbacks will be made on the signaling thread. The application should return the callback as quickly as possible to avoid blocking the signaling thread. Resource intensive processes should be posted to a different thread.

The worker thread is used to handle more resource intensive processes such as data streaming.

---

Stream APIs (mediastream.h)

---

Class MediaStreamTrackInterface

This class declares an abstract interface to the media stream track which represents a media source in the user agent.

class MediaStreamTrackInterface : public talk_base::RefCountInterface,
                                 public NotifierInterface {
public:
 enum TrackState {
   kInitializing,
   kLive = 1,
   kEnded = 2,
   kFailed = 3,
 };
 virtual std::string kind() const = 0;
 virtual std::string label() const = 0;
 virtual bool enabled() const = 0;
 virtual TrackState state() const = 0;
 virtual bool set_enabled(bool enable) = 0;
 virtual bool set_state(TrackState new_state) = 0;
};

MediaStreamTrackInterface::TrackState

This enumerator is used to specify the type of the track states.

Syntax

 enum TrackState {
   kInitializing,
   kLive = 1,
   kEnded = 2,
   kFailed = 3,
 };

Remarks

kInitializing - Track is beeing negotiated.
kLive - Track is alive.
kEnded - Trackhas ended.
kFailed - Track negotiation failed.

MediaStreamTrackInterface::kind

Returns the string "audio" if the track is an audio track, "video" if the track is a video track, or a user-agent defined string otherwise.

Syntax

 virtual std::string kind() const = 0;

MediaStreamTrackInterface::label

Returns the label of the track, if any. If the track has no label, it returns the empty string.

Syntax

 virtual std::string label() const = 0;

MediaStreamTrackInterface::enabled

Returns “true” if the track is enabled, “false” otherwise.

Syntax

 virtual bool enabled() const = 0;

MediaStreamTrackInterface::state

Returns the current state of the track.

Syntax

 virtual TrackState state() const = 0;

MediaStreamTrackInterface::set_enabled

Enable (true) or disable (false) a track.

Syntax

 virtual bool set_enabled(bool enable) = 0;

Remarks

Not implemented.

MediaStreamTrackInterface::set_state

Set the track to a new state.

Syntax

 virtual bool set_state(TrackState new_state) = 0;

Remarks

This method is used by the PeerConnection internally, the application should not call it directly.

---

Class VideoTrackInterface

The VideoTrackInterface derives from MediaStreamTrackInterface with two extra interfaces to get and set the video renderer.

class VideoTrackInterface : public MediaStreamTrackInterface {
public:
 virtual void SetRenderer(VideoRendererWrapperInterface* renderer) = 0;
 virtual VideoRendererWrapperInterface* GetRenderer() = 0;
protected:
 virtual ~VideoTrackInterface() {}

};

---

Class LocalVideoTrackInterface

The LocalVideoTrackInterface derives from VideoTrackInterface with one extra interface to get the video capture associated with the track.

class LocalVideoTrackInterface : public VideoTrackInterface {

public:

 virtual cricket::VideoCapturer* GetVideoCapture() = 0;

protected:

 virtual ~LocalVideoTrackInterface() {}

};

---

Class AudioTrackInterface

The AudioTrackInterface derives from MediaStreamTrackInterface and currently without extra interface.

class AudioTrackInterface : public MediaStreamTrackInterface {

public:

protected:

 virtual ~AudioTrackInterface() {}

};

---

Class LocalAudioTrackInterface

The LocalAudioTrackInterface derives from AudioTrackInterface with one extra interface to get audio device associated with this track.

class LocalAudioTrackInterface : public AudioTrackInterface {

public:

 virtual AudioDeviceModule* GetAudioDevice() =  0;

protected:

 virtual ~LocalAudioTrackInterface() {}

};

---

Class cricket::VideoRenderer, cricket::VideoCapturer

These classes are defined by the open source projectlibjingle and will not be described in detail here..

---

Class webrtc::AudioDeviceModule

The AudioDeviceModule is defined by the open source projectwebrtc. Please refer to this link for a detailed definition.

---

Class MediaStreamInterface

This class declares an abstract interface to MediaStream which, typically but not necessarily, represents streams of media data of audio and/or video content.

EachMediaStream object can contain zero or more tracks, in particular audio and video tracks. All tracks in a MediaStream are intended to be synchronized when rendered. Different MediaStreams do not need to be synchronized.

class MediaStreamInterface : public talk_base::RefCountInterface,

                            public NotifierInterface {

public:

 virtual std::string label() const = 0;

 virtual AudioTracks* audio_tracks() = 0;

 virtual VideoTracks* video_tracks() = 0;

 enum ReadyState {

   kInitializing,

   kLive = 1,  // Stream alive

   kEnded = 2,  // Stream have ended

 };

 virtual ReadyState ready_state() = 0;

protected:

 virtual ~MediaStreamInterface() {}

};

MediaStreamInterface::label

Returns a label that is unique to this stream, so that streams can be recognized after they are sent through the PeerConnection APIs.

Syntax

 virtual std::string label() const = 0;

MediaStreamInterface::audio_tracks

Returns a pointer to a list of objects of typeMediaStreamTrack representing the audio tracks associated with this MediaStream.

Syntax

 virtual AudioTracks* audio_tracks() = 0;

MediaStreamInterface::video_tracks

Returns a pointer to a list of objects of typeMediaStreamTrack representing the video tracks associated with this MediaStream.

Syntax

 virtual VideoTracks* video_tracks() = 0;

MediaStreamInterface::ready_state

Returns the current ready state of the MediaStream.

Syntax

 virtual ReadyState ready_state() = 0;

---

Class LocalMediaStreamInterface

The LocalMediaStreamInterface derives from the MediaStreamInterface with extra interfaces to add media tracks to the MediaStream.

class LocalMediaStreamInterface : public MediaStreamInterface {
public:
 virtual bool AddTrack(AudioTrackInterface* track) = 0;
 virtual bool AddTrack(VideoTrackInterface* track) = 0;

};

---

PeerConnection APIs (peerconnection.h)

---

Class StreamCollectionInterface

This class declares a MediaStream container interface.

class StreamCollectionInterface : public talk_base::RefCountInterface {
public:
 virtual size_t count() = 0;
 virtual MediaStreamInterface* at(size_t index) = 0;
 virtual MediaStreamInterface* find(const std::string& label) = 0;
protected:
 ~StreamCollectionInterface() {}
};

StreamCollectionInterface::count

Returns the number of MediaStreams in the collection.

Syntax

size_t count() = 0;

StreamCollectionInterface::at

Returns a reference to the MediaStream at the position index in the collection.

Syntax

MediaStreamInterface* at(size_t index) = 0;

Parameters

index[in]Position of aMediaStream in the collection.

StreamCollectionInterface::find

Searches the collection for a MediaStream with given label and returns a reference to it if found, otherwise it returns NULL.

Syntax

MediaStreamInterface* find(const std::string& label) = 0;

Parameters

label[in]The label value to be searched for.

---

Class PeerConnectionObserver

This class declares an abstract interface for a user defined observer. It is up to the PeerConnection user to implement a derived class which implements the observer class. The observer is registered when the PeerConnection is created using the PeerConnectionFactoryInterface.

class PeerConnectionObserver {
public:
 enum StateType {
   kReadyState,
   kIceState,
   kSdpState,
 };
 virtual void OnError() = 0;
 virtual void OnMessage(const std::string& msg) = 0;
 virtual void OnSignalingMessage(const std::string& msg) = 0;
 virtual void OnStateChange(StateType state_changed) = 0;
 virtual void OnAddStream(MediaStreamInterface* stream) = 0;
 virtual void OnRemoveStream(MediaStreamInterface* stream) = 0;
protected:
 ~PeerConnectionObserver() {}
};

PeerConnectionObserver::StateType

This enumerator is used to specify the type of the state machines.

Syntax

 enum StateType {
   kReadyState,
   kIceState,
   kSdpState,
 };

PeerConnectionObserver::OnError

This method will be calledwhen an error occursin PeerConnection.

Syntax

void OnError() = 0;

Remarks

Not implemented.

PeerConnectionObserver::OnMessage

This method will be calledwhen a text message is received from the remote peer.

Syntax

void OnMessage(const std::string& msg) = 0;

PeerConnectionObserver::OnSignalingMessage

This method is called once a signaling message is ready.

Syntax

void OnSignalingMessage(const std::string& msg) = 0;

Parameters

msg[in] AROAP format signaling message.

Remarks

The user should send the signaling message from the callback to the remote peer.

PeerConnectionObserver::OnStateChange

This method is called when any of the 3 state machines (ReadyState, SdpState or IceState) has a state change.

Syntax

 virtual void OnStateChange(StateType state_changed) = 0;

Parameters

state_changed[in] Specify which state machine’s state has changed.

Remarks

The IceState is not implemented.

PeerConnectionObserver::OnAddStream

This method is called when media is received on a new stream from remote peer.

Syntax

 virtual void OnAddStream(MediaStreamInterface* stream) = 0;

Parameters

stream[in] The handler to the remote media stream.

Remarks

The user may use this event to set the renderer for the received media stream.

PeerConnectionObserver::OnRemoveStream

This method is calledwhen the remote peer closes a stream.

Syntax

 virtual void OnRemoveStream(MediaStreamInterface* stream) = 0;

Parameters

stream[in] The handler to the closed remote media stream.

---

Class PortAllocatorFactoryInterface

This class declares a factory interface for creating the cricket::PortAllocator, which is used for the ICE negotiation. The PeerConnection factory will use this interface (if supplied) to create the PortAllocator for PeerConnection. The application may provide its own implementation of the PortAllocator by implementing the PortAllocatorFactoryInterface - basically the CreatePortAllocator method.

class PortAllocatorFactoryInterface : public talk_base::RefCountInterface {
public:
 struct StunConfiguration {
   StunConfiguration(const std::string& address, int port)
       : server(address, port) {}
   talk_base::SocketAddress server;
 };
 struct TurnConfiguration {
   TurnConfiguration(const std::string& address,
                     int port,
                     const std::string& user_name,
                     const std::string& password)
       : server(address, port),
         username(username),
         password(password) {}
   talk_base::SocketAddress server;
   std::string username;
   std::string password;
 };
 virtual cricket::PortAllocator* CreatePortAllocator(
     const std::vector<StunConfiguration>& stun_servers,
     const std::vector<TurnConfiguration>& turn_configurations) = 0;
protected:
 PortAllocatorFactoryInterface() {}
 ~PortAllocatorFactoryInterface() {}
};

PortAllocatorFactoryInterface::CreatePortAllocator

This method returns an instance of the PortAllocator class.

Syntax

 virtual cricket::PortAllocator* CreatePortAllocator(
     const std::vector<StunConfiguration>& stun_servers,
     const std::vector<TurnConfiguration>& turn_configurations) = 0;

Parameters

stun_servers[in] A configuration list of the STUN servers.

turn_servers[in] A configuration list of the TURN servers.

Remarks

TURN is currently not implemented.

---

Class PeerConnectionFactoryInterface

The PeerConnectionFactoryInterface is the factory interface used for creating PeerConnection, media stream and media tracks.

class PeerConnectionFactoryInterface : public talk_base::RefCountInterface {
public:
 virtual talk_base::scoped_refptr<PeerConnectionInterface>
     CreatePeerConnection(const std::string& config,
                          PeerConnectionObserver* observer) = 0;
 virtual talk_base::scoped_refptr<LocalMediaStreamInterface>
     CreateLocalMediaStream(const std::string& label) = 0;
 virtual talk_base::scoped_refptr<LocalVideoTrackInterface>
     CreateLocalVideoTrack(const std::string& label,
                           cricket::VideoCapturer* video_device) = 0;
 virtual talk_base::scoped_refptr<LocalAudioTrackInterface>
     CreateLocalAudioTrack(const std::string& label,
                           AudioDeviceModule* audio_device) = 0;
protected:
 PeerConnectionFactoryInterface() {}
 ~PeerConnectionFactoryInterface() {}
};

PeerConnectionFactoryInterface::CreatePeerConnection

Create an instance of the PeerConnection.

Syntax

 virtual talk_base::scoped_refptr<PeerConnectionInterface>
     CreatePeerConnection(const std::string& config,
                        PeerConnectionObserver* observer) = 0;

Parameters

config[in] The configuration string gives the address of a STUN or TURN server to use to establish the connection. The format is defined in the webrtc-api.

observer [in] A pointer to an instance of the PeerConnectionObserver derived class.

Remarks

TURN is currently not supported.

The allowed formats for the config string are:

"TYPE 203.0.113.2:3478"

Indicates a specific IP address and port for the server.

"TYPE relay.example.net:3478"

Indicates a specific host and port for the server; the user agent will look up the IP address in DNS.

"TYPE example.net"

Indicates a specific domain for the server; the user agent will look up the IP address and port in DNS.

The "TYPE" is one of:

STUN

Indicates a STUN server

STUNS

Indicates a STUN server that is to be contacted using a TLS session.

TURN

Indicates a TURN server

TURNS

Indicates a TURN server that is to be contacted using a TLS session.

PeerConnectionFactoryInterface::CreateLocalMediaStream

Create an instance of a local media stream.

Syntax

 virtual talk_base::scoped_refptr<LocalMediaStreamInterface>
     CreateLocalMediaStream(const std::string& label) = 0;

Parameters

label[in] Desired local media stream label.

PeerConnectionFactoryInterface::CreateLocalVideoTrack

Create an instance of a local video track.

Syntax

 virtual talk_base::scoped_refptr<LocalVideoTrackInterface>
     CreateLocalVideoTrack(const std::string& label,
                           cricket::VideoCapturer* video_device) = 0;

Parameters

label[in] Desired local video track label.
video_device[in] Pointer to the video capture device that is going to associate with this track.

PeerConnectionFactoryInterface::CreateLocalAudioTrack

Create an instance of a local audio track.

Syntax

 virtual talk_base::scoped_refptr<LocalVideoTrackInterface>
     CreateLocalAudioTrack(const std::string& label,
                           AudioDeviceModule* audio_device) = 0;

Parameters

label[in] Desired local audio track label.
audio_device[in] Pointer to the audio device that is going to associate with this track.

---

Function CreatePeerConnectionFactory

Create a new instance of PeerConnectionFactoryInterface.

Syntax

talk_base::scoped_refptr<PeerConnectionFactoryInterface>
CreatePeerConnectionFactory();

Remarks

The PeerConnectionFactoryInterface instance generated by this function will create required resources internally - including libjingle threads, socket and network manager factory classes for networking.

---

Function CreatePeerConnectionFactory

Create a new instance of PeerConnectionFactoryInterface with the given libjingle threads and portallocator factory.

Syntax

talk_base::scoped_refptr<PeerConnectionFactoryInterface>
CreatePeerConnectionFactory(talk_base::Thread* worker_thread,
                           talk_base::Thread* signaling_thread,
                           PortAllocatorFactoryInterface* factory,
                           AudioDeviceModule* default_adm);

Remarks

This function can be used when the application wants to provide its own implementation of the threads and the portallocator.
Ownership of the arguments are not transferred to this object and must remain in scope for the lifetime of the PeerConnectionFactoryInterface.

---

Class PeerConnectionInterface

class PeerConnectionInterface : public talk_base::RefCountInterface {
public:
 enum ReadyState {
   kNew,
   kNegotiating,
   kActive,
   kClosing,
   kClosed,
 };
 enum SdpState {
   kSdpNew,
   kSdpIdle,
   kSdpWaiting,
 };
 virtual void ProcessSignalingMessage(const std::string& msg) = 0;
 virtual bool Send(const std::string& msg) = 0;
 virtual talk_base::scoped_refptr<StreamCollectionInterface>
     local_streams() = 0;
 virtual talk_base::scoped_refptr<StreamCollectionInterface>
     remote_streams() = 0;
 virtual void AddStream(LocalMediaStreamInterface* stream) = 0;
 virtual void RemoveStream(LocalMediaStreamInterface* stream) = 0;
 virtual void CommitStreamChanges() = 0;
 virtual void Close() = 0;
 virtual ReadyState ready_state() = 0;
 virtual SdpState sdp_state() = 0;
protected:
 ~PeerConnectionInterface() {}
};

PeerConnectionInterface::ReadyState

This enumerator is used to specify the type of the ready state.

• kNew - The object was just created and its ICE and SDP Agent have not yet been started.
• kNegotiating - The peerConenction object is attempting to get to the point where media can flow.
• kActive - A connection has been formed and if any media streams were successfully negotiated, any relevant media can be streaming.
• kClosing - The object is starting to shut down after the close()method has been invoked.
• kClosed - The shutting down is completed.

Syntax
 enum ReadyState {
   kNew,
   kNegotiating,
   kActive,
   kClosing,
   kClosed,
 };

PeerConnectionInterface::SdpState

This enumerator is used to specify the type of the SDP state.

• kSdpNew - The object was just created and SDP Agent have not yet been started.
• kSdpIdle - A valid offer answer pair has been exchanged and the SDP Agent is waiting for the next SDP transaction.
• kSdpWaiting - The SDP Agent has sent an SDP offer and is waiting for a response.

Syntax

 enum SdpState {

   kSdpNew,  // TODO(ronghuawu): kSdpNew is not defined in the spec.

   kSdpIdle,

   kSdpWaiting,

 };

PeerConnectionInterface::ProcessSignalingMessage

Handle the signaling message from the remote peer.

Syntax

 virtual void ProcessSignalingMessage(const std::string& msg) = 0;

Parameters

msg

[in] AROAP format signaling message.

Remarks

The order of messages is important. Passing messages to the PeerConnection in a different order than they were generated by the remote peer can prevent a successful connection from being established or degrade the connection's quality if one is established.

PeerConnectionInterface::Send

Send a text message to the remote peer over the data stream.

Syntax

 virtual bool Send(const std::string& msg) = 0;  // TODO(ronghuawu): This is not defined in the spec.

Parameters

msg

[in] The text message to be sent to the remote peer.

Remarks

Not currently implemented.

PeerConnectionInterface::local_streams

Returns a live array containing the streams that the user agent is currently attempting to transmit to the remote peer (those that were added with A

ddStream()).

Syntax

 virtual talk_base::scoped_refptr<StreamCollectionInterface>

     local_streams() = 0;

PeerConnectionInterface::remote_streams

Returns a live array containing the streams that the user agent is currently receiving from the remote peer.

This array is updated when OnAddStream and OnRemoveStream callbacks are fired.

Syntax

 virtual talk_base::scoped_refptr<StreamCollectionInterface>

     remote_streams() = 0;

PeerConnectionInterface::AddStream

Add a local stream to the array of the streams that the user agent is currently attempting to transmit to the remote peer. This function only adds the stream to the array and does not trigger any changes until CommitStreamChanges is called.

Syntax

 virtual void AddStream(LocalMediaStreamInterface* stream) = 0;

Parameters

stream

[in] Pointer to the local media stream to be added.

PeerConnectionInterface::RemoveStream

Remove a local stream from the array of the streams that the user agent is currently attempting to transmit to the remote peer. This function only removes the stream from the array and does not trigger any changes until CommitStreamChanges is called.

Syntax

 virtual void RemoveStream(LocalMediaStreamInterface* stream) = 0;

Parameters

stream

[in] Pointer to the local media stream to be removed.

PeerConnectionInterface::CommitStreamChanges

Commit Stream changes made by the AddStream and RemoveStream. This will start sending media on new added streams and stop sending media on removed streams.

Syntax

 virtual void CommitStreamChanges() = 0;

PeerConnectionInterface::Close

Close the current session. This will trigger a Shutdown message to be sent and the ready state will change to kClosing.

Syntax

 virtual void Close() = 0;

PeerConnectionInterface::ready_state

Return thePeerConnection object's readiness state, represented by enum ReadyState.

Syntax

 virtual ReadyState ready_state() = 0;

PeerConnectionInterface::sdp_state

Return the state of thePeerConnection SDP Agent, represented by enum SdpState.

Syntax

 virtual SdpState sdp_state() = 0;

---

Reference

The current HTML5 specification for WebRTC is here:

http://dev.w3.org/2011/webrtc/editor/webrtc.html

The source code of the WebRTC Native API is here:

https://code.google.com/p/libjingle/source/browse/trunk/talk/app/webrtc/

Client and server sample apps can be found here:

https://code.google.com/p/libjingle/source/browse/trunk/talk/#talk%2Fexamples%2Fpeerconnection