OpenAI recently added the ability to connect to its Realtime API via Webrtc, enabling realtime speech to speech AI apps. Other AI platforms will likely follow suit.
The webrtc implementation on the Realtime API uses data channels for issuing instructions, context, updates etc.
I intend to use Janus SIP plugin with this API to connect telephony users to LLMs.
Are there any plans to add data channel support to the SIP plugin? This would make Janus an essential tool for integrating telephony stacks with LLMs.
Thank you for the great work - I’ve used Janus for 5+ years now to build mission-critical apps.
We already have partial data channels support in the SIP plugin in a couple of efforts, most noticeable the one that tries to add RTT T.140 support. That said, it’s a very specific and vertical usage of the data channel.
I don’t think what you want to do makes much sense in the SIP plugin, and it would be hard to implement too. First of all, it would mean having an asymmetry in the SDP between the WebRTC and the SIP side: the WebRTC side would have an additional m-line for data channels that wouldn’t exist on the SIP side, making any bridging of the SDPs between the two technologies cumbersome. Besides, even ignoring that aspect, that would imply that the SIP plugin would need to have knowledge of the OpenAI messaging semantics, as it’s the SIP plugin that would need to send commands (which ones? when?) or handle events (to do what with?). I’m not saying it’s impossible, just that I see an integration complex, and with some issues in terms of interfaces (how to control all that).
From how I see it, initial experimentation with this OpenAI feature probably makes more sense in a more generic plugin like the Lua or Duktape plugin, where you have complete control on what you want to do with the data channel messages coming in and out, while at the same time allowing you to relay RTP packets contextually.
Thank you for the comprehensive reply Lorenzo. Here’s what I think:
Granted, datachannels support on the SIP plugin would result in SDP asymmetry on the webrtc side vs on the SIP side, since SIP can’t support datachannels.
On the SIP plugin being mandated to understand vendor-specific messaging semantics, I assume that it wouldn’t work that way, because the SIP plugin could expose:
A generic way to send datachannels message on webrtc leg via API
A generic way to receive datachannels messages from webrtc leg via API
My use case is to connect LLMs to ongoing phone calls. LLMs like Gemini & OpenAI support webrtc, and there’s consensus that WebRTC is the most convenient & open-source way of transporting realtime media with LLMs. Janus’ SIP plugin seems like the perfect bridge between LLMs & calls.
No thoughts for now, as I’m not convinced yet. It makes little sense to me to expose an API to send messages via datachannel to “yourself” (as in theory who controls the signalling is the WebRTC peer). I do realize this is a different scenario, where there’s probably an application server doing the Janus API part to simply bridge the WebRTC call to something else, rather than dealing it directly as a browser would do, but I have to think of how people would use this in general, rather than vendor-specific cases. Last year for instance I talked a bit about how data channels could help in the SIP plugin too, but in that case that was in service to applications on the SIP side: WebRTC and SIP not just audio and video @ OpenSIPS 2024 | PPT.