Hi, we have problems with stale remote publishers.
We have following situation:
we have server1 and server2
application on server1 sending add_remote_publisher to server2
server2 handles and responses with publish_remotely
then server1 reboots abnormally(e.g. janus segfaults or server hard reboots)
server2 will have publisher sitting in room forever
or the opposite server2 reboots server1 will have running remotization forever.
Is it possible to implement some sort of keepalive for remote publishers to avoid such situations(e.g. like keepalive for janus sessions)?
We can implement this in our server application logic, but it is very fragile and leads us to have large state in application to keep it up - we have to monitor remotizations and publishers lists and compare it with remote state to mantain this mechanism working.
Remote publishers management is done via synchronous requests, a bit like creating rooms. Exactly as rooms don’t have a keepalive (and in the past some have asked for it for reasons similar to the ones you described), I’m not sure remote publishers should have one either, especially considering we don’t have keepalives for RTP forwarders either, which are similar in that sense. In fact, while I guess it may be relatively easy to have a keepalive for a remote publisher recipient (since there’s a dedicated thread for the network loop that could have an additional check for self cleanup), this wouldn’t be easy or possible for a remotized publisher: when and how do you decide you should stop sending RTP packets to a remote publisher whose server died?
For these reasons I think it’s easier to do this from the orchestration layer.
For doing orchestration, in case we lost all state, we need to get list of all publishers and check if remote publisher exists on destination server and this should be done periodically(e.g. remote server is restarted). And on destination server we also need to do some checks - list all remotes for every publisher in the room and check if source still have publisher for it. It isn’t easy task to do and leads us to have duplication in state and additional memory/cpu usage for orchestration.
I realized two mechanizms to ensure remotization is working:
we have to call add_remote_publisher from some session and handle. If this handle is detached or session were closed by keepalive timeout we could cleanup all remote publishers/remotizations what were created from this session.
We can have last_keepalive value for remote publisher session and for janus_videoroom_remote_recipient and add new api calls to keepalive_remote_publisher and keepalive_remote_recipient (naming could be any) - this could be called by orchestration layer.
So now have:
JanusSourceSession on server1 and JanusDestinationSession on server2, in our case this is Elixir processes with state keeping reference to ws janus client, session_id and handle_id
on source server JanusSourceSession pediodically sends keepalive to janus and stays alive. It used for add_remote_publisher/update_remote_publisher/remove_remote_publisher calls. So if this session isn’t alive anymore we could unpublish it by janus automatically.
on destination server JanusDestinationSession used to call publish_remotely/unpublish_remotely and also if session isn’t alive we could remove remotizations which were created through it.
Also JanusSourceSession could send periodical keepalives to JanusDestinationSession via external api of our orchestration and based on these keepalives send keepalive_remote_publisher/keepalive_remote_recipient to janus to update last_keepalive value.
Btw In case if remote publishers were implemented using ice/dtls we could use DTLS hangup events from webrtc to unpublish/recreate remote publishers, but now we don’t have mechanism to detect if something wrong with remotization.