Hello @lorenzo
We are running some Videoroom based app and get following crash time to time:
g_mutex_clear() called on uninitialised or locked mutex
AddressSanitizer:DEADLYSIGNAL
=================================================================
==1==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7fb24b8a3898 bp 0x7fb24ba96e90 sp 0x7fb24011e140 T15)
==1==The signal is caused by a READ memory access.
==1==Hint: this fault was caused by a dereference of a high value address (see register values below). Dissassemble the provided pc to learn which register was used.
(janus:1): GLib-CRITICAL **: 15:24:46.902: g_hash_table_iter_next: assertion 'ri->version == ri->hash_table->version' failed
[4961190077698169] Negotiation update, checking what changed...
[2776306228127586] Negotiation update, checking what changed...
[4210725300649175] Negotiation update, checking what changed...
[4105025948236915] Negotiation update, checking what changed...
#0 0x7fb24b8a3898 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x28898)
#1 0x7fb24c2d3f62 in g_mutex_clear (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0xa4f62)
#2 0x7fb240150bce in janus_videoroom_publisher_stream_free plugins/janus_videoroom.c:2363
#3 0x7fb24014a890 in janus_videoroom_publisher_stream_unref plugins/janus_videoroom.c:2345
#4 0x7fb24c26f281 (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x40281)
#5 0x7fb24c26fee2 in g_hash_table_remove_all (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x40ee2)
#6 0x7fb24018b6c0 in janus_videoroom_hangup_media_internal plugins/janus_videoroom.c:8602
#7 0x7fb24018e5e0 in janus_videoroom_hangup_media plugins/janus_videoroom.c:8455
#8 0x7fb240237081 in janus_videoroom_handler plugins/janus_videoroom.c:10150
#9 0x7fb24c2b3a50 (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x84a50)
#10 0x7fb24b90fac2 (/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
#11 0x7fb24b9a0a03 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x125a03)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0x28898) in abort
Thread T15 created by T0 here:
[6070812113618180] Negotiation update, checking what changed...
[7524763982887299] Negotiation update, checking what changed...
#0 0x7fb24c664685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x7fb24c2d9504 (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0xaa504)
==1==ABORTING
Is it known issue?
Janus is running in Docker container and crash happens when about 7 participants is active in videoroom. Some participants have pretty bad internet connection so it can be a reason.
Also, we have some browser based clients (based on janus.js), Android clients based on our own client implementation (WebSockets transport), Cordova iOs client based on iosrtc plugin and janus.js. Both Android and iOs clients use patched Google WebRTC library… I don’t think it can be related but just in case…
Hello @atoppi, @lorenzo
On previous week we got another crash in videoroom
log
'“=================================================================”
[janus.plugin.videoroom-0x6040003dd090] No WebRTC media anymore
[831277794535105] WebRTC resources freed
'“==1==ERROR: AddressSanitizer: heap-use-after-free on address 0x60c0004c515c at pc 0x7f02fc3855de bp 0x7f02fc244220 sp 0x7f02fc244210”
WRITE of size 4 at 0x60c0004c515c thread T13
[8192100835266345] Handle and related resources freed
[janus.plugin.videoroom-0x6040004a0ed0] No WebRTC media anymore
[698466082305799] WebRTC resources freed
Detaching handle from JANUS VideoRoom plugin
[698466082305799] Handle and related resources freed
Detaching handle from JANUS VideoRoom plugin
[janus.plugin.videoroom-0x6040005a2590] No WebRTC media anymore
[74049221036384] WebRTC resources freed
[74049221036384] Handle and related resources freed #0 0x7f02fc3855dd in janus_videoroom_handler plugins/janus_videoroom.c:11679 #1 0x7f030720aa50 (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x84a50) #2 0x7f0306866ac2 (/lib/x86_64-linux-gnu/libc.so.6+0x94ac2) #3 0x7f03068f7a03 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x125a03)
0x60c0004c515c is located 92 bytes inside of 128-byte region [0x60c0004c5100,0x60c0004c5180)
freed by thread T21211 here: #0 0x7f0307617537 in __interceptor_free …/…/…/…/src/libsanitizer/asan/asan_malloc_linux.cpp:127 #1 0x7f02fc276d25 in janus_videoroom_subscriber_free plugins/janus_videoroom.c:2329 #2 0x7f02fc2b4374 in janus_videoroom_destroy_session plugins/janus_videoroom.c:4030 #3 0x55978ae4caa0 in janus_ice_outgoing_traffic_handle /tmp/janus-gateway/src/ice.c:4620 #4 0x55978ae578a3 in janus_ice_outgoing_traffic_dispatch /tmp/janus-gateway/src/ice.c:551 #5 0x7f03071dbd3a in g_main_context_dispatch (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x55d3a)
previously allocated by thread T13 here: #0 0x7f0307617a57 in __interceptor_calloc …/…/…/…/src/libsanitizer/asan/asan_malloc_linux.cpp:154 #1 0x7f03071e4c50 in g_malloc0 (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x5ec50) #2 0x7f030720aa50 (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x84a50)
Thread T13 created by T0 here:
[janus.plugin.videoroom-0x60400034f790] No WebRTC media anymore
Detaching handle from JANUS VideoRoom plugin
[601684311122291] WebRTC resources freed #0 0x7f03075bb685 in __interceptor_pthread_create …/…/…/…/src/libsanitizer/asan/asan_interceptors.cpp:216 #1 0x7f0307230504 (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0xaa504)
Thread T21211 created by T8 here: #0 0x7f03075bb685 in __interceptor_pthread_create …/…/…/…/src/libsanitizer/asan/asan_interceptors.cpp:216 #1 0x7f0307230504 (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0xaa504)
Thread T8 created by T0 here: #0 0x7f03075bb685 in __interceptor_pthread_create …/…/…/…/src/libsanitizer/asan/asan_interceptors.cpp:216 #1 0x7f0307230504 (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0xaa504)
SUMMARY: AddressSanitizer: heap-use-after-free plugins/janus_videoroom.c:11679 in janus_videoroom_handler
Shadow bytes around the buggy address:
0x0c18800909d0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c18800909e0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
0x0c18800909f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
0x0c1880090a00: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c1880090a10: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
'“=>0x0c1880090a20: fd fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd”
0x0c1880090a30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c1880090a40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c1880090a50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c1880090a60: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c1880090a70: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
'“==1==ABORTING”
Unfortunately it was running from v1.2.1 with #3331 backported, not from master. So I didn’t create issue on GitHub since it can be something already fixed (Is it?).
Crash was happen when multiple participants was disconnecting from VideoRoom almost at the same time. Some of them had pretty bad internet quality.
Unfortunately It’s impossibile to confirm if this is something that has already been fixed.
In any case, as you already know, we need to confirm that this happens on a recent tag too, then you can proceed opening an issue providing useful debugging data with matching source lines.
