Thread limit (4194304 threads) exceeded. Dying

General
1

Hi there

Anyone else got this issue besides me? It seems I can’t even get an useful asan dump. I’m not able to figure out what could cause the process to create such an excessive number of threads to hit the kernel limit of 4 million. So weird, got it on 3 different instances of janus running the same build number, and really close as time:

Apr 26 15:20:40 janus9vm janus[1780]: Janus version: 1203 (1.2.3)
Apr 26 15:20:40 janus9vm janus[1780]: Janus commit: 263f1b0ac1d1a4c12c4fa18d675a85234a005118
Apr 26 15:20:40 janus9vm janus[1780]: Compiled on: Fri 26 Apr 2024 02:19:46 PM CEST
May 27 20:32:20 janus9vm janus[1780]: ==1780==AddressSanitizer: Thread limit (4194304 threads) exceeded. Dying.
May 27 20:32:20 janus9vm systemd[1]: janus.service: Main process exited, code=exited, status=1/FAILURE

Apr 26 12:30:01 janus7vm janus[1804]: Janus version: 1203 (1.2.3)
Apr 26 12:30:01 janus7vm janus[1804]: Janus commit: 263f1b0ac1d1a4c12c4fa18d675a85234a005118
Apr 26 12:30:01 janus7vm janus[1804]: Compiled on: Fri 26 Apr 2024 10:34:41 AM CEST
May 28 01:40:44 janus7vm janus[1804]: ==1804==AddressSanitizer: Thread limit (4194304 threads) exceeded. Dying.
May 28 01:40:44 janus7vm systemd[1]: janus.service: Main process exited, code=exited, status=1/FAILURE

Apr 26 15:33:57 janus2vm janus[1739]: Janus version: 1203 (1.2.3)
Apr 26 15:33:57 janus2vm janus[1739]: Janus commit: 263f1b0ac1d1a4c12c4fa18d675a85234a005118
Apr 26 15:33:57 janus2vm janus[1739]: Compiled on: Fri 26 Apr 2024 03:30:46 PM CEST
May 28 07:43:47 janus2vm janus[1739]: ==1739==AddressSanitizer: Thread limit (4194304 threads) exceeded. Dying.
May 28 07:43:48 janus2vm systemd[1]: janus.service: Main process exited, code=exited, status=1/FAILURE

I wonder if anyone else encountered this.

2

This may actually be AddressSanitizer, since it keeps track of every thing that was allocated to figure out leaks. It’s also the reason why when you use AddressSanitizer the memory keeps on growing even if you don’t leak anything (it keeps stuff in memory on its own). As such, you probably don’t have that many threads, but those many threads may have happened over the course of that instance lifetime. Remember you can limit the number of threads using event loops in the core (more info in janus.jcfg).

1 Like
3

I will try to play with the event_loops parameters in janus.jcfg and see how it plays out. Not sure at this moment if I should also enable the allow_loop_indication, but for now I will only enable the event_loops and test. Thanks for such a quick response @lorenzo !!

4

@lorenzo , do you think it might not be any bug whatsoever, but just the process dies because AddressSanitizer keeps in memory all the threads created by janus and when it reaches 4mil and hits the kernel limit it just dies? I was thinking and I think it might be the case. Because they started to die in about the same time, because they were compiled with libasan aproximately in the same time, so it makes sense they would hit the threads history in just about the same time. It’s a wild guess based on my logic (which may or may not be correct)

5

Yes that was my guess, and why I pointed out how AS works.