tryton-server not accepting new connections
I regularly have to restart tryton-server to be able to connect to it with tryton-client. I'm currently running tryton-server 5.0.39-1~11bullseye+1 on a Debian 11 system.
When I don't have the issue, the trytond process is constantly polling the various file descriptors. When I have the issue, the trytond process is stuck trying to read a socket:
root@helios:~# strace -p 160721
strace: Process 160721 attached
read(5, ^Cstrace: Process 160721 detached
<detached ...>
root@helios:~# ls -al /proc/160721/fd
total 0
dr-x------ 2 tryton tryton 0 Oct 29 07:00 .
dr-xr-xr-x 9 tryton tryton 0 Oct 29 06:56 ..
lr-x------ 1 tryton tryton 64 Oct 29 09:29 0 -> /dev/null
lrwx------ 1 tryton tryton 64 Oct 29 09:29 1 -> 'socket:[91047185]'
lrwx------ 1 tryton tryton 64 Oct 29 09:29 2 -> 'socket:[91047185]'
l-wx------ 1 tryton tryton 64 Oct 29 09:29 3 -> /var/log/tryton/trytond.log
lrwx------ 1 tryton tryton 64 Oct 29 09:29 4 -> 'socket:[91047195]'
lrwx------ 1 tryton tryton 64 Oct 29 09:29 5 -> 'socket:[96288386]'
lrwx------ 1 tryton tryton 64 Oct 29 09:29 6 -> 'socket:[93797060]'
Looking at the connections, we see some data that is waiting to be processed? And we see recent connections from random IP, likely bots looking for things to exploit. I'm assuming that those requests are confusing something in tryton which then gets stuck. I'm using the embedded server and I have not setup an external WSGI handler.
root@helios:~# netstat -tupan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
[...]
tcp6 20 0 :::8000 :::* LISTEN 160721/python3
[...]
tcp6 109 0 195.154.119.178:8000 103.143.197.10:42810 CLOSE_WAIT -
tcp6 518 0 2001:bc8:6005:136::8000 2a01:e0a:435:20d0:44986 CLOSE_WAIT -
tcp6 57 0 195.154.119.178:8000 103.131.90.74:56812 CLOSE_WAIT -
tcp6 5 0 195.154.119.178:8000 154.86.16.143:56130 CLOSE_WAIT -
tcp6 518 0 2001:bc8:6005:136::8000 2a01:e0a:435:20d0:44992 CLOSE_WAIT -
tcp6 113 0 195.154.119.178:8000 95.72.73.228:41228 CLOSE_WAIT -
tcp6 1 0 195.154.119.178:8000 154.86.16.143:53128 CLOSE_WAIT -
tcp6 518 0 2001:bc8:6005:136::8000 2a01:e0a:435:20d0:44990 CLOSE_WAIT -
tcp6 42 0 195.154.119.178:8000 154.86.16.143:58346 CLOSE_WAIT -
tcp6 153 0 195.154.119.178:8000 35.176.126.200:59784 CLOSE_WAIT -
tcp6 153 0 195.154.119.178:8000 154.86.16.143:50740 CLOSE_WAIT -
tcp6 0 0 ::1:49098 ::1:80 TIME_WAIT -
tcp6 518 0 2001:bc8:6005:136::8000 2a01:e0a:435:20d0:44988 CLOSE_WAIT -
tcp6 0 0 195.154.119.178:8000 185.184.233.85:1253 ESTABLISHED 160721/python3
tcp6 14 0 195.154.119.178:8000 154.86.16.143:58410 CLOSE_WAIT -
tcp6 147 0 195.154.119.178:8000 192.241.198.97:42784 CLOSE_WAIT -
tcp6 13 0 195.154.119.178:8000 154.86.16.143:58216 CLOSE_WAIT -
tcp6 69 0 195.154.119.178:8000 154.86.16.143:57766 CLOSE_WAIT -
In trytond.conf I have those settings that are likely relevant for this issue:
[web]
listen = [::]:8000
[ssl]
privatekey = /var/lib/dehydrated/certs/tryton.freexian.com/privkey.pem
certificate = /var/lib/dehydrated/certs/tryton.freexian.com/fullchain.pem
I have enabled DEBUG logging in /etc/tryton/trytond_log.conf now and I will share the output next time I get hit by this blocking behaviour. But right now I have not seen anything suspicious in the log file.