Issue 10921

Title
tryton-server not accepting new connections
Priority
bug
Status
need-eg
Nosy list
ced, rhertzog, yangoon
Assigned to
Keywords

Created on 2021-10-29.11:55:54 by rhertzog, last changed 1 month ago by ced.

Messages

Author: [hidden] (ced) Tryton committer Tryton translator
Date: 2021-10-29.13:08:13

The werkzeug simple server is not intended to be used in production: https://werkzeug.palletsprojects.com/en/2.0.x/serving/.
So for me you should run it at least behind a reverse proxy and the best would be inside a production-ready WSGI server.
If you still have trouble with such setup than it may be a Tryton's issue.

Author: [hidden] (rhertzog)
Date: 2021-10-29.11:55:54

I regularly have to restart tryton-server to be able to connect to it with tryton-client. I'm currently running tryton-server 5.0.39-1~11bullseye+1 on a Debian 11 system.

When I don't have the issue, the trytond process is constantly polling the various file descriptors. When I have the issue, the trytond process is stuck trying to read a socket:

<root@helios>:~# strace -p 160721
strace: Process 160721 attached
read(5, ^Cstrace: Process 160721 detached
 &lt;detached ...>

<root@helios>:~# ls -al /proc/160721/fd
total 0
dr-x------ 2 tryton tryton  0 Oct 29 07:00 .
dr-xr-xr-x 9 tryton tryton  0 Oct 29 06:56 ..
lr-x------ 1 tryton tryton 64 Oct 29 09:29 0 -> /dev/null
lrwx------ 1 tryton tryton 64 Oct 29 09:29 1 -> 'socket:[91047185]'
lrwx------ 1 tryton tryton 64 Oct 29 09:29 2 -> 'socket:[91047185]'
l-wx------ 1 tryton tryton 64 Oct 29 09:29 3 -> /var/log/tryton/trytond.log
lrwx------ 1 tryton tryton 64 Oct 29 09:29 4 -> 'socket:[91047195]'
lrwx------ 1 tryton tryton 64 Oct 29 09:29 5 -> 'socket:[96288386]'
lrwx------ 1 tryton tryton 64 Oct 29 09:29 6 -> 'socket:[93797060]'

Looking at the connections, we see some data that is waiting to be processed? And we see recent connections from random IP, likely bots looking for things to exploit. I'm assuming that those requests are confusing something in tryton which then gets stuck. I'm using the embedded server and I have not setup an external WSGI handler.

<root@helios>:~# netstat -tupan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
[...]
tcp6      20      0 :::8000                 :::*                    LISTEN      160721/python3      
[...]
tcp6     109      0 195.154.119.178:8000    103.143.197.10:42810    CLOSE_WAIT  -                   
tcp6     518      0 2001:bc8:6005:136::8000 2a01:e0a:435:20d0:44986 CLOSE_WAIT  -                   
tcp6      57      0 195.154.119.178:8000    103.131.90.74:56812     CLOSE_WAIT  -                   
tcp6       5      0 195.154.119.178:8000    154.86.16.143:56130     CLOSE_WAIT  -                   
tcp6     518      0 2001:bc8:6005:136::8000 2a01:e0a:435:20d0:44992 CLOSE_WAIT  -                   
tcp6     113      0 195.154.119.178:8000    95.72.73.228:41228      CLOSE_WAIT  -                   
tcp6       1      0 195.154.119.178:8000    154.86.16.143:53128     CLOSE_WAIT  -                   
tcp6     518      0 2001:bc8:6005:136::8000 2a01:e0a:435:20d0:44990 CLOSE_WAIT  -                   
tcp6      42      0 195.154.119.178:8000    154.86.16.143:58346     CLOSE_WAIT  -                   
tcp6     153      0 195.154.119.178:8000    35.176.126.200:59784    CLOSE_WAIT  -                   
tcp6     153      0 195.154.119.178:8000    154.86.16.143:50740     CLOSE_WAIT  -                   
tcp6       0      0 ::1:49098               ::1:80                  TIME_WAIT   -                   
tcp6     518      0 2001:bc8:6005:136::8000 2a01:e0a:435:20d0:44988 CLOSE_WAIT  -                   
tcp6       0      0 195.154.119.178:8000    185.184.233.85:1253     ESTABLISHED 160721/python3      
tcp6      14      0 195.154.119.178:8000    154.86.16.143:58410     CLOSE_WAIT  -                   
tcp6     147      0 195.154.119.178:8000    192.241.198.97:42784    CLOSE_WAIT  -                   
tcp6      13      0 195.154.119.178:8000    154.86.16.143:58216     CLOSE_WAIT  -                   
tcp6      69      0 195.154.119.178:8000    154.86.16.143:57766     CLOSE_WAIT  -                   

In trytond.conf I have those settings that are likely relevant for this issue:

[web]
listen = [::]:8000
[ssl]
privatekey = /var/lib/dehydrated/certs/tryton.freexian.com/privkey.pem
certificate = /var/lib/dehydrated/certs/tryton.freexian.com/fullchain.pem

I have enabled DEBUG logging in /etc/tryton/trytond_log.conf now and I will share the output next time I get hit by this blocking behaviour. But right now I have not seen anything suspicious in the log file.

History
Date User Action Args
2021-10-29 13:08:13cedsetmessages: + msg71387
nosy: + ced
status: unread -> need-eg
2021-10-29 12:26:50yangoonsetnosy: + yangoon
2021-10-29 11:55:54rhertzogcreate