Hi,
we're using rsyslog to send logs to ES using omelasticsearch
One issue we're having is that from time to time, after a network issue,
the queue for messages going to ES keeps growing, We are writing to
multiple indices in the same cluster, ant this issue is only happening for
some of the indices (and not always the same ones).
What I found so far:
Dropping traffic from ES is a relatively reliable way to trigger the issue:
iptables -A INPUT -p tcp --source <ES proxy IP> --sport 9200 -j DROP ;
sleep 180 ; iptables -D INPUT -p tcp --source <ES proxy IP> --sport 9200 -j
DROP
Force-closing ES sockets in the worker (via GDB) fixes the issue (queue :
for FD in `lsof -p $PID -n -a -i TCP:wap-wsp -a -s TCP:ESTABLISHED -F f |
grep '^f' | cut -d f -f 2`; do gdb -batch -ex 'set logging on' -ex 'set
logging redirect on' -ex "attach $PID" -ex "call shutdown($FD, 1)" -ex
'detach' -ex 'quit'; done
My current theory is that, one or more libcurl requests gets in a state
where rsyslog has sent a request to ES, ES has tried to send the answer
back to rsyslog for some time, but then gave up and closed the connection,
and since there is no timeout on the request, omelasticsearch is stuck
forever.
But of course I might be dead wrong and there is a simple explanation.
I wanted to try to set a timeout on the ES request, to see if this fixes
the issue, but first I wanted to ask if there is a specific reason why
there is no option to set the timeout on the ES request, or it just was not
implemented at the time?
Kind regards,
Mattia
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
we're using rsyslog to send logs to ES using omelasticsearch
One issue we're having is that from time to time, after a network issue,
the queue for messages going to ES keeps growing, We are writing to
multiple indices in the same cluster, ant this issue is only happening for
some of the indices (and not always the same ones).
What I found so far:
Dropping traffic from ES is a relatively reliable way to trigger the issue:
iptables -A INPUT -p tcp --source <ES proxy IP> --sport 9200 -j DROP ;
sleep 180 ; iptables -D INPUT -p tcp --source <ES proxy IP> --sport 9200 -j
DROP
Force-closing ES sockets in the worker (via GDB) fixes the issue (queue :
for FD in `lsof -p $PID -n -a -i TCP:wap-wsp -a -s TCP:ESTABLISHED -F f |
grep '^f' | cut -d f -f 2`; do gdb -batch -ex 'set logging on' -ex 'set
logging redirect on' -ex "attach $PID" -ex "call shutdown($FD, 1)" -ex
'detach' -ex 'quit'; done
My current theory is that, one or more libcurl requests gets in a state
where rsyslog has sent a request to ES, ES has tried to send the answer
back to rsyslog for some time, but then gave up and closed the connection,
and since there is no timeout on the request, omelasticsearch is stuck
forever.
But of course I might be dead wrong and there is a simple explanation.
I wanted to try to set a timeout on the ES request, to see if this fixes
the issue, but first I wanted to ask if there is a specific reason why
there is no option to set the timeout on the ES request, or it just was not
implemented at the time?
Kind regards,
Mattia
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.