Hi,
in my first message you can find many informations...
The only TLS available is TLS 1.2 and the openssl version is OpenSSL 1.0.2k-fips (the last one of RedHat 7.9), we have this random problem only from a customer that is using .net. In my opinion it should be a client problem but hard to demonstrate.
Which specific directives do you want to see of httpd-ssl.conf?
Matteo
-----Original Message-----
From: Daniel Ferradal <dferradal@apache.org>
Sent: domenica 23 maggio 2021 20:49
To: <users@httpd.apache.org> <users@httpd.apache.org>
Subject: [External] Re: [users@httpd] Struggling with "decryption failed or bad record mac" error
This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments.
Hello,
Perhaps you may provide more info. Like the openssl version you are using, your SSL related directives in your server, the openssl version or SSL version of the client, the protocol the client is trying to use.
Also, is this happening with all clients? just one?
Can you reproduce it with "openssl s_client -connect" command? or even curl? etc.
El vie, 21 may 2021 a las 12:25, Piemonti, Matteo
(<matteo.piemonti@accenture.com.invalid>) escribió:
>
> Hi,
>
> we’re having a weird error on Apache httpd server that I can’t understand how to troubleshoot it and not clear to me if it is an our problem (apache http server) or a problem of the caller.
>
>
>
> We have actually this configuration:
>
>
>
> Server version: Apache/2.4.46 (Unix)
>
> Server built: May 13 2021 05:46:31
>
> Server's Module Magic Number: 20120211:93
>
> Server loaded: APR 1.6.5, APR-UTIL 1.6.1
>
> Compiled using: APR 1.6.5, APR-UTIL 1.6.1
>
> Architecture: 64-bit
>
> Server MPM: event
>
> threaded: yes (fixed thread count)
>
> forked: yes (variable process count)
>
> Server compiled with....
>
> -D APR_HAS_SENDFILE
>
> -D APR_HAS_MMAP
>
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>
> -D APR_USE_SYSVSEM_SERIALIZE
>
> -D APR_USE_PTHREAD_SERIALIZE
>
> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>
> -D APR_HAS_OTHER_CHILD
>
> -D AP_HAVE_RELIABLE_PIPED_LOGS
>
> -D DYNAMIC_MODULE_LIMIT=256
>
> -D HTTPD_ROOT="/data/apache2_frontend"
>
> -D SUEXEC_BIN="/data/apache2_frontend/bin/suexec"
>
> -D DEFAULT_PIDLOG="logs/httpd.pid"
>
> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>
> -D DEFAULT_ERRORLOG="logs/error_log"
>
> -D AP_TYPES_CONFIG_FILE="conf/mime.types"
>
> -D SERVER_CONFIG_FILE="conf/httpd.conf"
>
>
>
> The problem we have is that during ssl handshake we can see (only with debug or tcpdump) an “SSL Library Error: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac" into apache httpd error_log.
>
> No other logs are written into access_log.
>
> How is possible to troubleshoot it and understand where is the problem
> (caller? network? receiver?)
>
>
>
> Some logs from trace:
>
>
>
> [Wed May 12 17:52:04.134409 2021] [ssl:debug] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(1741): [client ip:port] AH02275:
> Certificate Verification, depth 2, CRL checking mode: none (0)
> [subject: CN=etc etc etc]
>
> [Wed May 12 17:52:04.134553 2021] [ssl:debug] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(1741): [client ip:port] AH02275:
> Certificate Verification, depth 1, CRL checking mode: none (0)
> [subject: CN=etc etc etc]
>
> [Wed May 12 17:52:04.134681 2021] [ssl:debug] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(1741): [client ip:port] AH02275:
> Certificate Verification, depth 0, CRL checking mode: none (0)
> [subject: CN=etc etc etc]
>
> [Wed May 12 17:52:04.134705 2021] [ssl:trace3] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(2192): [client ip:port] OpenSSL:
> Loop: SSLv3 read client certificate A
>
> [Wed May 12 17:52:04.138368 2021] [ssl:trace3] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(2192): [client ip:port] OpenSSL:
> Loop: SSLv3 read client key exchange A
>
> [Wed May 12 17:52:04.138492 2021] [ssl:trace3] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(2192): [client ip:port] OpenSSL:
> Loop: SSLv3 read certificate verify A
>
> [Wed May 12 17:52:04.138513 2021] [ssl:trace4] [pid 10532:tid
> 140112100849408] ssl_engine_io.c(2214): [client ip:port] OpenSSL: read
> 5/5 bytes from BIO#7f6e2000ff60 [mem: 7f6e2c06f653]
>
> [Wed May 12 17:52:04.138519 2021] [ssl:trace4] [pid 10532:tid
> 140112100849408] ssl_engine_io.c(2214): [client ip:port] OpenSSL: read
> 1/1 bytes from BIO#7f6e2000ff60 [mem: 7f6e2c06f658]
>
> [Wed May 12 17:52:04.138568 2021] [ssl:trace4] [pid 10532:tid
> 140112100849408] ssl_engine_io.c(2214): [client ip:port] OpenSSL: read
> 5/5 bytes from BIO#7f6e2000ff60 [mem: 7f6e2c06f653]
>
> [Wed May 12 17:52:04.138586 2021] [ssl:trace4] [pid 10532:tid
> 140112100849408] ssl_engine_io.c(2214): [client ip:port] OpenSSL: read
> 40/40 bytes from BIO#7f6e2000ff60 [mem: 7f6e2c06f658]
>
> [Wed May 12 17:52:04.138600 2021] [ssl:trace4] [pid 10532:tid
> 140112100849408] ssl_engine_io.c(2214): [client ip:port] OpenSSL:
> write 7/7 bytes to BIO#7f6e20010f50 [mem: 7f6e3c03f763]
>
> [Wed May 12 17:52:04.138607 2021] [core:trace6] [pid 10532:tid
> 140112100849408] core_filters.c(525): [client ip:port] will flush
> because of FLUSH bucket
>
> [Wed May 12 17:52:04.138639 2021] [ssl:trace4] [pid 10532:tid
> 140112100849408] ssl_engine_io.c(2214): [client ip:port] OpenSSL:
> write 7/7 bytes to BIO#7f6e20011d50 [mem: 7f6e20004950]
>
> [Wed May 12 17:52:04.138669 2021] [core:trace6] [pid 10532:tid
> 140112100849408] core_filters.c(525): [client ip:port] will flush
> because of FLUSH bucket
>
> [Wed May 12 17:52:04.138676 2021] [ssl:trace3] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(2202): [client ip:port] OpenSSL:
> Write: error
>
> [Wed May 12 17:52:04.138680 2021] [ssl:trace3] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(2221): [client ip:port] OpenSSL:
> Exit: error in error
>
> [Wed May 12 17:52:04.138690 2021] [ssl:info] [pid 10532:tid
> 140112100849408] [client ip:port] AH02008: SSL library error 1 in
> handshake (server server:port)
>
> [Wed May 12 17:52:04.138711 2021] [ssl:info] [pid 10532:tid
> 140112100849408] SSL Library Error: error:1408F119:SSL
> routines:SSL3_GET_RECORD:decryption failed or bad record mac
>
> [Wed May 12 17:52:04.138720 2021] [ssl:info] [pid 10532:tid
> 140112100849408] [client ip:port] AH01998: Connection closed to child
> 448 with abortive shutdown (server server:port)
>
>
>
>
>
> Thank you
>
>
>
> Matteo Piemonti
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
> ______________________________________________________________________
> ________________
>
> www.accenture.com
--
Daniel Ferradal
HTTPD Project
#httpd help at Libera.Chat
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
???????????????????????????????????????????????????????????????????????F?V?7V'67&?&R?R???âW6W'2?V?7V'67&?&T?GGB?6?R??&p?f?"FF?F????6????G2?R???âW6W'2?V??GGB?6?R??&p
in my first message you can find many informations...
The only TLS available is TLS 1.2 and the openssl version is OpenSSL 1.0.2k-fips (the last one of RedHat 7.9), we have this random problem only from a customer that is using .net. In my opinion it should be a client problem but hard to demonstrate.
Which specific directives do you want to see of httpd-ssl.conf?
Matteo
-----Original Message-----
From: Daniel Ferradal <dferradal@apache.org>
Sent: domenica 23 maggio 2021 20:49
To: <users@httpd.apache.org> <users@httpd.apache.org>
Subject: [External] Re: [users@httpd] Struggling with "decryption failed or bad record mac" error
This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments.
Hello,
Perhaps you may provide more info. Like the openssl version you are using, your SSL related directives in your server, the openssl version or SSL version of the client, the protocol the client is trying to use.
Also, is this happening with all clients? just one?
Can you reproduce it with "openssl s_client -connect" command? or even curl? etc.
El vie, 21 may 2021 a las 12:25, Piemonti, Matteo
(<matteo.piemonti@accenture.com.invalid>) escribió:
>
> Hi,
>
> we’re having a weird error on Apache httpd server that I can’t understand how to troubleshoot it and not clear to me if it is an our problem (apache http server) or a problem of the caller.
>
>
>
> We have actually this configuration:
>
>
>
> Server version: Apache/2.4.46 (Unix)
>
> Server built: May 13 2021 05:46:31
>
> Server's Module Magic Number: 20120211:93
>
> Server loaded: APR 1.6.5, APR-UTIL 1.6.1
>
> Compiled using: APR 1.6.5, APR-UTIL 1.6.1
>
> Architecture: 64-bit
>
> Server MPM: event
>
> threaded: yes (fixed thread count)
>
> forked: yes (variable process count)
>
> Server compiled with....
>
> -D APR_HAS_SENDFILE
>
> -D APR_HAS_MMAP
>
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>
> -D APR_USE_SYSVSEM_SERIALIZE
>
> -D APR_USE_PTHREAD_SERIALIZE
>
> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>
> -D APR_HAS_OTHER_CHILD
>
> -D AP_HAVE_RELIABLE_PIPED_LOGS
>
> -D DYNAMIC_MODULE_LIMIT=256
>
> -D HTTPD_ROOT="/data/apache2_frontend"
>
> -D SUEXEC_BIN="/data/apache2_frontend/bin/suexec"
>
> -D DEFAULT_PIDLOG="logs/httpd.pid"
>
> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>
> -D DEFAULT_ERRORLOG="logs/error_log"
>
> -D AP_TYPES_CONFIG_FILE="conf/mime.types"
>
> -D SERVER_CONFIG_FILE="conf/httpd.conf"
>
>
>
> The problem we have is that during ssl handshake we can see (only with debug or tcpdump) an “SSL Library Error: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac" into apache httpd error_log.
>
> No other logs are written into access_log.
>
> How is possible to troubleshoot it and understand where is the problem
> (caller? network? receiver?)
>
>
>
> Some logs from trace:
>
>
>
> [Wed May 12 17:52:04.134409 2021] [ssl:debug] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(1741): [client ip:port] AH02275:
> Certificate Verification, depth 2, CRL checking mode: none (0)
> [subject: CN=etc etc etc]
>
> [Wed May 12 17:52:04.134553 2021] [ssl:debug] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(1741): [client ip:port] AH02275:
> Certificate Verification, depth 1, CRL checking mode: none (0)
> [subject: CN=etc etc etc]
>
> [Wed May 12 17:52:04.134681 2021] [ssl:debug] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(1741): [client ip:port] AH02275:
> Certificate Verification, depth 0, CRL checking mode: none (0)
> [subject: CN=etc etc etc]
>
> [Wed May 12 17:52:04.134705 2021] [ssl:trace3] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(2192): [client ip:port] OpenSSL:
> Loop: SSLv3 read client certificate A
>
> [Wed May 12 17:52:04.138368 2021] [ssl:trace3] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(2192): [client ip:port] OpenSSL:
> Loop: SSLv3 read client key exchange A
>
> [Wed May 12 17:52:04.138492 2021] [ssl:trace3] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(2192): [client ip:port] OpenSSL:
> Loop: SSLv3 read certificate verify A
>
> [Wed May 12 17:52:04.138513 2021] [ssl:trace4] [pid 10532:tid
> 140112100849408] ssl_engine_io.c(2214): [client ip:port] OpenSSL: read
> 5/5 bytes from BIO#7f6e2000ff60 [mem: 7f6e2c06f653]
>
> [Wed May 12 17:52:04.138519 2021] [ssl:trace4] [pid 10532:tid
> 140112100849408] ssl_engine_io.c(2214): [client ip:port] OpenSSL: read
> 1/1 bytes from BIO#7f6e2000ff60 [mem: 7f6e2c06f658]
>
> [Wed May 12 17:52:04.138568 2021] [ssl:trace4] [pid 10532:tid
> 140112100849408] ssl_engine_io.c(2214): [client ip:port] OpenSSL: read
> 5/5 bytes from BIO#7f6e2000ff60 [mem: 7f6e2c06f653]
>
> [Wed May 12 17:52:04.138586 2021] [ssl:trace4] [pid 10532:tid
> 140112100849408] ssl_engine_io.c(2214): [client ip:port] OpenSSL: read
> 40/40 bytes from BIO#7f6e2000ff60 [mem: 7f6e2c06f658]
>
> [Wed May 12 17:52:04.138600 2021] [ssl:trace4] [pid 10532:tid
> 140112100849408] ssl_engine_io.c(2214): [client ip:port] OpenSSL:
> write 7/7 bytes to BIO#7f6e20010f50 [mem: 7f6e3c03f763]
>
> [Wed May 12 17:52:04.138607 2021] [core:trace6] [pid 10532:tid
> 140112100849408] core_filters.c(525): [client ip:port] will flush
> because of FLUSH bucket
>
> [Wed May 12 17:52:04.138639 2021] [ssl:trace4] [pid 10532:tid
> 140112100849408] ssl_engine_io.c(2214): [client ip:port] OpenSSL:
> write 7/7 bytes to BIO#7f6e20011d50 [mem: 7f6e20004950]
>
> [Wed May 12 17:52:04.138669 2021] [core:trace6] [pid 10532:tid
> 140112100849408] core_filters.c(525): [client ip:port] will flush
> because of FLUSH bucket
>
> [Wed May 12 17:52:04.138676 2021] [ssl:trace3] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(2202): [client ip:port] OpenSSL:
> Write: error
>
> [Wed May 12 17:52:04.138680 2021] [ssl:trace3] [pid 10532:tid
> 140112100849408] ssl_engine_kernel.c(2221): [client ip:port] OpenSSL:
> Exit: error in error
>
> [Wed May 12 17:52:04.138690 2021] [ssl:info] [pid 10532:tid
> 140112100849408] [client ip:port] AH02008: SSL library error 1 in
> handshake (server server:port)
>
> [Wed May 12 17:52:04.138711 2021] [ssl:info] [pid 10532:tid
> 140112100849408] SSL Library Error: error:1408F119:SSL
> routines:SSL3_GET_RECORD:decryption failed or bad record mac
>
> [Wed May 12 17:52:04.138720 2021] [ssl:info] [pid 10532:tid
> 140112100849408] [client ip:port] AH01998: Connection closed to child
> 448 with abortive shutdown (server server:port)
>
>
>
>
>
> Thank you
>
>
>
> Matteo Piemonti
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
> ______________________________________________________________________
> ________________
>
> www.accenture.com
--
Daniel Ferradal
HTTPD Project
#httpd help at Libera.Chat
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
???????????????????????????????????????????????????????????????????????F?V?7V'67&?&R?R???âW6W'2?V?7V'67&?&T?GGB?6?R??&p?f?"FF?F????6????G2?R???âW6W'2?V??GGB?6?R??&p