Mailing List Archive: Bug - Strange issue with mod_perl 2.0.10 / Apache 2.4 corrupting nfreeze data

Bug - Strange issue with mod_perl 2.0.10 / Apache 2.4 corrupting nfreeze data

Apr 3, 2016, 6:11 AM

Post #1 of 6 (3455 views)

-------------8<---------- Start Bug Report ------------8<----------
1. Problem Description:

I'm running Perl 5.20.2 on a Gentoo virtual machine. We've had some
code in place that uses a mod_perl handler to send complex hashref
based structures between two servers using nfreeze / thaw; the code
needed to be high performance therefore we used nfreeze/thaw rather
than XML or similar. This has been working correctly on a variety of
architectures since July 2014.

The mod_perl server does:

print nfreeze( $message_hash )

The client connects to the server use LWP::UserAgent and does:

my $message = thaw( $response->decoded_content() );

The message from client to server is similarly encoded with nfreeze /
thaw in a cgi parameter in multi-part-form-data.

This has all worked perfectly up until I upgraded to Apache 2.4 /
mod_perl 2.0.10 - I've even downgraded just those two packages to
mod_perl 2.0.8 / Apache 2.2.31 and it starts working again. The message
from the client gets to the server correctly and the server logs and
processes the message correctly, but it then fails with an 'Out of
memory!' error from the client when the response is received (due to
this bug in Storable.pm) - however, regardless of that bug, something
is corrupting the data in transit...

I've picked apart what's going on and found there are an additional two
bytes being inserted somehow. I've recorded the data in a /tmp/ file
before the print statement in the mod_perl handler and I've used perl
-d to inspect what's received:

DB<3> say "0x$_" for unpack "(H2)*", $response->content()
0x05
0x0a
0x03
0x00
0x00
0x00
0x01
0x08
0xc2
0x81
0x00
0x00
0x00
0x02
0x6f
0x6b

However, the server sent:

0x05
0x0a
0x03
0x00
0x00
0x00
0x01
0x08
0x00
0x00
0x00
0x02
0x6f
0x6b

So somehow 2 bytes have appeared in the middle of the message 0x08 and
0xc2.

I've also tried adding Content-Type => 'x-application/binary' and
Content-Length headers to see if that fixes the issue.

Thanks!

2. Used Components and their Configuration:

*** mod_perl version 2.000010

*** using /usr/lib64/perl5/vendor_perl/5.20.2/x86_64-linux-thread-
multi/Apache2/BuildConfig.pm

*** Makefile.PL options:
MP_APR_CONFIG  => /usr/bin/apr-1-config
MP_APR_LIB     => aprext
MP_APXS        => /usr/sbin/apxs2
MP_COMPAT_1X   => 1
MP_DEBUG       => 0
MP_GENERATE_XS => 1
MP_LIBNAME     => mod_perl
MP_TRACE       => 0
MP_USE_DSO     => 1

*** The httpd binary was not found

*** (apr|apu)-config linking info

-laprutil-1 -lldap -llber -ldb-4.8 -lgdbm  -lexpat
-L/var/tmp/portage/dev-libs/apr-util-1.5.4/temp
-lapr-1 -luuid -lrt -lcrypt  -lpthread -ldl

*** /usr/bin/perl -V
Summary of my perl5 (revision 5 version 20 subversion 2) configuration:

Platform:
    osname=linux, osvers=2.6.32-openvz-feoktistov.1, archname=x86_64-
linux-thread-multi
    uname='linux alex.zednax.com 2.6.32-openvz-feoktistov.1 #2 smp wed
apr 6 14:38:37 bst 2011 x86_64 intel(r) xeon(r) cpu l5630 @ 2.13ghz
genuineintel gnulinux '
    config_args='-des -Duseshrplib -Darchname=x86_64-linux-thread
-Dcc=x86_64-pc-linux-gnu-gcc -Doptimize=-O -pipe -Dldflags=-Wl,-O1
-Wl,--as-needed -Dprefix=/usr -Dinstallprefix=/usr
-Dsiteprefix=/usr/local -Dvendorprefix=/usr -Dscriptdir=/usr/bin
-Dprivlib=/usr/lib64/perl5/5.20.2
-Darchlib=/usr/lib64/perl5/5.20.2/x86_64-linux-thread-multi
-Dsitelib=/usr/local/lib64/perl5/5.20.2
-Dsitearch=/usr/local/lib64/perl5/5.20.2/x86_64-linux-thread-multi
-Dvendorlib=/usr/lib64/perl5/vendor_perl/5.20.2
-Dvendorarch=/usr/lib64/perl5/vendor_perl/5.20.2/x86_64-linux-thread-
multi -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3
-Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3
-Dvendorman1dir=/usr/share/man/man1 -Dvendorman3dir=/usr/share/man/man3
-Dman1ext=1 -Dman3ext=3pm -Dlibperl=libperl.so.5.20.2
-Dlocincpth=/usr/include  -Dglibpth=/lib64 /usr/lib64  -Duselargefiles
-Dd_semctl_semun -Dcf_by=Gentoo -Dmyhostname=localhost -Dperladmin=root
@localhost -Dinstallusrbinperl=n -Ud_csh -Uusenm -Di_ndbm -Di_gdbm
-Di_db -Dusethreads -DDEBUGGING=none -Dinc_version_list=5.20.0/x86_64-
linux-thread-multi 5.20.0 5.20.1/x86_64-linux-thread-multi 5.20.1  -
Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Dnoextensions=ODBM_File'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
Compiler:
    cc='x86_64-pc-linux-gnu-gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE
-fwrapv -fno-strict-aliasing -pipe -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
    optimize='-O -pipe',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing
-pipe'
    ccversion='', gccversion='4.7.3', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=8, prototype=define
Linker and Libraries:
    ld='x86_64-pc-linux-gnu-gcc', ldflags ='-Wl,-O1 -Wl,--as-needed'
    libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/lib/gcc/x86_64-pc-
linux-gnu/4.7.3/include-fixed /usr/lib /lib/../lib64 /usr/lib/../lib64
/lib
    libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
-lgdbm_compat
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=libc-2.21.so, so=so, useshrplib=true,
libperl=libperl.so.5.20.2
    gnulibc_version='2.21'
Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O -pipe -Wl,-O1 -Wl,--as-
needed'

Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
                        PERL_DONT_CREATE_GVSV
                        PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
                        PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
                        PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
                        USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS
                        USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
                        USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_PERLIO
                        USE_PERL_ATOF USE_REENTRANT_API
Locally applied patches:
gentoo/hints_hpux - Fix hpux hints
gentoo/aix_soname - aix gcc detection and shared library soname
support
gentoo/EUMM-RUNPATH - https://bugs.gentoo.org/105054
cpan/ExtUtils-MakeMaker: drop $PORTAGE_TMPDIR from LD_RUN_PATH
gentoo/config_over - Remove -rpath and append LDFLAGS to
lddlflags
gentoo/opensolaris_headers - [PATCH] Add headers for
opensolaris
gentoo/patchlevel - List packaged patches for perl-5.20.2(#1)
in patchlevel.h
gentoo/cpanplus_definstalldirs - Configure CPANPLUS to use the
site directories by default.
gentoo/cleanup-paths - [PATCH] Cleanup PATH and shrpenv
gentoo/enc2xs - Tweak enc2xs to follow symlinks and ignore
missing @INC directories.
gentoo/enc2xs_checksums -
gentoo/darwin-cc-ld - https://bugs.gentoo.org/297751 [PATCH]
darwin: Use $CC to link
gentoo/cpan_definstalldirs - Provide a sensible INSTALLDIRS
default for modules installed from CPAN.
gentoo/interix - [PATCH] Fix interix hints
gentoo/create_libperl_soname - https://bugs.gentoo.org/286840
[PATCH] Set libperl soname
gentoo/mod_paths - Add /etc/perl to @INC
gentoo/EUMM_delete_packlist -
gentoo/drop_fstack_protector - https://bugs.gentoo.org/348557
[PATCH] Don't force -fstack-protector on everyone
gentoo/usr_local - [PATCH] Remove /usr/local paths
gentoo/D-SHA-CFLAGS - https://bugs.gentoo.org/506818 [PATCH] Do
not set custom CFLAGS in cpan/Digest-SHA
gentoo/io_socket_ip_tests -
debian/cpan-missing-site-dirs - Fix CPAN::FirstTime defaults
with nonexisting site dirs if a parent is writable
debian/regcomp-mips-optim - Downgrade the optimization of
regcomp.c on mips and mipsel due to a gcc-4.9 bug
debian/perldoc-less-R - Tell the 'less' pager to allow terminal
escape sequences
debian/makemaker-pasthru - Pass LD settings through to
subdirectories
fixes/net_smtp_docs - [rt.cpan.org #36038] Document the
Net::SMTP 'Port' option
fixes/memoize_storable_nstore - [rt.cpan.org #77790]
Memoize::Storable: respect 'nstore' option not respected
fixes/document_makemaker_ccflags - [rt.cpan.org #68613]
Document that CCFLAGS should include $Config{ccflags}
Built under linux
Compiled at Mar 27 2016 10:44:32
%ENV:
    PERL_LWP_USE_HTTP_10="1"
@INC:
    /etc/perl
    /usr/local/lib64/perl5/5.20.2/x86_64-linux-thread-multi
    /usr/local/lib64/perl5/5.20.2
    /usr/lib64/perl5/vendor_perl/5.20.2/x86_64-linux-thread-multi
    /usr/lib64/perl5/vendor_perl/5.20.2
    /usr/local/lib64/perl5
    /usr/lib64/perl5/vendor_perl
    /usr/lib64/perl5/5.20.2/x86_64-linux-thread-multi
    /usr/lib64/perl5/5.20.2
    .

*** Packages of interest status:

Apache2            : -
Apache2::Request   : 2.13
CGI                : 3.65, 4.21, 4.22
ExtUtils::MakeMaker: 6.98
LWP                : 6.15
mod_perl           : -
mod_perl2          : 2.000010

3. This is the core dump trace: (if you get a core dump):

[CORE TRACE COMES HERE]

This report was generated by /usr/bin/mp2bug on Sun Apr  3 13:00:59
2016 GMT.

-------------8<---------- End Bug Report --------------8<----------

Note: Complete the rest of the details and post this bug report to
modperl <at> perl.apache.org. To subscribe to the list send an empty
email to modperl-subscribe@perl.apache.org.

Re: Bug - Strange issue with mod_perl 2.0.10 / Apache 2.4 corrupting nfreeze data [ In reply to ]

vv.lists at wanadoo

Apr 3, 2016, 8:03 AM

Post #2 of 6 (3401 views)

Permalink

On Sun, 03 Apr 2016 14:11:23 +0100
Alex Masidlover <alex.masidlover@zednax.com> wrote:

>
> This has all worked perfectly up until I upgraded to Apache 2.4 /
> mod_perl 2.0.10 -

After upgrading to Apache 2.4 and mod_perl 2.0.9, I had to make those two changes to my application :

In a PerlOutputFilterHandler, change '$content .= $buffer' to '$content .= decode_utf8($buffer)'

And in response handlers, change '$args{$_} = $req->param($_)' to '$args{$_} = decode_utf8($req->param($_))'

Not sure it applies to your case, but something changed in Apache 2.4 concerning UTF-8 data.

If I understood correctly, anything that goes through APR::Table is considered UTF-8, however the SvUTF8 flag is not set, so you get double encoding when processing your data.

--
Bien à vous, Vincent Veyron

https://marica.fr/
Gestion des contentieux, des dossiers de sinistres assurance et des contrats pour le service juridique

Re: Bug - Strange issue with mod_perl 2.0.10 / Apache 2.4 corrupting nfreeze data [ In reply to ]

alex.masidlover at zednax

Aug 7, 2016, 6:37 AM

Post #3 of 6 (3171 views)

Permalink

On Sun, 2016-04-03 at 17:03 +0200, Vincent Veyron wrote:
> On Sun, 03 Apr 2016 14:11:23 +0100
> Alex Masidlover <alex.masidlover@zednax.com> wrote:
>
> >
> >
> > This has all worked perfectly up until I upgraded to Apache 2.4 /
> > mod_perl 2.0.10 -
> After upgrading to Apache 2.4 and mod_perl 2.0.9, I had to make those
> two changes to my application :
>
> In a PerlOutputFilterHandler, change '$content .= $buffer' to
> '$content .= decode_utf8($buffer)'
>
> And in response handlers, change '$args{$_} = $req->param($_)' to
> '$args{$_} = decode_utf8($req->param($_))'
>
> Not sure it applies to your case, but something changed in Apache 2.4
> concerning UTF-8 data.
>
> If I understood correctly, anything that goes through APR::Table is
> considered UTF-8, however the SvUTF8 flag is not set, so you get
> double encoding when processing your data.
>

Thanks for the suggestion - its taken a while but I've manage to free
up a machine to roll forwards and test on; unfortunately I've got no
experience with Apache and filters. I've come up with this:

package Zymonic::Decryptor::Filter;
use strict;

use base qw(Apache2::Filter);
use Encode qw(decode);
use constant BUFF_LEN => 1024;
use Apache2::Const -compile => 'OK';

sub utf8_filter : FilterRequestHandler {

my $f = shift;

my $content = '';
while($f->read(my $buffer, BUFF_LEN)) {
$content .= decode_utf8($buffer);
}

$f->print($content);

return Apache2::Const::OK;

}

but when I add it with:

PerlOutputFilterHandler Zymonic::Decryptor::Filter::utf8_filter

I get:

:Apache2 IO flush: (500) Unknown error 500 at -e line 0

in the Apache error log whenever I hit the server with a request.

I'm guessing I've done something very wrong with the filter, but I can
find very few examples of how to use Apache2::Filter.

Thanks,

Alex

--
Technical Director - Zednax Limited
W: http://www.zednax.com
T: +44 333 444 0160
F: +44 161 660 8010

Zednax Limited is registered in England and Wales, Company no. 05321754.
Registered address: Meadow House, Meadow Lane, Nottingham, NG2 3HS.
Zednax Limited is VAT registered, VAT registration no. GB 855 4468 92.

Re: Bug - Strange issue with mod_perl 2.0.10 / Apache 2.4 corrupting nfreeze data [ In reply to ]

wrowe at rowe-clan

Aug 7, 2016, 8:45 AM

Post #4 of 6 (3171 views)

Permalink

On Sun, Aug 7, 2016 at 8:37 AM, Alex Masidlover <alex.masidlover@zednax.com>
wrote:

> On Sun, 2016-04-03 at 17:03 +0200, Vincent Veyron wrote:
> > On Sun, 03 Apr 2016 14:11:23 +0100
> > Alex Masidlover <alex.masidlover@zednax.com> wrote:
> >
> > > This has all worked perfectly up until I upgraded to Apache 2.4 /
> > > mod_perl 2.0.10 -
> > After upgrading to Apache 2.4 and mod_perl 2.0.9, I had to make those
> > two changes to my application :
> >
> > In a PerlOutputFilterHandler, change '$content .= $buffer' to
> > '$content .= decode_utf8($buffer)'
> >
> > And in response handlers, change '$args{$_} = $req->param($_)' to
> > '$args{$_} = decode_utf8($req->param($_))'
> >
> > Not sure it applies to your case, but something changed in Apache 2.4
> > concerning UTF-8 data.
> >
> > If I understood correctly, anything that goes through APR::Table is
> > considered UTF-8, however the SvUTF8 flag is not set, so you get
> > double encoding when processing your data.
>

Just to be clear, everything in APR::Table is expected to be ASCII
(or EBCDIC, on those oddball architectures)... and "opaque text",
e.g. characters 128-255 on ASCII architectures is permitted but
not defined.

It's easy to verify that they qualify as UTF-8, because the coding is very
predictable. E.g. 0xFE or 0xFF are not characters, and others are valid
only when in the correct sequence, c.f. the validation logic in;

http://svn.apache.org/repos/asf/apr/apr/trunk/misc/win32/utf8.c

> What a browser sends is not guaranteed, it may be sending ISO-8859-1,
or UTF8 (or even problematic ISO-2022-JP where the ASCII char ' ' or '\'
may occur as a continuation character causing parsing issues. Modern
browsers appear to all be defaulting to UTF-8 finally, but you may have
many legacy browsers out there.

In your decode_utf8 logic, ensure that your app checks for failure! And
where the output fails, try treating it in a code page such as 8859-1.

Re: Bug - Strange issue with mod_perl 2.0.10 / Apache 2.4 corrupting nfreeze data [ In reply to ]

vv.lists at wanadoo

Aug 7, 2016, 10:06 AM

Post #5 of 6 (3171 views)

Permalink

On Sun, 07 Aug 2016 14:37:28 +0100
Alex Masidlover <alex.masidlover@zednax.com> wrote:
>
> Thanks for the suggestion - its taken a while but I've manage to free
> up a machine to roll forwards and test on; unfortunately I've got no
> experience with Apache and filters. I've come up with this:
>
> package Zymonic::Decryptor::Filter;
> use strict;
>
> use base qw(Apache2::Filter);
> use Encode qw(decode);
> use constant BUFF_LEN => 1024;
> use Apache2::Const -compile => 'OK';
>
> sub utf8_filter : FilterRequestHandler {
>
> my $f = shift;
>
> my $content = '';
> while($f->read(my $buffer, BUFF_LEN)) {
> $content .= decode_utf8($buffer);
> }
>
> $f->print($content);
>
> return Apache2::Const::OK;
>
> }
>
> but when I add it with:
>
> PerlOutputFilterHandler Zymonic::Decryptor::Filter::utf8_filter
>
>
> I get:
>
> :Apache2 IO flush: (500) Unknown error 500 at -e line 0
>
> in the Apache error log whenever I hit the server with a request.
>
> I'm guessing I've done something very wrong with the filter, but I can
> find very few examples of how to use Apache2::Filter.
>

I use the streaming filter. Try the recipe for package Apache::HijackFull; (page 13) in :

http://www.modperlcookbook.org/~geoff/slides/WUC/2005/mp2_filters-printable.pdf

(you need to check for seen_eos to print)

This is my code :

http://pastebin.com/dEWqdtsR

Note that I had to place Encode::decode_utf8 inside the 'if ($f->seen_eos)' block

--

Bien à vous, Vincent Veyron

https://marica.fr/
Gestion des sinistres assurances, des dossiers contentieux et des contrats pour le service juridique

Re: Bug - Strange issue with mod_perl 2.0.10 / Apache 2.4 corrupting nfreeze data [ In reply to ]

vv.lists at wanadoo

Aug 7, 2016, 10:42 AM

Post #6 of 6 (3169 views)

Permalink

I forgot to add : if you were not using filters previously, this may not solve your problem.

Do you have accented characters in your queries' parameters? if so those need to be decoded (anything that uses APR::Table)

--
Bien à vous, Vincent Veyron

https://libremen.com
Logiciels de gestion, libres