Mailing List Archive

Too very slow pages on ic 5.10.0
---- (1) -----
after 3 months of installation and light use my old production catalogue
with:
- interchange 5.10.0
- debian 9
- perl 5.24.1
- mysqld 10.1.26-MariaDB-0+deb9u1
- hypervisor (vmware player 12, 4GB ram 1 cpu 4 cores)
I notice that for the pages with heavy query became very slow and
sometimes bring to "Gateway Timeout"
"top" utility display cpu to 100% during page wait sometimes for mysql
and sometimes for interchange

I test the same catalogue in the same pages with the same db on the old
server:
- interchange 5.6.3
- debian 6
- perl 5.10.1
- mysqld 5.1.73-1-log
- hypervisor (virtualbox 4GB ram 1 cpu)

the pages in the old server are quicker from 4 to 14 times !

---- (2) -----
This now is a serious problem for me and my client and in the attempt to
quickly solve it I try to install (develop server) interchange 5.6.3 on
the new server debian 9 but there is a problem on make:

root /usr/local/src/interchange-5.6.3 # make
"/usr/bin/perl" "-Iblib/arch" "-Iblib/lib" relocate.pl scripts/ic_mod_perl
Can't locate scripts/initp.pl in @INC (@INC contains: blib/arch blib/lib
/etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1
/usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24
/usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24
/usr/share/perl/5.24 /usr/local/lib/site_perl
/usr/lib/x86_64-linux-gnu/perl-base) at relocate.pl line 26.
Makefile:898: recipe for target 'scripts/ic_mod_perl' failed
make: *** [scripts/ic_mod_perl] Error 2

Any suggestion ?

--
"Fino alla bara sinpara"
"Up to demise we rize"

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users
Re: Too very slow pages on ic 5.10.0 [ In reply to ]
> -----Original Message-----
> From: interchange-users-bounces@icdevgroup.org [mailto:interchange-users-
> bounces@icdevgroup.org] On Behalf Of marco
> Sent: Saturday, March 10, 2018 14:45
> To: interchange-users@icdevgroup.org
> Subject: [ic] Too very slow pages on ic 5.10.0
>
> ---- (1) -----
> after 3 months of installation and light use my old production catalogue
> with:
> - interchange 5.10.0
> - debian 9
> - perl 5.24.1
> - mysqld 10.1.26-MariaDB-0+deb9u1
> - hypervisor (vmware player 12, 4GB ram 1 cpu 4 cores) I notice that for
the
> pages with heavy query became very slow and sometimes bring to "Gateway
> Timeout"
> "top" utility display cpu to 100% during page wait sometimes for mysql and
> sometimes for interchange
>
> I test the same catalogue in the same pages with the same db on the old
> server:
> - interchange 5.6.3
> - debian 6
> - perl 5.10.1
> - mysqld 5.1.73-1-log
> - hypervisor (virtualbox 4GB ram 1 cpu)
>
> the pages in the old server are quicker from 4 to 14 times !
>
> ---- (2) -----
> This now is a serious problem for me and my client and in the attempt to
> quickly solve it I try to install (develop server) interchange 5.6.3 on
the new
> server debian 9 but there is a problem on make:
>
> root /usr/local/src/interchange-5.6.3 # make "/usr/bin/perl" "-Iblib/arch"
"-
> Iblib/lib" relocate.pl scripts/ic_mod_perl Can't locate scripts/initp.pl
in @INC
> (@INC contains: blib/arch blib/lib /etc/perl /usr/local/lib/x86_64-linux-
> gnu/perl/5.24.1
> /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24
> /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24
> /usr/share/perl/5.24 /usr/local/lib/site_perl
> /usr/lib/x86_64-linux-gnu/perl-base) at relocate.pl line 26.
> Makefile:898: recipe for target 'scripts/ic_mod_perl' failed
> make: *** [scripts/ic_mod_perl] Error 2
>
> Any suggestion ?

I have a bit of difficulty understanding what the install sequence is that
you are trying to do ... This is what I believe the most basic installation
commands:
tar xvzf interchange-X.Y.Z.tar.gz
cd interchange-X.Y.Z
perl Makefile.PL
make
make test
make install

From what I understand from the installation code the perl Makefile.PL
command would create the scripts/initp.pl file .. If this did not happen I'd
assume there is an earlier error that you encountered in your installation.

If this is a completely 'fresh' install you might want to just remove the
/usr/local/src/interchange-5.6.3 folder and go into /usr/local/src/ and get
your interchange tar.gz there and follow the above steps ..

Unless you of course have a different approach for installation which maybe
you could share for us to understand what you usually do.

>
> --
> "Fino alla bara sinpara"
> "Up to demise we rize"
>
> _______________________________________________
> interchange-users mailing list
> interchange-users@icdevgroup.org
> http://www.icdevgroup.org/mailman/listinfo/interchange-users


_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users
Re: Too very slow pages on ic 5.10.0 [ In reply to ]
On 2018-03-10 05:44 AM, marco wrote:
>
> ---- (2) -----
> This now is a serious problem for me and my client and in the attempt to quickly solve it I try to install (develop server) interchange 5.6.3 on the new server debian 9 but there is a problem on make:
>
> root /usr/local/src/interchange-5.6.3 # make
> "/usr/bin/perl" "-Iblib/arch" "-Iblib/lib" relocate.pl scripts/ic_mod_perl
> Can't locate scripts/initp.pl in @INC (@INC contains: blib/arch blib/lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at relocate.pl line 26.
> Makefile:898: recipe for target 'scripts/ic_mod_perl' failed
> make: *** [scripts/ic_mod_perl] Error 2
>
> Any suggestion ?
>

I think you have missed running "perl Makefile.PL" before running make.

cheers,

Andrew


--
https://github.com/pullingshots ... https://metacpan.org/author/BAERGAJ

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users
Re: Too very slow pages on ic 5.10.0 [ In reply to ]
Il 10/03/2018 17:46, Gert van der Spoel ha scritto:
>> -----Original Message-----
>> From: interchange-users-bounces@icdevgroup.org [mailto:interchange-users-
>> bounces@icdevgroup.org] On Behalf Of marco
>> Sent: Saturday, March 10, 2018 14:45
>> To: interchange-users@icdevgroup.org
>> Subject: [ic] Too very slow pages on ic 5.10.0
>>
>> ---- (1) -----
>> after 3 months of installation and light use my old production catalogue
>> with:
>> - interchange 5.10.0
>> - debian 9
>> - perl 5.24.1
>> - mysqld 10.1.26-MariaDB-0+deb9u1
>> - hypervisor (vmware player 12, 4GB ram 1 cpu 4 cores) I notice that for
> the
>> pages with heavy query became very slow and sometimes bring to "Gateway
>> Timeout"
>> "top" utility display cpu to 100% during page wait sometimes for mysql and
>> sometimes for interchange
>>
>> I test the same catalogue in the same pages with the same db on the old
>> server:
>> - interchange 5.6.3
>> - debian 6
>> - perl 5.10.1
>> - mysqld 5.1.73-1-log
>> - hypervisor (virtualbox 4GB ram 1 cpu)
>>
>> the pages in the old server are quicker from 4 to 14 times !
>>
>> ---- (2) -----
>> This now is a serious problem for me and my client and in the attempt to
>> quickly solve it I try to install (develop server) interchange 5.6.3 on
> the new
>> server debian 9 but there is a problem on make:
>>
>> root /usr/local/src/interchange-5.6.3 # make "/usr/bin/perl" "-Iblib/arch"
> "-
>> Iblib/lib" relocate.pl scripts/ic_mod_perl Can't locate scripts/initp.pl
> in @INC
>> (@INC contains: blib/arch blib/lib /etc/perl /usr/local/lib/x86_64-linux-
>> gnu/perl/5.24.1
>> /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24
>> /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24
>> /usr/share/perl/5.24 /usr/local/lib/site_perl
>> /usr/lib/x86_64-linux-gnu/perl-base) at relocate.pl line 26.
>> Makefile:898: recipe for target 'scripts/ic_mod_perl' failed
>> make: *** [scripts/ic_mod_perl] Error 2
>>
>> Any suggestion ?
>
> I have a bit of difficulty understanding what the install sequence is that
> you are trying to do ... This is what I believe the most basic installation
> commands:
> tar xvzf interchange-X.Y.Z.tar.gz
> cd interchange-X.Y.Z
> perl Makefile.PL
> make
> make test
> make install
>
>>From what I understand from the installation code the perl Makefile.PL
> command would create the scripts/initp.pl file .. If this did not happen I'd
> assume there is an earlier error that you encountered in your installation.
>
> If this is a completely 'fresh' install you might want to just remove the
> /usr/local/src/interchange-5.6.3 folder and go into /usr/local/src/ and get
> your interchange tar.gz there and follow the above steps ..
>
> Unless you of course have a different approach for installation which maybe
> you could share for us to understand what you usually do.
>
My way to install is as you suggest:
perl -MCPAN -e 'install Bundle::Interchange'
tar xvzf interchange-X.Y.Z.tar.gz
cd interchange-X.Y.Z
perl Makefile.PL
make
make test
make install

BUT when I had installed ic 5.10.0 I had the same make "Error 2" and so,
from mailing list suggestion, I download this version from github
https://github.com/interchange/interchange/archive/master.zip
and followed that instructions with success.

"Fresh" ? The ic 5.6.3 installation tried now is from an alternative dir
but in the same server debian 9 where run slow ic 5.10.0. I had tried
this method of more ic versions on the same server more times in the
past. With a soft link dir 'interchange' pointing to the installed
testing version for example /usr/local/interchange-5.6.3 or
/usr/local/interchange-5.10.0 I can test and try different version on
the same catalogue.

As your suggestion I tried to move interchange-5.6.3.tar.gz to
/usr/src/interchange-5.6.3 rather than /usr/local/src/interchange-5.6.3
and follow same way to compile but I obtain the same error.

Catching the output of install sequence I do not find any hints:
-------------------------------------------------------------
root /usr/src/interchange-5.6.3 # perl Makefile.PL
Checking if your kit is complete...
Looks good



Interchange cannot be run as root. Which user
should run Interchange? interch

Interchange V5.6.3

Copyright (C) 2002-2010 Interchange Development Group.
Copyright (C) 1996-2002 Red Hat, Inc.
Interchange is free under the terms of the GNU General Public License.

http://www.icdevgroup.org/




Where is your Interchange to be installed?
/usr/local/interchange-5.6.3

Generating a Unix-style Makefile
Writing Makefile for Interchange
Writing MYMETA.yml and MYMETA.json

root /usr/src/interchange-5.6.3 # make
cp lib/Vend/Payment/EFSNet.pm blib/lib/Vend/Payment/EFSNet.pm
cp lib/Vend/Email.pm blib/lib/Vend/Email.pm
cp lib/Vend/Payment/PSiGate.pm blib/lib/Vend/Payment/PSiGate.pm
cp lib/Vend/Payment/PRI.pm blib/lib/Vend/Payment/PRI.pm
cp lib/Vend/Payment/BusinessOnlinePayment.pm
blib/lib/Vend/Payment/BusinessOnlinePayment.pm
cp lib/Vend/Cron.pm blib/lib/Vend/Cron.pm
cp lib/Vend/Payment.pm blib/lib/Vend/Payment.pm
cp lib/Vend/Data.pm blib/lib/Vend/Data.pm
cp lib/Vend/MakeCat.pm blib/lib/Vend/MakeCat.pm
cp lib/Vend/Payment/ICS.pm blib/lib/Vend/Payment/ICS.pm
cp lib/Vend/Payment/ECHO.pm blib/lib/Vend/Payment/ECHO.pm
cp lib/Vend/Options/Simple.pm blib/lib/Vend/Options/Simple.pm
cp lib/Vend/Page.pm blib/lib/Vend/Page.pm
cp lib/Vend/Glimpse.pm blib/lib/Vend/Glimpse.pm
cp lib/Vend/Payment/Protx2.pm blib/lib/Vend/Payment/Protx2.pm
cp lib/Vend/Options/Old48.pm blib/lib/Vend/Options/Old48.pm
cp lib/Vend/Menu.pm blib/lib/Vend/Menu.pm
cp lib/Vend/CounterFile.pm blib/lib/Vend/CounterFile.pm
cp lib/Vend/Order.pm blib/lib/Vend/Order.pm
cp lib/Vend/Payment/Sage.pm blib/lib/Vend/Payment/Sage.pm
cp lib/Vend/External.pm blib/lib/Vend/External.pm
cp lib/Vend/Error.pm blib/lib/Vend/Error.pm
cp hints.pl blib/lib/hints.pl
cp lib/Vend/Payment/BoA.pm blib/lib/Vend/Payment/BoA.pm
cp lib/Vend/File.pm blib/lib/Vend/File.pm
cp lib/Vend/Config.pm blib/lib/Vend/Config.pm
cp lib/Vend/Document.pm blib/lib/Vend/Document.pm
cp lib/Vend/Dispatch.pm blib/lib/Vend/Dispatch.pm
cp lib/Vend/Payment/CyberCash.pm blib/lib/Vend/Payment/CyberCash.pm
cp lib/Vend/Interpolate.pm blib/lib/Vend/Interpolate.pm
cp lib/Vend/Payment/MCVE.pm blib/lib/Vend/Payment/MCVE.pm
cp install_lsb.pl blib/lib/install_lsb.pl
cp lib/Vend/Cart.pm blib/lib/Vend/Cart.pm
cp lib/Vend/Parser.pm blib/lib/Vend/Parser.pm
cp lib/Vend/Accounting.pm blib/lib/Vend/Accounting.pm
cp lib/Vend/Parse.pm blib/lib/Vend/Parse.pm
cp lib/Vend/Options.pm blib/lib/Vend/Options.pm
cp lib/Vend/Form.pm blib/lib/Vend/Form.pm
cp lib/Vend/DbSearch.pm blib/lib/Vend/DbSearch.pm
cp lib/Vend/Payment/Linkpoint.pm blib/lib/Vend/Payment/Linkpoint.pm
cp lib/Vend/Payment/NetBilling.pm blib/lib/Vend/Payment/NetBilling.pm
cp lib/Vend/Imagemap.pm blib/lib/Vend/Imagemap.pm
cp lib/Vend/Options/Matrix.pm blib/lib/Vend/Options/Matrix.pm
cp lib/Vend/ModPerl.pm blib/lib/Vend/ModPerl.pm
cp lib/Vend/Payment/AuthorizeNet.pm blib/lib/Vend/Payment/AuthorizeNet.pm
cp lib/Vend/Payment/Getitcard.pm blib/lib/Vend/Payment/Getitcard.pm
cp lib/Vend/Accounting/SQL_Ledger.pm blib/lib/Vend/Accounting/SQL_Ledger.pm
cp lib/Vend/CharSet.pm blib/lib/Vend/CharSet.pm
cp lib/Vend/Control.pm blib/lib/Vend/Control.pm
cp lib/Vend/Payment/Ezic.pm blib/lib/Vend/Payment/Ezic.pm
cp lib/Vend/Table/DB_File.pm blib/lib/Vend/Table/DB_File.pm
cp lib/Vend/SessionDB.pm blib/lib/Vend/SessionDB.pm
cp lib/Vend/Tags.pm blib/lib/Vend/Tags.pm
AutoSplitting blib/lib/Vend/Tags.pm (blib/lib/auto/Vend/Tags)
cp lib/Vend/SOAP.pm blib/lib/Vend/SOAP.pm
AutoSplitting blib/lib/Vend/SOAP.pm (blib/lib/auto/Vend/SOAP)
cp lib/Vend/SQL_Parser.pm blib/lib/Vend/SQL_Parser.pm
cp lib/Vend/Table/DBI_CompositeKey.pm
blib/lib/Vend/Table/DBI_CompositeKey.pm
cp relocate.pl blib/lib/relocate.pl
cp lib/Vend/Track.pm blib/lib/Vend/Track.pm
cp lib/Vend/Table/DBI.pm blib/lib/Vend/Table/DBI.pm
cp lib/Vend/Scan.pm blib/lib/Vend/Scan.pm
cp lib/Vend/Payment/TCLink.pm blib/lib/Vend/Payment/TCLink.pm
cp lib/Vend/Table/Shadow.pm blib/lib/Vend/Table/Shadow.pm
cp lib/Vend/Table/LDAP.pm blib/lib/Vend/Table/LDAP.pm
cp lib/Vend/TextSearch.pm blib/lib/Vend/TextSearch.pm
cp lib/Vend/Subs.pm blib/lib/Vend/Subs.pm
cp lib/Vend/Ship.pm blib/lib/Vend/Ship.pm
cp lib/Vend/SOAP/Transport.pm blib/lib/Vend/SOAP/Transport.pm
cp lib/Vend/Payment/Skipjack.pm blib/lib/Vend/Payment/Skipjack.pm
cp lib/Vend/UserControl.pm blib/lib/Vend/UserControl.pm
cp lib/Vend/Table/Editor.pm blib/lib/Vend/Table/Editor.pm
cp lib/Vend/Session.pm blib/lib/Vend/Session.pm
cp lib/Vend/Util.pm blib/lib/Vend/Util.pm
cp lib/Vend/Table/Common.pm blib/lib/Vend/Table/Common.pm
cp lib/Vend/UserDB.pm blib/lib/Vend/UserDB.pm
cp lib/Vend/Search.pm blib/lib/Vend/Search.pm
cp lib/Vend/Table/SDBM.pm blib/lib/Vend/Table/SDBM.pm
cp lib/Vend/Ship/QueryUPS.pm blib/lib/Vend/Ship/QueryUPS.pm
cp lib/Vend/Payment/TestPayment.pm blib/lib/Vend/Payment/TestPayment.pm
cp lib/Vend/Table/InMemory.pm blib/lib/Vend/Table/InMemory.pm
cp lib/Vend/Table/GDBM.pm blib/lib/Vend/Table/GDBM.pm
cp lib/Vend/Payment/Signio.pm blib/lib/Vend/Payment/Signio.pm
cp lib/Vend/Server.pm blib/lib/Vend/Server.pm
cp lib/Vend/Swish.pm blib/lib/Vend/Swish.pm
cp lib/Vend/Ship/Postal.pm blib/lib/Vend/Ship/Postal.pm
cp lib/Vend/RefSearch.pm blib/lib/Vend/RefSearch.pm
cp lib/Vend/SessionFile.pm blib/lib/Vend/SessionFile.pm
cp lib/Vend/Payment/iTransact.pm blib/lib/Vend/Payment/iTransact.pm
"/usr/bin/perl" "-Iblib/arch" "-Iblib/lib" relocate.pl scripts/makecat
Can't locate scripts/initp.pl in @INC (@INC contains: blib/arch blib/lib
/etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1
/usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24
/usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24
/usr/share/perl/5.24 /usr/local/lib/site_perl
/usr/lib/x86_64-linux-gnu/perl-base) at relocate.pl line 26.
Makefile:916: recipe for target 'scripts/makecat' failed
make: *** [scripts/makecat] Error 2


--
"Fino alla bara sinpara"
"Up to demise we rize"

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users
Re: Too very slow pages on ic 5.10.0 [ In reply to ]
> On Mar 10, 2018, at 10:48 AM, Andrew Baerg <andrew@pullingshots.ca> wrote:
>
>> On 2018-03-10 05:44 AM, marco wrote:
>>
>> ---- (2) -----
>> This now is a serious problem for me and my client and in the attempt to quickly solve it I try to install (develop server) interchange 5.6.3 on the new server debian 9 but there is a problem on make:
>>
>> root /usr/local/src/interchange-5.6.3 # make
>> "/usr/bin/perl" "-Iblib/arch" "-Iblib/lib" relocate.pl scripts/ic_mod_perl
>> Can't locate scripts/initp.pl in @INC (@INC contains: blib/arch blib/lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at relocate.pl line 26.
>> Makefile:898: recipe for target 'scripts/ic_mod_perl' failed
>> make: *** [scripts/ic_mod_perl] Error 2
>>
>> Any suggestion ?
>>
>
> I think you have missed running "perl Makefile.PL" before running make.

I believe the issue here is perl 5.24 did away with including '.' as an implicit include path. You would need to patch the IC 5.6 codebase to add '.' back into @INC or it won't be able to find the relative include.

Best,

David

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users
Re: Too very slow pages on ic 5.10.0 [ In reply to ]
Il 10/03/2018 22:11, David Christensen ha scritto:
>
>> On Mar 10, 2018, at 10:48 AM, Andrew Baerg <andrew@pullingshots.ca> wrote:
>>
>>> On 2018-03-10 05:44 AM, marco wrote:
>>>
>>> ---- (2) -----
>>> This now is a serious problem for me and my client and in the attempt to quickly solve it I try to install (develop server) interchange 5.6.3 on the new server debian 9 but there is a problem on make:
>>>
>>> root /usr/local/src/interchange-5.6.3 # make
>>> "/usr/bin/perl" "-Iblib/arch" "-Iblib/lib" relocate.pl scripts/ic_mod_perl
>>> Can't locate scripts/initp.pl in @INC (@INC contains: blib/arch blib/lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at relocate.pl line 26.
>>> Makefile:898: recipe for target 'scripts/ic_mod_perl' failed
>>> make: *** [scripts/ic_mod_perl] Error 2
>>>
>>> Any suggestion ?
>>>
>>
>> I think you have missed running "perl Makefile.PL" before running make.
>
> I believe the issue here is perl 5.24 did away with including '.' as an implicit include path. You would need to patch the IC 5.6 codebase to add '.' back into @INC or it won't be able to find the relative include.
>
> Best,
>
> David

Great David, is this !
Before compilation:
export PERL5LIB=.
And I can have standard and my old catalogs running.

Some warnings on run interchange daemon probably due to perl 5.24.1 or
not yet analized catalog.cfg configuration:

---- (1) -----
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/(\$O{ <-- HERE (.*?)})/ at
/usr/local/src/interchange-5.6.3/blib/lib/Vend/Ship.pm line 980, <DATA>
line 1.

---- (2) -----
Global symbol "$table" requires explicit package name
...

---------- Test speed --------------
With ic 5.10.0 and debian 9 from "top" monitor I have displayed cpu
activity for 2 heavy pages

a) a sheet of about 50 fields on 20 records (every pages) with a lot of
query and in some cases take about 2 minutes or more of 100% cpu
activity for mysql

b) the other is a more light sheet that engage interchange daemon to
100% cpu for about 30 seconds

With ic 5.6.3 and debian 9
a) give about the same time,
b) is improved to 6 s.

With ic 5.6.3 and debian 6
a) run in 5-10 s,
b) 6 s

--
"Fino alla bara sinpara"
"Up to demise we rize"

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users
Re: Too very slow pages on ic 5.10.0 [ In reply to ]
On 11/03/18 01:44, marco wrote:
> after 3 months of installation and light use my old production catalogue
> with:

> I notice that for the pages with heavy query became very slow and
> sometimes bring to "Gateway Timeout"
> "top" utility display cpu to 100% during page wait sometimes for mysql
> and sometimes for interchange

You need to benchmark the code on the problematic pages and find out
exactly which lines or blocks of code are causing the issue.

This is most likely going to be the result of lots of DB hits from some
very inefficient code. Look for calls to [data] or [field], or similar
tags that waste db hits inside of a [loop] or similar. Rewrite your
code so that it does a single query to the db to get all the data you
need instead and it will end up being way more efficient.

> I test the same catalogue in the same pages with the same db on the old
> server:
>
> the pages in the old server are quicker from 4 to 14 times !

This is not necessarily an Interchange difference here. In fact I would
be more likely to look at differences in the db that means that the new
db can't handle all the hits of the old one, or maybe the new system is
not as fast with disk IO. Compare mysql settings on the old and new server.

> This now is a serious problem for me and my client and in the attempt to
> quickly solve it I try to install (develop server) interchange 5.6.3 on
> the new server

I fear you may be barking up the wrong tree here, but others have helped
you with the installation issues so I won't go into that.


Peter

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users
Re: Too very slow pages on ic 5.10.0 [ In reply to ]
On 2018-03-11 12:51 PM, Peter wrote:
> On 11/03/18 01:44, marco wrote:
>> after 3 months of installation and light use my old production catalogue
>> with:
>
>> I notice that for the pages with heavy query became very slow and
>> sometimes bring to "Gateway Timeout"
>> "top" utility display cpu to 100% during page wait sometimes for mysql
>> and sometimes for interchange
>
> You need to benchmark the code on the problematic pages and find out
> exactly which lines or blocks of code are causing the issue.
>
> This is most likely going to be the result of lots of DB hits from some
> very inefficient code. Look for calls to [data] or [field], or similar
> tags that waste db hits inside of a [loop] or similar. Rewrite your
> code so that it does a single query to the db to get all the data you
> need instead and it will end up being way more efficient.
>

Totally agree with this.

Also, if you are processing large queries with [query sql="..." type=list] ... [/query], consider using the hashref option [query sql="..." hashref="foo"][/query] and then processing the output in a calc block:

[calc]
foreach $line (@{$Tmp->{foo}}) {
....
}
[/calc]

I have seen page load times go from seconds to milliseconds using this strategy.

Not sure if it applies to the OP's situation, but I seem to remember having some pages that were processing large amounts of data using [query type=list] really slow down on newer Perls with UTF8 enabled in the DBD::Pg driver.

cheers,

Andrew


--
https://github.com/pullingshots ... https://metacpan.org/author/BAERGAJ

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users
Re: Too very slow pages on ic 5.10.0 [ In reply to ]
Quoting Peter (peter@pajamian.dhs.org):
> On 11/03/18 01:44, marco wrote:
> > after 3 months of installation and light use my old production catalogue
> > with:
>
> > I notice that for the pages with heavy query became very slow and
> > sometimes bring to "Gateway Timeout"
> > "top" utility display cpu to 100% during page wait sometimes for mysql
> > and sometimes for interchange
>
> You need to benchmark the code on the problematic pages and find out
> exactly which lines or blocks of code are causing the issue.
>
> This is most likely going to be the result of lots of DB hits from some
> very inefficient code. Look for calls to [data] or [field], or similar
> tags that waste db hits inside of a [loop] or similar. Rewrite your
> code so that it does a single query to the db to get all the data you
> need instead and it will end up being way more efficient.

Don't discount the possibility that the I/O on this virtual instance
is slower than on the other one. Are there demanding applications which
show the same speed on the new container? Try doing some large DB queries
on both the old and the new and comparing.

--
Mike Heins
phone +1.765.253.4194 <mike@heins.com>

The sun, with all those planets revolving around it and dependent on it,
can still ripen a bunch of grapes as if it had nothing else in the
universe to do. -- Galileo

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users
Re: Too very slow pages on ic 5.10.0 [ In reply to ]
> On Mar 11, 2018, at 11:01 PM, Mike Heins <mike@heins.net> wrote:
>
> Quoting Peter (peter@pajamian.dhs.org <mailto:peter@pajamian.dhs.org>):
>> On 11/03/18 01:44, marco wrote:
>>> after 3 months of installation and light use my old production catalogue
>>> with:
>>
>>> I notice that for the pages with heavy query became very slow and
>>> sometimes bring to "Gateway Timeout"
>>> "top" utility display cpu to 100% during page wait sometimes for mysql
>>> and sometimes for interchange
>>
>> You need to benchmark the code on the problematic pages and find out
>> exactly which lines or blocks of code are causing the issue.
>>
>> This is most likely going to be the result of lots of DB hits from some
>> very inefficient code. Look for calls to [data] or [field], or similar
>> tags that waste db hits inside of a [loop] or similar. Rewrite your
>> code so that it does a single query to the db to get all the data you
>> need instead and it will end up being way more efficient.
>
> Don't discount the possibility that the I/O on this virtual instance
> is slower than on the other one. Are there demanding applications which
> show the same speed on the new container? Try doing some large DB queries
> on both the old and the new and comparing.


With newer perls, my personal experience is that parsing slowdowns are caused by using regexes against Unicode strings. If you’re using PostgreSQL with DBD::Pg >= 3, this can also materialize itself *even without* explicit configuration on the interchange side (i.e., MV_UTF8 and friends). If this matches your situation: Newer perl, PostgreSQL, DBD::Pg >= 3 (*no* MV_UTF8 configuration) you can turn off the automatic unicodization of database data via this in your catalog.cfg:

DatabaseDefault PG_ENABLE_UTF8 0

The issue is that DBD::Pg >= 3 by default will automatically decode any data it pulls from the server if the client_encoding is UTF8, unless you explicitly disable that functionality, which means that any page where IC interpolates data that came from the database will have the page string upgraded to Unicode, which will take the slower regex path. This is particularly noticeable in perls ~ 5.14 - 5.20, but is still substantially slower than the non-unicode path in 5.24.

Not sure if that might help explain things, but this is something we’ve run into with recent perls. Additionally, the normal database optimization approaches mentioned elsewhere in this thread will also always be good too.

Best,

David
--
David Christensen
End Point Corporation
david@endpoint.com
785-727-1171
Re: Too very slow pages on ic 5.10.0 [ In reply to ]
Il 12/03/2018 04:30, Andrew Baerg ha scritto:
> On 2018-03-11 12:51 PM, Peter wrote:
>> On 11/03/18 01:44, marco wrote:
>>> after 3 months of installation and light use my old production catalogue
>>> with:
>>
>>> I notice that for the pages with heavy query became very slow and
>>> sometimes bring to "Gateway Timeout"
>>> "top" utility display cpu to 100% during page wait sometimes for mysql
>>> and sometimes for interchange
>>
>> You need to benchmark the code on the problematic pages and find out
>> exactly which lines or blocks of code are causing the issue.
>>
>> This is most likely going to be the result of lots of DB hits from some
>> very inefficient code. Look for calls to [data] or [field], or similar
>> tags that waste db hits inside of a [loop] or similar. Rewrite your
>> code so that it does a single query to the db to get all the data you
>> need instead and it will end up being way more efficient.
>>
>
> Totally agree with this.
>
> Also, if you are processing large queries with [query sql="..." type=list] ... [/query], consider using the hashref option [query sql="..." hashref="foo"][/query] and then processing the output in a calc block:
>
> [calc]
> foreach $line (@{$Tmp->{foo}}) {
> ....
> }
> [/calc]
>
> I have seen page load times go from seconds to milliseconds using this strategy.
>
> Not sure if it applies to the OP's situation, but I seem to remember having some pages that were processing large amounts of data using [query type=list] really slow down on newer Perls with UTF8 enabled in the DBD::Pg driver.
>
> cheers,
>
> Andrew

My code ? I am sure is inefficient code ;-) but is about 10/15 years
that is running with different mix of os, perl, ic and mysql version ...
the problem in not my dirty code but why this dirty code run so slow in
the last (5.10.0) ic version.

Maybe my enviro configuration issue, seems a mysql/mariaDb problem bound
to utf8 encoding.

Strange behavior: pages slow very down only from the second/third
different mysql query.

--
"Fino alla bara sinpara"
"Up to demise we rize"

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users
Re: Too very slow pages on ic 5.10.0 [ In reply to ]
Il 12/03/2018 05:01, Mike Heins ha scritto:
> Quoting Peter (peter@pajamian.dhs.org):
>> On 11/03/18 01:44, marco wrote:
>>> after 3 months of installation and light use my old production catalogue
>>> with:
>>
>>> I notice that for the pages with heavy query became very slow and
>>> sometimes bring to "Gateway Timeout"
>>> "top" utility display cpu to 100% during page wait sometimes for mysql
>>> and sometimes for interchange
>>
>> You need to benchmark the code on the problematic pages and find out
>> exactly which lines or blocks of code are causing the issue.
>>
>> This is most likely going to be the result of lots of DB hits from some
>> very inefficient code. Look for calls to [data] or [field], or similar
>> tags that waste db hits inside of a [loop] or similar. Rewrite your
>> code so that it does a single query to the db to get all the data you
>> need instead and it will end up being way more efficient.
>
> Don't discount the possibility that the I/O on this virtual instance
> is slower than on the other one. Are there demanding applications which
> show the same speed on the new container? Try doing some large DB queries
> on both the old and the new and comparing.

Same queries on the old and the new servers from mysql workbench. Timing
is good namely not too long, only with join on not primary key becomes
high but in both server.

In the new server (debian 9, ic 5.10.0) mysql query is faster from 10%
to 20% than old server (debian 6, ic 5.6.3) in the opposite way getting
ic pages.

I repeat myself: getting ic pages the new server is a little faster than
the old (as mysql above query says) for the first queries/pages then
pages becomes very slow only on the new server.

--
"Fino alla bara sinpara"
"Up to demise we rize"

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users
Re: Too very slow pages on ic 5.10.0 [ In reply to ]
Il 12/03/2018 21:03, marco ha scritto:
> Il 12/03/2018 05:01, Mike Heins ha scritto:
>> Quoting Peter (peter@pajamian.dhs.org):
>>> On 11/03/18 01:44, marco wrote:
>>>> after 3 months of installation and light use my old production
>>>> catalogue
>>>> with:
>>>
>>>> I notice that for the pages with heavy query became very slow and
>>>> sometimes bring to "Gateway Timeout"
>>>> "top" utility display cpu to 100% during page wait sometimes for mysql
>>>> and sometimes for interchange
>>>
>>> You need to benchmark the code on the problematic pages and find out
>>> exactly which lines or blocks of code are causing the issue.
>>>
>>> This is most likely going to be the result of lots of DB hits from some
>>> very inefficient code. Look for calls to [data] or [field], or similar
>>> tags that waste db hits inside of a [loop] or similar. Rewrite your
>>> code so that it does a single query to the db to get all the data you
>>> need instead and it will end up being way more efficient.
>>
>> Don't discount the possibility that the I/O on this virtual instance
>> is slower than on the other one. Are there demanding applications which
>> show the same speed on the new container? Try doing some large DB queries
>> on both the old and the new and comparing.
>
> Same queries on the old and the new servers from mysql workbench. Timing
> is good namely not too long, only with join on not primary key becomes
> high but in both server.
>
> In the new server (debian 9, ic 5.10.0) mysql query is faster from 10%
> to 20% than old server (debian 6, ic 5.6.3) in the opposite way getting
> ic pages.
>
> I repeat myself: getting ic pages the new server is a little faster than
> the old (as mysql above query says) for the first queries/pages then
> pages becomes very slow only on the new server.

Final summary: two server
old) debian6, ic 5.6.3, mysql
new) debian9, ic 5.6.10, mariadb
Same db data, same catalogue that has run for more then 10 years with
different os (debian, ubuntu), ic (more than 5 version), mysql versions.
With new) some pages slow down also 20 times.
With the help of David Christensen we found 2 indipendent problems:

1) ic 5.6.10 with utf8 enabled make an heavy regex processing.
Solution by David, old catalogue dont need utf8:
a) in /usr/local/interchange/bin/interchange perl script before "use
strict":
$ENV{MINIVEND_DISABLE_UTF8}=1
b) in catalogue.cfg:
DatabaseDefault MYSQL_ENABLE_UTF8=0

2) the new) on heavy and multiple sql query, after less than 10 and more
than 2 request of the same page with different query, mariadb suddenly
has great slow down due sql processing.
We haven't find any problem on mariadb configuration and so not found
any direct and complete solution. My quick/2 and dirty/2 solution was to
run db in a separate vm server with debian6 and mysql. I dont know if
with debian9, ic 5.10.0 and oracle mysql the problem persist.

--
"Fino alla bara sinpara"
"Up to demise we rize"

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users