Mailing List Archive: Time for new hardware?

Time for new hardware?

Sep 19, 2016, 2:41 PM

Post #1 of 26 (6274 views)

I've been making a lot of optimizations lately and I think I'm to the
point where my 4 cores just aren't able to keep up with demand during
peak traffic hours each day. Pages load quickly when I see fewer than
4 busy interchange processes in top but things slow down drastically
after that. Once I'm OK with the degree of optimization my ITL pages
have undergone and I'm not IO-bound or memory-bound, is it time to
throw CPU at the problem?

I'm a little puzzled because I've seen my server perform much better
under much greater loads in the past. I thought my growing mysql
tables could be the problem so I set up indexes and it has helped but
my server still struggles under loads it used to handle without issue.
I did notice that my tables are split about 50/50 between InnoDB and
MyISAM and I'd like to make all of them InnoDB.

- Grant

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

racke at linuxia

Sep 20, 2016, 2:27 AM

Post #2 of 26 (6258 views)

Permalink

On 09/19/2016 11:41 PM, Grant wrote:
> I've been making a lot of optimizations lately and I think I'm to the
> point where my 4 cores just aren't able to keep up with demand during
> peak traffic hours each day. Pages load quickly when I see fewer than
> 4 busy interchange processes in top but things slow down drastically
> after that. Once I'm OK with the degree of optimization my ITL pages
> have undergone and I'm not IO-bound or memory-bound, is it time to
> throw CPU at the problem?
>
> I'm a little puzzled because I've seen my server perform much better
> under much greater loads in the past. I thought my growing mysql
> tables could be the problem so I set up indexes and it has helped but
> my server still struggles under loads it used to handle without issue.
> I did notice that my tables are split about 50/50 between InnoDB and
> MyISAM and I'd like to make all of them InnoDB.
>
> - Grant

Yeah, I would recommend to switch all them to InnoDB for consistency
and real transactions.

You probably can find sufficient resources on MySQL optimization with
InnoDB if that is your bottleneck.

May I ask which webserver you are using?

Regards
Racke

--
Ecommerce and Linux consulting + Perl and web application programming.
Debian and Sympa administration.

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

emailgrant at gmail

Sep 20, 2016, 5:20 AM

Post #3 of 26 (6258 views)

Permalink

>> I've been making a lot of optimizations lately and I think I'm to the
>> point where my 4 cores just aren't able to keep up with demand during
>> peak traffic hours each day. Pages load quickly when I see fewer than
>> 4 busy interchange processes in top but things slow down drastically
>> after that. Once I'm OK with the degree of optimization my ITL pages
>> have undergone and I'm not IO-bound or memory-bound, is it time to
>> throw CPU at the problem?
>>
>> I'm a little puzzled because I've seen my server perform much better
>> under much greater loads in the past. I thought my growing mysql
>> tables could be the problem so I set up indexes and it has helped but
>> my server still struggles under loads it used to handle without issue.
>> I did notice that my tables are split about 50/50 between InnoDB and
>> MyISAM and I'd like to make all of them InnoDB.
>>
>> - Grant
>
> Yeah, I would recommend to switch all them to InnoDB for consistency
> and real transactions.
>
> You probably can find sufficient resources on MySQL optimization with
> InnoDB if that is your bottleneck.

Actually IO doesn't seem to be a bottleneck at this point. It seems
to be CPU as things slow down once I have 4 busy interchange processes
on my quad-core CPU and iotop does not show much activity at that
point. Besides ITL optimization, is this a clear case of needing a
faster CPU and more cores or is there anything else to consider first?

> May I ask which webserver you are using?

I'm using nginx reverse proxied to apache2.

- Grant

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

racke at linuxia

Sep 20, 2016, 5:57 AM

Post #4 of 26 (6257 views)

Permalink

On 09/20/2016 02:20 PM, Grant wrote:
>>> I've been making a lot of optimizations lately and I think I'm to the
>>> point where my 4 cores just aren't able to keep up with demand during
>>> peak traffic hours each day. Pages load quickly when I see fewer than
>>> 4 busy interchange processes in top but things slow down drastically
>>> after that. Once I'm OK with the degree of optimization my ITL pages
>>> have undergone and I'm not IO-bound or memory-bound, is it time to
>>> throw CPU at the problem?
>>>
>>> I'm a little puzzled because I've seen my server perform much better
>>> under much greater loads in the past. I thought my growing mysql
>>> tables could be the problem so I set up indexes and it has helped but
>>> my server still struggles under loads it used to handle without issue.
>>> I did notice that my tables are split about 50/50 between InnoDB and
>>> MyISAM and I'd like to make all of them InnoDB.
>>>
>>> - Grant
>>
>> Yeah, I would recommend to switch all them to InnoDB for consistency
>> and real transactions.
>>
>> You probably can find sufficient resources on MySQL optimization with
>> InnoDB if that is your bottleneck.
>
>
> Actually IO doesn't seem to be a bottleneck at this point. It seems
> to be CPU as things slow down once I have 4 busy interchange processes
> on my quad-core CPU and iotop does not show much activity at that
> point. Besides ITL optimization, is this a clear case of needing a
> faster CPU and more cores or is there anything else to consider first?
>
>
>> May I ask which webserver you are using?
>
>
> I'm using nginx reverse proxied to apache2.
>
> - Grant

Did you consider to get rid of Apache?

Regards
Racke

--
Ecommerce and Linux consulting + Perl and web application programming.
Debian and Sympa administration.

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

emailgrant at gmail

Sep 20, 2016, 6:12 AM

Post #5 of 26 (6257 views)

Permalink

>>>> I've been making a lot of optimizations lately and I think I'm to the
>>>> point where my 4 cores just aren't able to keep up with demand during
>>>> peak traffic hours each day. Pages load quickly when I see fewer than
>>>> 4 busy interchange processes in top but things slow down drastically
>>>> after that. Once I'm OK with the degree of optimization my ITL pages
>>>> have undergone and I'm not IO-bound or memory-bound, is it time to
>>>> throw CPU at the problem?
>>>>
>>>> I'm a little puzzled because I've seen my server perform much better
>>>> under much greater loads in the past. I thought my growing mysql
>>>> tables could be the problem so I set up indexes and it has helped but
>>>> my server still struggles under loads it used to handle without issue.
>>>> I did notice that my tables are split about 50/50 between InnoDB and
>>>> MyISAM and I'd like to make all of them InnoDB.
>>>>
>>>> - Grant
>>>
>>> Yeah, I would recommend to switch all them to InnoDB for consistency
>>> and real transactions.
>>>
>>> You probably can find sufficient resources on MySQL optimization with
>>> InnoDB if that is your bottleneck.
>>
>>
>> Actually IO doesn't seem to be a bottleneck at this point. It seems
>> to be CPU as things slow down once I have 4 busy interchange processes
>> on my quad-core CPU and iotop does not show much activity at that
>> point. Besides ITL optimization, is this a clear case of needing a
>> faster CPU and more cores or is there anything else to consider first?
>>
>>
>>> May I ask which webserver you are using?
>>
>>
>> I'm using nginx reverse proxied to apache2.
>
> Did you consider to get rid of Apache?

I'm certainly planning to do that but if I watch top it looks like my
apache2 processes *barely* get above idle even when the web server is
under heavy load. The interchange processes get hammered.

- Grant

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

racke at linuxia

Sep 20, 2016, 6:16 AM

Post #6 of 26 (6258 views)

Permalink

On 09/20/2016 03:12 PM, Grant wrote:
>>>>> I've been making a lot of optimizations lately and I think I'm to the
>>>>> point where my 4 cores just aren't able to keep up with demand during
>>>>> peak traffic hours each day. Pages load quickly when I see fewer than
>>>>> 4 busy interchange processes in top but things slow down drastically
>>>>> after that. Once I'm OK with the degree of optimization my ITL pages
>>>>> have undergone and I'm not IO-bound or memory-bound, is it time to
>>>>> throw CPU at the problem?
>>>>>
>>>>> I'm a little puzzled because I've seen my server perform much better
>>>>> under much greater loads in the past. I thought my growing mysql
>>>>> tables could be the problem so I set up indexes and it has helped but
>>>>> my server still struggles under loads it used to handle without issue.
>>>>> I did notice that my tables are split about 50/50 between InnoDB and
>>>>> MyISAM and I'd like to make all of them InnoDB.
>>>>>
>>>>> - Grant
>>>>
>>>> Yeah, I would recommend to switch all them to InnoDB for consistency
>>>> and real transactions.
>>>>
>>>> You probably can find sufficient resources on MySQL optimization with
>>>> InnoDB if that is your bottleneck.
>>>
>>>
>>> Actually IO doesn't seem to be a bottleneck at this point. It seems
>>> to be CPU as things slow down once I have 4 busy interchange processes
>>> on my quad-core CPU and iotop does not show much activity at that
>>> point. Besides ITL optimization, is this a clear case of needing a
>>> faster CPU and more cores or is there anything else to consider first?
>>>
>>>
>>>> May I ask which webserver you are using?
>>>
>>>
>>> I'm using nginx reverse proxied to apache2.
>>
>> Did you consider to get rid of Apache?
>
>
> I'm certainly planning to do that but if I watch top it looks like my
> apache2 processes *barely* get above idle even when the web server is
> under heavy load. The interchange processes get hammered.
>

Which mode/settings do you use for the Interchange server? How
do you connect Nginx and Interchange?

Regards
Racke

--
Ecommerce and Linux consulting + Perl and web application programming.
Debian and Sympa administration.

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

emailgrant at gmail

Sep 20, 2016, 6:31 AM

Post #7 of 26 (6254 views)

Permalink

>>>>>> I've been making a lot of optimizations lately and I think I'm to the
>>>>>> point where my 4 cores just aren't able to keep up with demand during
>>>>>> peak traffic hours each day. Pages load quickly when I see fewer than
>>>>>> 4 busy interchange processes in top but things slow down drastically
>>>>>> after that. Once I'm OK with the degree of optimization my ITL pages
>>>>>> have undergone and I'm not IO-bound or memory-bound, is it time to
>>>>>> throw CPU at the problem?
>>>>>>
>>>>>> I'm a little puzzled because I've seen my server perform much better
>>>>>> under much greater loads in the past. I thought my growing mysql
>>>>>> tables could be the problem so I set up indexes and it has helped but
>>>>>> my server still struggles under loads it used to handle without issue.
>>>>>> I did notice that my tables are split about 50/50 between InnoDB and
>>>>>> MyISAM and I'd like to make all of them InnoDB.
>>>>>>
>>>>>> - Grant
>>>>>
>>>>> Yeah, I would recommend to switch all them to InnoDB for consistency
>>>>> and real transactions.
>>>>>
>>>>> You probably can find sufficient resources on MySQL optimization with
>>>>> InnoDB if that is your bottleneck.
>>>>
>>>>
>>>> Actually IO doesn't seem to be a bottleneck at this point. It seems
>>>> to be CPU as things slow down once I have 4 busy interchange processes
>>>> on my quad-core CPU and iotop does not show much activity at that
>>>> point. Besides ITL optimization, is this a clear case of needing a
>>>> faster CPU and more cores or is there anything else to consider first?
>>>>
>>>>
>>>>> May I ask which webserver you are using?
>>>>
>>>>
>>>> I'm using nginx reverse proxied to apache2.
>>>
>>> Did you consider to get rid of Apache?
>>
>>
>> I'm certainly planning to do that but if I watch top it looks like my
>> apache2 processes *barely* get above idle even when the web server is
>> under heavy load. The interchange processes get hammered.
>>
>
> Which mode/settings do you use for the Interchange server? How
> do you connect Nginx and Interchange?

Here are my active traffic settings from interchange.cfg:

PreFork Yes
StartServers 12
MaxServers 0
MaxRequestsPerChild 100
HouseKeeping 2
PIDcheck 3600
ChildLife 3600

apache2 listens on a particular port and nginx sends page requests to
apache2 over that port via the proxy_pass directive. apache2 then
hands off to interchange via Interchange::Link.

- Grant

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

mikeh at endpoint

Sep 20, 2016, 6:41 AM

Post #8 of 26 (6253 views)

Permalink

Quoting Grant (emailgrant@gmail.com):
> >>>>>> I've been making a lot of optimizations lately and I think I'm to the
> >>>>>> point where my 4 cores just aren't able to keep up with demand during
> >>>>>> peak traffic hours each day. Pages load quickly when I see fewer than
> >>>>>> 4 busy interchange processes in top but things slow down drastically
> >>>>>> after that. Once I'm OK with the degree of optimization my ITL pages
> >>>>>> have undergone and I'm not IO-bound or memory-bound, is it time to
> >>>>>> throw CPU at the problem?
> >>>>>>
> >>>>>> I'm a little puzzled because I've seen my server perform much better
> >>>>>> under much greater loads in the past. I thought my growing mysql
> >>>>>> tables could be the problem so I set up indexes and it has helped but
> >>>>>> my server still struggles under loads it used to handle without issue.
> >>>>>> I did notice that my tables are split about 50/50 between InnoDB and
> >>>>>> MyISAM and I'd like to make all of them InnoDB.
> >>>>>>
> >>>>>> - Grant
> >>>>>
> >>>>> Yeah, I would recommend to switch all them to InnoDB for consistency
> >>>>> and real transactions.
> >>>>>
> >>>>> You probably can find sufficient resources on MySQL optimization with
> >>>>> InnoDB if that is your bottleneck.
> >>>>
> >>>>
> >>>> Actually IO doesn't seem to be a bottleneck at this point. It seems
> >>>> to be CPU as things slow down once I have 4 busy interchange processes
> >>>> on my quad-core CPU and iotop does not show much activity at that
> >>>> point. Besides ITL optimization, is this a clear case of needing a
> >>>> faster CPU and more cores or is there anything else to consider first?
> >>>>
> >>>>
> >>>>> May I ask which webserver you are using?
> >>>>
> >>>>
> >>>> I'm using nginx reverse proxied to apache2.
> >>>
> >>> Did you consider to get rid of Apache?
> >>
> >>
> >> I'm certainly planning to do that but if I watch top it looks like my
> >> apache2 processes *barely* get above idle even when the web server is
> >> under heavy load. The interchange processes get hammered.
> >>
> >
> > Which mode/settings do you use for the Interchange server? How
> > do you connect Nginx and Interchange?
>
>
> Here are my active traffic settings from interchange.cfg:
>
> PreFork Yes
> StartServers 12
> MaxServers 0
> MaxRequestsPerChild 100
> HouseKeeping 2
> PIDcheck 3600
> ChildLife 3600
>
> apache2 listens on a particular port and nginx sends page requests to
> apache2 over that port via the proxy_pass directive. apache2 then
> hands off to interchange via Interchange::Link.

Apache is almost certainly not your problem.

CPU usage is not always Interchange -- your database uses CPU.

My top suspect is SQL queries that are not optimized. In my experience,
adding a few indices to help key queries usually solves this type of
problem.

--
Mike Heins
End Point -- Expert Internet Consulting http://www.endpoint.com/
phone +1.765.253.4194 <mikeh@endpoint.com>

Fast, reliable, cheap. Pick two and we'll talk.
-- unknown

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

racke at linuxia

Sep 20, 2016, 6:41 AM

Post #9 of 26 (6254 views)

Permalink

On 09/20/2016 03:31 PM, Grant wrote:
>>>>>>> I've been making a lot of optimizations lately and I think I'm to the
>>>>>>> point where my 4 cores just aren't able to keep up with demand during
>>>>>>> peak traffic hours each day. Pages load quickly when I see fewer than
>>>>>>> 4 busy interchange processes in top but things slow down drastically
>>>>>>> after that. Once I'm OK with the degree of optimization my ITL pages
>>>>>>> have undergone and I'm not IO-bound or memory-bound, is it time to
>>>>>>> throw CPU at the problem?
>>>>>>>
>>>>>>> I'm a little puzzled because I've seen my server perform much better
>>>>>>> under much greater loads in the past. I thought my growing mysql
>>>>>>> tables could be the problem so I set up indexes and it has helped but
>>>>>>> my server still struggles under loads it used to handle without issue.
>>>>>>> I did notice that my tables are split about 50/50 between InnoDB and
>>>>>>> MyISAM and I'd like to make all of them InnoDB.
>>>>>>>
>>>>>>> - Grant
>>>>>>
>>>>>> Yeah, I would recommend to switch all them to InnoDB for consistency
>>>>>> and real transactions.
>>>>>>
>>>>>> You probably can find sufficient resources on MySQL optimization with
>>>>>> InnoDB if that is your bottleneck.
>>>>>
>>>>>
>>>>> Actually IO doesn't seem to be a bottleneck at this point. It seems
>>>>> to be CPU as things slow down once I have 4 busy interchange processes
>>>>> on my quad-core CPU and iotop does not show much activity at that
>>>>> point. Besides ITL optimization, is this a clear case of needing a
>>>>> faster CPU and more cores or is there anything else to consider first?
>>>>>
>>>>>
>>>>>> May I ask which webserver you are using?
>>>>>
>>>>>
>>>>> I'm using nginx reverse proxied to apache2.
>>>>
>>>> Did you consider to get rid of Apache?
>>>
>>>
>>> I'm certainly planning to do that but if I watch top it looks like my
>>> apache2 processes *barely* get above idle even when the web server is
>>> under heavy load. The interchange processes get hammered.
>>>
>>
>> Which mode/settings do you use for the Interchange server? How
>> do you connect Nginx and Interchange?
>
>
> Here are my active traffic settings from interchange.cfg:
>
> PreFork Yes
> StartServers 12
> MaxServers 0
> MaxRequestsPerChild 100
> HouseKeeping 2
> PIDcheck 3600
> ChildLife 3600
>
> apache2 listens on a particular port and nginx sends page requests to
> apache2 over that port via the proxy_pass directive. apache2 then
> hands off to interchange via Interchange::Link.

We are using Nginx + FastCGI to connect to Interchange in most of our
installations now.

Regards
Racke

--
Ecommerce and Linux consulting + Perl and web application programming.
Debian and Sympa administration.

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

emailgrant at gmail

Sep 20, 2016, 8:01 AM

Post #10 of 26 (6253 views)

Permalink

>> >>>>>> I've been making a lot of optimizations lately and I think I'm to the
>> >>>>>> point where my 4 cores just aren't able to keep up with demand during
>> >>>>>> peak traffic hours each day. Pages load quickly when I see fewer than
>> >>>>>> 4 busy interchange processes in top but things slow down drastically
>> >>>>>> after that. Once I'm OK with the degree of optimization my ITL pages
>> >>>>>> have undergone and I'm not IO-bound or memory-bound, is it time to
>> >>>>>> throw CPU at the problem?
>> >>>>>>
>> >>>>>> I'm a little puzzled because I've seen my server perform much better
>> >>>>>> under much greater loads in the past. I thought my growing mysql
>> >>>>>> tables could be the problem so I set up indexes and it has helped but
>> >>>>>> my server still struggles under loads it used to handle without issue.
>> >>>>>> I did notice that my tables are split about 50/50 between InnoDB and
>> >>>>>> MyISAM and I'd like to make all of them InnoDB.
>> >>>>>>
>> >>>>>> - Grant
>> >>>>>
>> >>>>> Yeah, I would recommend to switch all them to InnoDB for consistency
>> >>>>> and real transactions.
>> >>>>>
>> >>>>> You probably can find sufficient resources on MySQL optimization with
>> >>>>> InnoDB if that is your bottleneck.
>> >>>>
>> >>>>
>> >>>> Actually IO doesn't seem to be a bottleneck at this point. It seems
>> >>>> to be CPU as things slow down once I have 4 busy interchange processes
>> >>>> on my quad-core CPU and iotop does not show much activity at that
>> >>>> point. Besides ITL optimization, is this a clear case of needing a
>> >>>> faster CPU and more cores or is there anything else to consider first?
>> >>>>
>> >>>>
>> >>>>> May I ask which webserver you are using?
>> >>>>
>> >>>>
>> >>>> I'm using nginx reverse proxied to apache2.
>> >>>
>> >>> Did you consider to get rid of Apache?
>> >>
>> >>
>> >> I'm certainly planning to do that but if I watch top it looks like my
>> >> apache2 processes *barely* get above idle even when the web server is
>> >> under heavy load. The interchange processes get hammered.
>> >>
>> >
>> > Which mode/settings do you use for the Interchange server? How
>> > do you connect Nginx and Interchange?
>>
>>
>> Here are my active traffic settings from interchange.cfg:
>>
>> PreFork Yes
>> StartServers 12
>> MaxServers 0
>> MaxRequestsPerChild 100
>> HouseKeeping 2
>> PIDcheck 3600
>> ChildLife 3600
>>
>> apache2 listens on a particular port and nginx sends page requests to
>> apache2 over that port via the proxy_pass directive. apache2 then
>> hands off to interchange via Interchange::Link.
>
> Apache is almost certainly not your problem.
>
> CPU usage is not always Interchange -- your database uses CPU.

Could the database be to blame if the interchange processes are the
ones using all the CPU in top? Would top show mysql using CPU if the
database were to blame?

> My top suspect is SQL queries that are not optimized. In my experience,
> adding a few indices to help key queries usually solves this type of
> problem.

I added a lot of table indexes lately and it did speed things up a lot
but it didn't solve the problem of http response time slowdowns during
peak traffic hours.

- Grant

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

emailgrant at gmail

Sep 20, 2016, 8:03 AM

Post #11 of 26 (6253 views)

Permalink

>>>>>>>> I've been making a lot of optimizations lately and I think I'm to the
>>>>>>>> point where my 4 cores just aren't able to keep up with demand during
>>>>>>>> peak traffic hours each day. Pages load quickly when I see fewer than
>>>>>>>> 4 busy interchange processes in top but things slow down drastically
>>>>>>>> after that. Once I'm OK with the degree of optimization my ITL pages
>>>>>>>> have undergone and I'm not IO-bound or memory-bound, is it time to
>>>>>>>> throw CPU at the problem?
>>>>>>>>
>>>>>>>> I'm a little puzzled because I've seen my server perform much better
>>>>>>>> under much greater loads in the past. I thought my growing mysql
>>>>>>>> tables could be the problem so I set up indexes and it has helped but
>>>>>>>> my server still struggles under loads it used to handle without issue.
>>>>>>>> I did notice that my tables are split about 50/50 between InnoDB and
>>>>>>>> MyISAM and I'd like to make all of them InnoDB.
>>>>>>>>
>>>>>>>> - Grant
>>>>>>>
>>>>>>> Yeah, I would recommend to switch all them to InnoDB for consistency
>>>>>>> and real transactions.
>>>>>>>
>>>>>>> You probably can find sufficient resources on MySQL optimization with
>>>>>>> InnoDB if that is your bottleneck.
>>>>>>
>>>>>>
>>>>>> Actually IO doesn't seem to be a bottleneck at this point. It seems
>>>>>> to be CPU as things slow down once I have 4 busy interchange processes
>>>>>> on my quad-core CPU and iotop does not show much activity at that
>>>>>> point. Besides ITL optimization, is this a clear case of needing a
>>>>>> faster CPU and more cores or is there anything else to consider first?
>>>>>>
>>>>>>
>>>>>>> May I ask which webserver you are using?
>>>>>>
>>>>>>
>>>>>> I'm using nginx reverse proxied to apache2.
>>>>>
>>>>> Did you consider to get rid of Apache?
>>>>
>>>>
>>>> I'm certainly planning to do that but if I watch top it looks like my
>>>> apache2 processes *barely* get above idle even when the web server is
>>>> under heavy load. The interchange processes get hammered.
>>>>
>>>
>>> Which mode/settings do you use for the Interchange server? How
>>> do you connect Nginx and Interchange?
>>
>>
>> Here are my active traffic settings from interchange.cfg:
>>
>> PreFork Yes
>> StartServers 12
>> MaxServers 0
>> MaxRequestsPerChild 100
>> HouseKeeping 2
>> PIDcheck 3600
>> ChildLife 3600
>>
>> apache2 listens on a particular port and nginx sends page requests to
>> apache2 over that port via the proxy_pass directive. apache2 then
>> hands off to interchange via Interchange::Link.
>
> We are using Nginx + FastCGI to connect to Interchange in most of our
> installations now.

That's where I want to end up. Is there good info online on how to
connect nginx and IC directly?

- Grant

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

racke at linuxia

Sep 20, 2016, 9:16 AM

Post #12 of 26 (6246 views)

Permalink

I will take care of that after the Perl Dancer conference

Regards Racke

On 20 September 2016 17:03:48 CEST, Grant <emailgrant@gmail.com> wrote:
>>>>>>>>> I've been making a lot of optimizations lately and I think I'm
>to the
>>>>>>>>> point where my 4 cores just aren't able to keep up with demand
>during
>>>>>>>>> peak traffic hours each day. Pages load quickly when I see
>fewer than
>>>>>>>>> 4 busy interchange processes in top but things slow down
>drastically
>>>>>>>>> after that. Once I'm OK with the degree of optimization my
>ITL pages
>>>>>>>>> have undergone and I'm not IO-bound or memory-bound, is it
>time to
>>>>>>>>> throw CPU at the problem?
>>>>>>>>>
>>>>>>>>> I'm a little puzzled because I've seen my server perform much
>better
>>>>>>>>> under much greater loads in the past. I thought my growing
>mysql
>>>>>>>>> tables could be the problem so I set up indexes and it has
>helped but
>>>>>>>>> my server still struggles under loads it used to handle
>without issue.
>>>>>>>>> I did notice that my tables are split about 50/50 between
>InnoDB and
>>>>>>>>> MyISAM and I'd like to make all of them InnoDB.
>>>>>>>>>
>>>>>>>>> - Grant
>>>>>>>>
>>>>>>>> Yeah, I would recommend to switch all them to InnoDB for
>consistency
>>>>>>>> and real transactions.
>>>>>>>>
>>>>>>>> You probably can find sufficient resources on MySQL
>optimization with
>>>>>>>> InnoDB if that is your bottleneck.
>>>>>>>
>>>>>>>
>>>>>>> Actually IO doesn't seem to be a bottleneck at this point. It
>seems
>>>>>>> to be CPU as things slow down once I have 4 busy interchange
>processes
>>>>>>> on my quad-core CPU and iotop does not show much activity at
>that
>>>>>>> point. Besides ITL optimization, is this a clear case of
>needing a
>>>>>>> faster CPU and more cores or is there anything else to consider
>first?
>>>>>>>
>>>>>>>
>>>>>>>> May I ask which webserver you are using?
>>>>>>>
>>>>>>>
>>>>>>> I'm using nginx reverse proxied to apache2.
>>>>>>
>>>>>> Did you consider to get rid of Apache?
>>>>>
>>>>>
>>>>> I'm certainly planning to do that but if I watch top it looks like
>my
>>>>> apache2 processes *barely* get above idle even when the web server
>is
>>>>> under heavy load. The interchange processes get hammered.
>>>>>
>>>>
>>>> Which mode/settings do you use for the Interchange server? How
>>>> do you connect Nginx and Interchange?
>>>
>>>
>>> Here are my active traffic settings from interchange.cfg:
>>>
>>> PreFork Yes
>>> StartServers 12
>>> MaxServers 0
>>> MaxRequestsPerChild 100
>>> HouseKeeping 2
>>> PIDcheck 3600
>>> ChildLife 3600
>>>
>>> apache2 listens on a particular port and nginx sends page requests
>to
>>> apache2 over that port via the proxy_pass directive. apache2 then
>>> hands off to interchange via Interchange::Link.
>>
>> We are using Nginx + FastCGI to connect to Interchange in most of our
>> installations now.
>
>
>That's where I want to end up. Is there good info online on how to
>connect nginx and IC directly?
>
>- Grant
>
>_______________________________________________
>interchange-users mailing list
>interchange-users@icdevgroup.org
>http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

jlavin at endpoint

Sep 21, 2016, 8:22 AM

Post #13 of 26 (6245 views)

Permalink

> > We are using Nginx + FastCGI to connect to Interchange in most of our
> > installations now.
>
> That's where I want to end up. Is there good info online on how to
> connect nginx and IC directly?

I tried it once:
https://gist.github.com/jdigory/50fc9cf94cd7ea5e8e4f

--
Josh Lavin
End Point Corporation

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

emailgrant at gmail

Sep 21, 2016, 8:40 AM

Post #14 of 26 (6245 views)

Permalink

>> > We are using Nginx + FastCGI to connect to Interchange in most of our
>> > installations now.
>>
>> That's where I want to end up. Is there good info online on how to
>> connect nginx and IC directly?
>
> I tried it once:
> https://gist.github.com/jdigory/50fc9cf94cd7ea5e8e4f

Very cool! Did you like how it turned out or did you switch back?

Should I be able to have URLs like example.com/page.html without any
folders between the domain and page?

- Grant

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

jon at endpoint

Sep 21, 2016, 8:46 AM

Post #15 of 26 (6245 views)

Permalink

On Wed, 21 Sep 2016, Grant wrote:

>>>> We are using Nginx + FastCGI to connect to Interchange in most of our
>>>> installations now.
>
> Should I be able to have URLs like example.com/page.html without any
> folders between the domain and page?

You can do that now, using classic CGI vlink and mod_rewrite.

I don't recommend it, though, because it means *every* bot and 404 request
goes through Interchange rather than being served statically by Apache,
greatly increasing the load on Interchange and your server.

Jon

--
Jon Jensen
End Point Corporation
https://www.endpoint.com/

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

jlavin at endpoint

Sep 21, 2016, 9:11 AM

Post #16 of 26 (6246 views)

Permalink

Quoting Grant (emailgrant@gmail.com):
> >> > We are using Nginx + FastCGI to connect to Interchange in most of our
> >> > installations now.
> >>
> >> That's where I want to end up. Is there good info online on how to
> >> connect nginx and IC directly?
> >
> > I tried it once:
> > https://gist.github.com/jdigory/50fc9cf94cd7ea5e8e4f
>
>
> Very cool! Did you like how it turned out or did you switch back?

I'm still running a Strap demo site on it. It's not production level,
but it works in that case. I'll be interested to see Racke's version.

> Should I be able to have URLs like example.com/page.html without any
> folders between the domain and page?

Yes, although that's not how my example works. You'd need to change the
last "location" setting, I think.

--
Josh Lavin
End Point Corporation

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

jlavin at endpoint

Sep 21, 2016, 9:33 AM

Post #17 of 26 (6245 views)

Permalink

Quoting Jon Jensen (jon@endpoint.com):
> On Wed, 21 Sep 2016, Grant wrote:
>
> > > > > We are using Nginx + FastCGI to connect to Interchange in
> > > > > most of our installations now.
> >
> > Should I be able to have URLs like example.com/page.html without any
> > folders between the domain and page?
>
> You can do that now, using classic CGI vlink and mod_rewrite.
>
> I don't recommend it, though, because it means *every* bot and 404 request
> goes through Interchange rather than being served statically by Apache,
> greatly increasing the load on Interchange and your server.

Right -- I may have posted this to the list once before, but here's what
I have typically used to avoid that, and only send to IC the requests it
would normally handle:

# Here is how I do a catalog at the root URL. This method prevents
# Interchange from having to handle all 404 requests, unless the request
# does not have an extension, or is a known file type that IC serves.

ScriptAlias /cgi-bin/ /home/mysite/cgi-bin/

RewriteEngine On

RewriteRule ^/index(\.html?)?$ / [R=301]
RewriteRule ^/?$ /cgi-bin/mysite/index [L,NS,PT]

RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} -d [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} -f
RewriteRule ^ - [L,NS]

RewriteCond %{REQUEST_FILENAME} \.
RewriteCond %{REQUEST_FILENAME} !\.(?:html|txt|xml|pdf|iif)$
RewriteRule ^ - [L,NS]

RewriteCond %{REQUEST_FILENAME} !^/cgi-bin/.+
RewriteRule ^(.+)$ /cgi-bin/mysite$1 [L,NS,PT]

https://github.com/jdigory/interchange-extras/blob/master/eg/httpd-example.conf

--
Josh Lavin
End Point Corporation

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

emailgrant at gmail

Sep 21, 2016, 9:49 AM

Post #18 of 26 (6245 views)

Permalink

>>>>> We are using Nginx + FastCGI to connect to Interchange in most of our
>>>>> installations now.
>>
>>
>> Should I be able to have URLs like example.com/page.html without any
>> folders between the domain and page?
>
>
> You can do that now, using classic CGI vlink and mod_rewrite.

Or via Interchange::Link which is how I do it currently.

> I don't recommend it, though, because it means *every* bot and 404 request
> goes through Interchange rather than being served statically by Apache,
> greatly increasing the load on Interchange and your server.

With nginx you can easily catch requests that don't match your URL
scheme and 'return 404;' without ever handing off to
apache/interchange.

- Grant

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

emailgrant at gmail

Sep 21, 2016, 9:52 AM

Post #19 of 26 (6246 views)

Permalink

>> >> > We are using Nginx + FastCGI to connect to Interchange in most of our
>> >> > installations now.
>> >>
>> >> That's where I want to end up. Is there good info online on how to
>> >> connect nginx and IC directly?
>> >
>> > I tried it once:
>> > https://gist.github.com/jdigory/50fc9cf94cd7ea5e8e4f
>>
>>
>> Very cool! Did you like how it turned out or did you switch back?
>
> I'm still running a Strap demo site on it. It's not production level,
> but it works in that case. I'll be interested to see Racke's version.

Me too. Looking forward to it Stefan!

- Grant

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

jon at endpoint

Sep 21, 2016, 9:55 AM

Post #20 of 26 (6246 views)

Permalink

On Wed, 21 Sep 2016, Grant wrote:

>> I don't recommend it, though, because it means *every* bot and 404
>> request goes through Interchange rather than being served statically by
>> Apache, greatly increasing the load on Interchange and your server.
>
> With nginx you can easily catch requests that don't match your URL
> scheme and 'return 404;' without ever handing off to apache/interchange.

Sure, if you have a URL scheme then you can route only that to
Interchange.

Most people wanting the root URL space to be handled by Interchange are
going to have trouble knowing what is and isn't an app-handled URL, if it
includes any arbitrary product ID:

/super-mega-bike
/deluxe-special-computer

etc.

Either you have to make a huge list of all known URLs, or look them up in
the database which isn't generally any faster than letting it hit
Interchange.

If you have known URL prefixes for your app:

/shop/super-mega-bike
/admin/login

then it's no problem, but that's not what "no URL prefix" means to me.

Jon

--
Jon Jensen
End Point Corporation
https://www.endpoint.com/

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

emailgrant at gmail

Sep 21, 2016, 10:05 AM

Post #21 of 26 (6246 views)

Permalink

>>> I don't recommend it, though, because it means *every* bot and 404
>>> request goes through Interchange rather than being served statically by
>>> Apache, greatly increasing the load on Interchange and your server.
>>
>>
>> With nginx you can easily catch requests that don't match your URL scheme
>> and 'return 404;' without ever handing off to apache/interchange.
>
>
> Sure, if you have a URL scheme then you can route only that to Interchange.
>
> Most people wanting the root URL space to be handled by Interchange are
> going to have trouble knowing what is and isn't an app-handled URL, if it
> includes any arbitrary product ID:
>
> /super-mega-bike
> /deluxe-special-computer
>
> etc.
>
> Either you have to make a huge list of all known URLs, or look them up in
> the database which isn't generally any faster than letting it hit
> Interchange.
>
> If you have known URL prefixes for your app:
>
> /shop/super-mega-bike
> /admin/login
>
> then it's no problem, but that's not what "no URL prefix" means to me.

If you can name all of your pages with the same extension (.html for
example) then you can throw out anything else. That and trashing any
request which includes more than one slash (if you don't use folders)
gets rid of a whole lot of junk. You can make exceptions for /admin/
and whatever else.

- Grant

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

jlavin at endpoint

Sep 21, 2016, 10:27 AM

Post #22 of 26 (6223 views)

Permalink

Quoting Grant (emailgrant@gmail.com):
> >>> I don't recommend it, though, because it means *every* bot and 404
> >>> request goes through Interchange rather than being served statically by
> >>> Apache, greatly increasing the load on Interchange and your server.
> >>
> >>
> >> With nginx you can easily catch requests that don't match your URL scheme
> >> and 'return 404;' without ever handing off to apache/interchange.
> >
> >
> > Sure, if you have a URL scheme then you can route only that to Interchange.
> >
> > Most people wanting the root URL space to be handled by Interchange are
> > going to have trouble knowing what is and isn't an app-handled URL, if it
> > includes any arbitrary product ID:
> >
> > /super-mega-bike
> > /deluxe-special-computer
> >
> > etc.
> >
> > Either you have to make a huge list of all known URLs, or look them up in
> > the database which isn't generally any faster than letting it hit
> > Interchange.
> >
> > If you have known URL prefixes for your app:
> >
> > /shop/super-mega-bike
> > /admin/login
> >
> > then it's no problem, but that's not what "no URL prefix" means to me.
>
>
> If you can name all of your pages with the same extension (.html for
> example) then you can throw out anything else. That and trashing any
> request which includes more than one slash (if you don't use folders)
> gets rid of a whole lot of junk. You can make exceptions for /admin/
> and whatever else.

Remember, most of the 404 stuff is PHP hacking attempts anyways.

--
Josh Lavin
End Point Corporation

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

peter at pajamian

Sep 21, 2016, 12:57 PM

Post #23 of 26 (6223 views)

Permalink

On 22/09/16 04:55, Jon Jensen wrote:
> Sure, if you have a URL scheme then you can route only that to Interchange.
>
> Most people wanting the root URL space to be handled by Interchange are
> going to have trouble knowing what is and isn't an app-handled URL, if
> it includes any arbitrary product ID:

The general strategy I employ here is to route a few select directories
statically, eg: js, css, images, etc and one named "static" as a kind of
place to be able to stick anything else static in without having to
create new aliases in apache or nginx. Everything else gets routed to
IC. This works out well and with a few other tweaks can show short,
clean URLs for IC products.

Keep in mind that URLs that you're going to link to, that are shown in
the location bar, and that you might ask someone to hand-type are
generally IC URLs and supporting resources (images, CSS, JS files, etc)
are what you want to be served statically by your web server. I tend to
like the more visible URLs to be short and neat.

Peter

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

racke at linuxia

Sep 22, 2016, 12:16 AM

Post #24 of 26 (6215 views)

Permalink

On 09/21/2016 06:52 PM, Grant wrote:
>>>>>> We are using Nginx + FastCGI to connect to Interchange in most of our
>>>>>> installations now.
>>>>>
>>>>> That's where I want to end up. Is there good info online on how to
>>>>> connect nginx and IC directly?
>>>>
>>>> I tried it once:
>>>> https://gist.github.com/jdigory/50fc9cf94cd7ea5e8e4f
>>>
>>>
>>> Very cool! Did you like how it turned out or did you switch back?
>>
>> I'm still running a Strap demo site on it. It's not production level,
>> but it works in that case. I'll be interested to see Racke's version.
>
>
> Me too. Looking forward to it Stefan!

My version is similar to Josh' ones. If you are using this setup with Fast CGI,
you need to make sure that enough FastCGI processes are available.

More later ...

Regards
Racke

--
Ecommerce and Linux consulting + Perl and web application programming.
Debian and Sympa administration.

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users

Re: Time for new hardware? [ In reply to ]

emailgrant at gmail

Sep 24, 2016, 5:35 PM

Post #25 of 26 (6209 views)

Permalink

>>>> I've been making a lot of optimizations lately and I think I'm to the
>>>> point where my 4 cores just aren't able to keep up with demand during
>>>> peak traffic hours each day. Pages load quickly when I see fewer than
>>>> 4 busy interchange processes in top but things slow down drastically
>>>> after that. Once I'm OK with the degree of optimization my ITL pages
>>>> have undergone and I'm not IO-bound or memory-bound, is it time to
>>>> throw CPU at the problem?
>>>>
>>>> I'm a little puzzled because I've seen my server perform much better
>>>> under much greater loads in the past. I thought my growing mysql
>>>> tables could be the problem so I set up indexes and it has helped but
>>>> my server still struggles under loads it used to handle without issue.
>>>> I did notice that my tables are split about 50/50 between InnoDB and
>>>> MyISAM and I'd like to make all of them InnoDB.
>>>>
>>>> - Grant
>>>
>>> Yeah, I would recommend to switch all them to InnoDB for consistency
>>> and real transactions.
>>>
>>> You probably can find sufficient resources on MySQL optimization with
>>> InnoDB if that is your bottleneck.
>>
>>
>> Actually IO doesn't seem to be a bottleneck at this point. It seems
>> to be CPU as things slow down once I have 4 busy interchange processes
>> on my quad-core CPU and iotop does not show much activity at that
>> point. Besides ITL optimization, is this a clear case of needing a
>> faster CPU and more cores or is there anything else to consider first?
>>
>>
>>> May I ask which webserver you are using?
>>
>>
>> I'm using nginx reverse proxied to apache2.
>>
>> - Grant
>
> Did you consider to get rid of Apache?

It turned out apache2 was part of the problem. This was a combination
of not enough apache2 processes as well as Odoo (formerly known as
OpenERP) causing some kind of issue I haven't figured out yet. The
fact that there were two problems made it much more difficult to
figure out.

I was surprised I needed more apache2 processes because mine are
always idle or nearly idle, even during periods of high traffic. With
nginx acting as a reverse proxy to apache2, I think each request that
hits interchange monopolizes an apache2 processes even if that process
isn't using CPU.

Odoo also operates behind an nginx reverse proxy on my machine and it
causes the TCP Queuing graph in munin to go crazy and disrupts
interchange service whenever someone is using it. I still don't know
why.

- Grant

_______________________________________________
interchange-users mailing list
interchange-users@icdevgroup.org
http://www.icdevgroup.org/mailman/listinfo/interchange-users