Mailing List Archive: [Swift] PUT Object performance problem

[Swift] PUT Object performance problem

Oct 29, 2018, 6:07 AM

Post #1 of 4 (724 views)

Hello, All

I have a terrible problem with object server.
Here is the case.
1. User upload an object to proxy-server
2. Proxy server try to connect with object-server
3. If one of object-server is slow to respond, proxy-server is waiting for
response.
3.1 While waiting for response, proxy-server can't do anything
4. So, The response of client request will be delayed.

In my opinion, this code seems to be a problem
(
https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L1734
)

```
with ResponseTimeout(node_timeout):
resp = conn.getexpect()
```

If node_timeout's value is 3 and object-server respond after 2 seconds,
proxy-server wait 2 seconds.

Because proxy-server wait for the above response, the execution of the
following code is delayed.
(
https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L627
)

```
for node in nodes:
try:
putter = self._make_putter(node, part, req, headers)
self.app.set_node_timing(node, putter.connect_duration)
return putter
```

This problem occurs when i do a ring rebalance.
When object-replicator delete a partition directory that are no longer
mine, the disk becomes very busy (Because of xfsaild daemon)
Because the disk are busy, object-server can't create diskfile during PUT
operation.

Is there anyone who is having problems like me?
How can I solve this problem?

I need everyone's help.
Thanks.

Best Regards
SeongSoo Cho

------
SeongSoo Cho (South Korea)

Re: [Swift] PUT Object performance problem [ In reply to ]

ppiyakk2 at printf

Oct 29, 2018, 7:45 AM

Post #2 of 4 (723 views)

Permalink

For more information, I use ocata version.

2018? 10? 29? (?) ?? 10:07, SeongSoo Cho <ppiyakk2@printf.kr>?? ??:

> Hello, All
>
> I have a terrible problem with object server.
> Here is the case.
> 1. User upload an object to proxy-server
> 2. Proxy server try to connect with object-server
> 3. If one of object-server is slow to respond, proxy-server is waiting for
> response.
> 3.1 While waiting for response, proxy-server can't do anything
> 4. So, The response of client request will be delayed.
>
> In my opinion, this code seems to be a problem
> (
> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L1734
> )
>
> ```
> with ResponseTimeout(node_timeout):
> resp = conn.getexpect()
> ```
>
> If node_timeout's value is 3 and object-server respond after 2 seconds,
> proxy-server wait 2 seconds.
>
> Because proxy-server wait for the above response, the execution of the
> following code is delayed.
> (
> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L627
> )
>
> ```
> for node in nodes:
> try:
> putter = self._make_putter(node, part, req, headers)
> self.app.set_node_timing(node, putter.connect_duration)
> return putter
> ```
>
> This problem occurs when i do a ring rebalance.
> When object-replicator delete a partition directory that are no longer
> mine, the disk becomes very busy (Because of xfsaild daemon)
> Because the disk are busy, object-server can't create diskfile during PUT
> operation.
>
> Is there anyone who is having problems like me?
> How can I solve this problem?
>
> I need everyone's help.
> Thanks.
>
> Best Regards
> SeongSoo Cho
>
> ------
> SeongSoo Cho (South Korea)
>
>

Re: [Swift] PUT Object performance problem [ In reply to ]

clay.gerrard at gmail

Oct 29, 2018, 9:42 AM

Post #3 of 4 (722 views)

Permalink

Obviously a re-balance will cost some IO, but it's normally perceptible to
the client unless you were already on a razor thin line.

Two config options seem obvious to think about experimenting with:

You could decrease the node_timeout and let the proxy try to write more to
handoffs
You could try to use some of the ionice options (or other tuning options)
to make the replicators hammer the disks a little less

It might having something to do with how you're organizing your ring weight
changes - could you describe how you're managing rings?
Could also be a io scheduling issue - are you using noop/deadline or cfq?
What kernel version?

-Clay

On Mon, Oct 29, 2018 at 9:55 AM SeongSoo Cho <ppiyakk2@printf.kr> wrote:

> For more information, I use ocata version.
>
> 2018? 10? 29? (?) ?? 10:07, SeongSoo Cho <ppiyakk2@printf.kr>?? ??:
>
>> Hello, All
>>
>> I have a terrible problem with object server.
>> Here is the case.
>> 1. User upload an object to proxy-server
>> 2. Proxy server try to connect with object-server
>> 3. If one of object-server is slow to respond, proxy-server is waiting
>> for response.
>> 3.1 While waiting for response, proxy-server can't do anything
>> 4. So, The response of client request will be delayed.
>>
>> In my opinion, this code seems to be a problem
>> (
>> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L1734
>> )
>>
>> ```
>> with ResponseTimeout(node_timeout):
>> resp = conn.getexpect()
>> ```
>>
>> If node_timeout's value is 3 and object-server respond after 2 seconds,
>> proxy-server wait 2 seconds.
>>
>> Because proxy-server wait for the above response, the execution of the
>> following code is delayed.
>> (
>> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L627
>> )
>>
>> ```
>> for node in nodes:
>> try:
>> putter = self._make_putter(node, part, req, headers)
>> self.app.set_node_timing(node, putter.connect_duration)
>> return putter
>> ```
>>
>> This problem occurs when i do a ring rebalance.
>> When object-replicator delete a partition directory that are no longer
>> mine, the disk becomes very busy (Because of xfsaild daemon)
>> Because the disk are busy, object-server can't create diskfile during PUT
>> operation.
>>
>> Is there anyone who is having problems like me?
>> How can I solve this problem?
>>
>> I need everyone's help.
>> Thanks.
>>
>> Best Regards
>> SeongSoo Cho
>>
>> ------
>> SeongSoo Cho (South Korea)
>>
>> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>

Re: [Swift] PUT Object performance problem [ In reply to ]

ppiyakk2 at printf

Oct 29, 2018, 6:45 PM

Post #4 of 4 (722 views)

Permalink

Hi clay

I just using the command like `swift-ring-builder object.builder
set_weight d234 2220`.
Then execute `swift-ring-builder object.builder rebalance` and push the
rings all object node.

Here is more detail information.
I/O scheduler : deadline
OS : CentOS 7.4
Kernel version : 3.10.0-693.11.1.el7.x86_64

I will try decrease the node_timeout from 3 to 1. (Do you have any
recommend value?)

On Tue, Oct 30, 2018 at 1:42 AM Clay Gerrard <clay.gerrard@gmail.com> wrote:

> Obviously a re-balance will cost some IO, but it's normally perceptible to
> the client unless you were already on a razor thin line.
>
> Two config options seem obvious to think about experimenting with:
>
> You could decrease the node_timeout and let the proxy try to write more to
> handoffs
> You could try to use some of the ionice options (or other tuning options)
> to make the replicators hammer the disks a little less
>
> It might having something to do with how you're organizing your ring
> weight changes - could you describe how you're managing rings?
> Could also be a io scheduling issue - are you using noop/deadline or cfq?
> What kernel version?
>
> -Clay
>
>
> On Mon, Oct 29, 2018 at 9:55 AM SeongSoo Cho <ppiyakk2@printf.kr> wrote:
>
>> For more information, I use ocata version.
>>
>> 2018? 10? 29? (?) ?? 10:07, SeongSoo Cho <ppiyakk2@printf.kr>?? ??:
>>
>>> Hello, All
>>>
>>> I have a terrible problem with object server.
>>> Here is the case.
>>> 1. User upload an object to proxy-server
>>> 2. Proxy server try to connect with object-server
>>> 3. If one of object-server is slow to respond, proxy-server is waiting
>>> for response.
>>> 3.1 While waiting for response, proxy-server can't do anything
>>> 4. So, The response of client request will be delayed.
>>>
>>> In my opinion, this code seems to be a problem
>>> (
>>> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L1734
>>> )
>>>
>>> ```
>>> with ResponseTimeout(node_timeout):
>>> resp = conn.getexpect()
>>> ```
>>>
>>> If node_timeout's value is 3 and object-server respond after 2 seconds,
>>> proxy-server wait 2 seconds.
>>>
>>> Because proxy-server wait for the above response, the execution of the
>>> following code is delayed.
>>> (
>>> https://github.com/openstack/swift/blob/stable/rocky/swift/proxy/controllers/obj.py#L627
>>> )
>>>
>>> ```
>>> for node in nodes:
>>> try:
>>> putter = self._make_putter(node, part, req, headers)
>>> self.app.set_node_timing(node, putter.connect_duration)
>>> return putter
>>> ```
>>>
>>> This problem occurs when i do a ring rebalance.
>>> When object-replicator delete a partition directory that are no longer
>>> mine, the disk becomes very busy (Because of xfsaild daemon)
>>> Because the disk are busy, object-server can't create diskfile during
>>> PUT operation.
>>>
>>> Is there anyone who is having problems like me?
>>> How can I solve this problem?
>>>
>>> I need everyone's help.
>>> Thanks.
>>>
>>> Best Regards
>>> SeongSoo Cho
>>>
>>> ------
>>> SeongSoo Cho (South Korea)
>>>
>>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>