Mailing List Archive

qtree snapmirror inexplicably slow once a week
Hi folks,

I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site that's slated to be decommed, but for now, I need to keep the lights on. There is a qtree snapmirror that, aside from Mondays, runs in about 3-4 hours. Mondays, it takes 2-4 days. The change rate on the source doesn't appear to be any different. Also, this was working happily for years and only started acting up lately. There are only 5 disks on the system, and while they're at 100% now, they're also at 100% during a sample taken while this context wasn't updating, and while it was updating in a way that finished in 4 hours.

So far, I've checked statit during the event and while idle, and I couldn't see anything that stuck out as a massive difference, except maybe name cache hits? I have 68% name cache hits during this transfer, and 100% while idle.

I don't see anything suspicious in the wafl scan status, and reallocate status -v shows:

Reallocation scans are on
No reallocation status.

CPU is idle, network has no errors. Does anyone have any suggestions? What kinds of things should I be looking at?

Thanks,

Basil

*************************************************************************
This message and any attachments (the "message") are confidential, intended
solely for the addressee(s), and may contain legally privileged information.
Any unauthorized use or dissemination is prohibited. E-mails are susceptible
to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or
affiliates shall be liable for the message if altered, changed or falsified.
Please visit http://sgasdisclosure.com for important information regarding
SG Americas Securities, LLC (“SGAS”). Please visit http://swapdisclosure.sgcib.com
for important information regarding swap transactions with SOCIETE GENERALE.
*************************************************************************
Re: qtree snapmirror inexplicably slow once a week [ In reply to ]
Are there any other snapmirror qtrees happening on the same volume on
Mondays. If yes the locks on snapshots might be the problem.

J


On 12/13/2016 11:04 AM, BERNTSEN Basil wrote:
> Hi folks,
>
>
>
> I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site
> that's slated to be decommed, but for now, I need to keep the lights on.
> There is a qtree snapmirror that, aside from Mondays, runs in about 3-4
> hours. Mondays, it takes 2-4 days. The change rate on the source doesn't
> appear to be any different. Also, this was working happily for years and
> only started acting up lately. There are only 5 disks on the system, and
> while they're at 100% now, they're also at 100% during a sample taken
> while this context wasn't updating, and while it was updating in a way
> that finished in 4 hours.
>
>
>
> So far, I've checked statit during the event and while idle, and I
> couldn't see anything that stuck out as a massive difference, except
> maybe name cache hits? I have 68% name cache hits during this transfer,
> and 100% while idle.
>
>
>
> I don't see anything suspicious in the wafl scan status, and reallocate
> status -v shows:
>
>
>
> Reallocation scans are on
>
> No reallocation status.
>
>
>
> CPU is idle, network has no errors. Does anyone have any suggestions?
> What kinds of things should I be looking at?
>
>
>
> Thanks,
>
>
>
> Basil
>
>
>
> *************************************************************************
> This message and any attachments (the "message") are confidential,
> intended solely for the addressee(s), and may contain legally privileged
> information. Any unauthorized use or dissemination is prohibited.
> E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any
> of its subsidiaries or affiliates shall be liable for the message if
> altered, changed or falsified. Please visit http://sgasdisclosure.com
> for important information regarding SG Americas Securities, LLC
> (“SGAS”). Please visit http://swapdisclosure.sgcib.com for important
> information regarding swap transactions with SOCIETE GENERALE.
> *************************************************************************
>
>
>
> _______________________________________________
> Toasters mailing list
> Toasters@teaparty.net
> http://www.teaparty.net/mailman/listinfo/toasters
>


--
Joel Krajden: Core Technologies Manager
Engineering & Computer Science Concordia University
EV-8.142 Tel: 514 848-2424 3052
joelk@encs.concordia.ca


_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
Re: qtree snapmirror inexplicably slow once a week [ In reply to ]
Check the etc/log/snapmirror file on the destination filer for clues.

J

On 12/14/2016 03:28 PM, Joel Krajden wrote:
> Are there any other snapmirror qtrees happening on the same volume on
> Mondays. If yes the locks on snapshots might be the problem.
>
> J
>
>
> On 12/13/2016 11:04 AM, BERNTSEN Basil wrote:
>> Hi folks,
>>
>>
>>
>> I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site
>> that's slated to be decommed, but for now, I need to keep the lights on.
>> There is a qtree snapmirror that, aside from Mondays, runs in about 3-4
>> hours. Mondays, it takes 2-4 days. The change rate on the source doesn't
>> appear to be any different. Also, this was working happily for years and
>> only started acting up lately. There are only 5 disks on the system, and
>> while they're at 100% now, they're also at 100% during a sample taken
>> while this context wasn't updating, and while it was updating in a way
>> that finished in 4 hours.
>>
>>
>>
>> So far, I've checked statit during the event and while idle, and I
>> couldn't see anything that stuck out as a massive difference, except
>> maybe name cache hits? I have 68% name cache hits during this transfer,
>> and 100% while idle.
>>
>>
>>
>> I don't see anything suspicious in the wafl scan status, and reallocate
>> status -v shows:
>>
>>
>>
>> Reallocation scans are on
>>
>> No reallocation status.
>>
>>
>>
>> CPU is idle, network has no errors. Does anyone have any suggestions?
>> What kinds of things should I be looking at?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Basil
>>
>>
>>
>> *************************************************************************
>> This message and any attachments (the "message") are confidential,
>> intended solely for the addressee(s), and may contain legally privileged
>> information. Any unauthorized use or dissemination is prohibited.
>> E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any
>> of its subsidiaries or affiliates shall be liable for the message if
>> altered, changed or falsified. Please visit http://sgasdisclosure.com
>> for important information regarding SG Americas Securities, LLC
>> (“SGAS”). Please visit http://swapdisclosure.sgcib.com for important
>> information regarding swap transactions with SOCIETE GENERALE.
>> *************************************************************************
>>
>>
>>
>> _______________________________________________
>> Toasters mailing list
>> Toasters@teaparty.net
>> http://www.teaparty.net/mailman/listinfo/toasters
>>
>
>


--
Joel Krajden: Core Technologies Manager
Engineering & Computer Science Concordia University
EV-8.142 Tel: 514 848-2424 3052
joelk@encs.concordia.ca


_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
Re: qtree snapmirror inexplicably slow once a week [ In reply to ]
because the disks were at 100% while the snapmirror is slow and also 100%
while the snapmirror finished in an acceptable time frame, does not mean
you have a similar workload. there is nowhere to go from there.
You cant get many iops out of 5 disks, my first though would be to look at
the clients, see which one is the hog and find out what it is doing. I
suppose the first thing I would do is see what protocol is doing the i/o
and go from there.



sysstat -x 1

might be useful for starters.

--JMS

On Tue, Dec 13, 2016 at 11:04 AM, BERNTSEN Basil <
basil.berntsen-ext@socgen.com> wrote:

> Hi folks,
>
>
>
> I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site
> that's slated to be decommed, but for now, I need to keep the lights on.
> There is a qtree snapmirror that, aside from Mondays, runs in about 3-4
> hours. Mondays, it takes 2-4 days. The change rate on the source doesn't
> appear to be any different. Also, this was working happily for years and
> only started acting up lately. There are only 5 disks on the system, and
> while they're at 100% now, they're also at 100% during a sample taken while
> this context wasn't updating, and while it was updating in a way that
> finished in 4 hours.
>
>
>
> So far, I've checked statit during the event and while idle, and I
> couldn't see anything that stuck out as a massive difference, except maybe
> name cache hits? I have 68% name cache hits during this transfer, and 100%
> while idle.
>
>
>
> I don't see anything suspicious in the wafl scan status, and reallocate
> status -v shows:
>
>
>
> Reallocation scans are on
>
> No reallocation status.
>
>
>
> CPU is idle, network has no errors. Does anyone have any suggestions? What
> kinds of things should I be looking at?
>
>
>
> Thanks,
>
>
>
> Basil
>
>
>
> *************************************************************************
> This message and any attachments (the "message") are confidential,
> intended solely for the addressee(s), and may contain legally privileged
> information. Any unauthorized use or dissemination is prohibited.
> E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any
> of its subsidiaries or affiliates shall be liable for the message if
> altered, changed or falsified. Please visit http://sgasdisclosure.com for
> important information regarding SG Americas Securities, LLC (“SGAS”).
> Please visit http://swapdisclosure.sgcib.com for important information
> regarding swap transactions with SOCIETE GENERALE.
> *************************************************************************
>
> _______________________________________________
> Toasters mailing list
> Toasters@teaparty.net
> http://www.teaparty.net/mailman/listinfo/toasters
>
>