Mailing List Archive: Slow linstor queries after recent upgrade

Slow linstor queries after recent upgrade

May 7, 2023, 5:45 AM

Post #1 of 4 (209 views)

Re: Slow linstor queries after recent upgrade [ In reply to ]

roland.kammerer at linbit

May 8, 2023, 12:46 AM

Post #2 of 4 (208 views)

Permalink

On Sun, May 07, 2023 at 02:45:53PM +0200, ?ukasz W?sikowski wrote:
> Hi,
>
> I'm using Linstor on 5 node Proxmox 7.x cluster. After recent upgrade I've
> got some performance issues with some linstor queries.
>
> Upgrade: linstor-common:amd64 (1.20.2-1, 1.22.1-1), python-linstor:amd64
> (1.16.0-1, 1.18.0-1), linstor-client:amd64 (1.16.0-1, 1.18.0-1),
> linstor-controller:amd64 (1.20.2-1, 1.22.1-1), linstor-satellite:amd64
> (1.20.2-1, 1.22.1-1), linstor-proxmox:amd64 (6.1.0-1, 7.0.0-1)

I'm pretty sure this isn't a LINSTOR issue but more a linstor-proxmox
issue: Proxmox calls the plugins status API *a* *lot*. And in case of
LINSTOR it isn't only that 1 nodes calls the controller a lot, but all
the nodes query it separately. There is a "cache" in the Proxmox API and
we even rolled our own that is stored persistently in the file system.
With 7.0.0 we are in a situation where we can not use the cache as
efficient as before, and I guess that shows. You can a) increase
"statuscache", or you can downgrade to 6.1.0. 6.1.0 should be fine, the
only thing that changed was improved storage information, but if you
have been happy so far, please downgrade. I have an idea how we can keep
the information in the cache more efficiently, but that will also
require some development effort in LINSTOR itself.

Regards, rck
_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user

Re: Slow linstor queries after recent upgrade [ In reply to ]

lukasz at wasikowski

May 10, 2023, 7:49 AM

Post #3 of 4 (205 views)

Permalink

W dniu 08.05.2023 o 09:46, Roland Kammerer pisze:
> On Sun, May 07, 2023 at 02:45:53PM +0200, ?ukasz W?sikowski wrote:
>> Hi,
>>
>> I'm using Linstor on 5 node Proxmox 7.x cluster. After recent upgrade I've
>> got some performance issues with some linstor queries.
>>
>> Upgrade: linstor-common:amd64 (1.20.2-1, 1.22.1-1), python-linstor:amd64
>> (1.16.0-1, 1.18.0-1), linstor-client:amd64 (1.16.0-1, 1.18.0-1),
>> linstor-controller:amd64 (1.20.2-1, 1.22.1-1), linstor-satellite:amd64
>> (1.20.2-1, 1.22.1-1), linstor-proxmox:amd64 (6.1.0-1, 7.0.0-1)
> I'm pretty sure this isn't a LINSTOR issue but more a linstor-proxmox
> issue: Proxmox calls the plugins status API *a* *lot*. And in case of
> LINSTOR it isn't only that 1 nodes calls the controller a lot, but all
> the nodes query it separately. There is a "cache" in the Proxmox API and
> we even rolled our own that is stored persistently in the file system.
> With 7.0.0 we are in a situation where we can not use the cache as
> efficient as before, and I guess that shows. You can a) increase
> "statuscache", or you can downgrade to 6.1.0. 6.1.0 should be fine, the
> only thing that changed was improved storage information, but if you
> have been happy so far, please downgrade. I have an idea how we can keep
> the information in the cache more efficiently, but that will also
> require some development effort in LINSTOR itself.

That helped, thank you!

--
Best regards
?ukasz W?sikowski
_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user

Re: Slow linstor queries after recent upgrade [ In reply to ]

roland.kammerer at linbit

May 10, 2023, 11:23 PM

Post #4 of 4 (204 views)

Permalink

On Wed, May 10, 2023 at 04:49:56PM +0200, ?ukasz W?sikowski wrote:
> W dniu 08.05.2023 o 09:46, Roland Kammerer pisze:
> > On Sun, May 07, 2023 at 02:45:53PM +0200, ?ukasz W?sikowski wrote:
> > > Hi,
> > >
> > > I'm using Linstor on 5 node Proxmox 7.x cluster. After recent upgrade I've
> > > got some performance issues with some linstor queries.
> > >
> > > Upgrade: linstor-common:amd64 (1.20.2-1, 1.22.1-1), python-linstor:amd64
> > > (1.16.0-1, 1.18.0-1), linstor-client:amd64 (1.16.0-1, 1.18.0-1),
> > > linstor-controller:amd64 (1.20.2-1, 1.22.1-1), linstor-satellite:amd64
> > > (1.20.2-1, 1.22.1-1), linstor-proxmox:amd64 (6.1.0-1, 7.0.0-1)
> > I'm pretty sure this isn't a LINSTOR issue but more a linstor-proxmox
> > issue: Proxmox calls the plugins status API *a* *lot*. And in case of
> > LINSTOR it isn't only that 1 nodes calls the controller a lot, but all
> > the nodes query it separately. There is a "cache" in the Proxmox API and
> > we even rolled our own that is stored persistently in the file system.
> > With 7.0.0 we are in a situation where we can not use the cache as
> > efficient as before, and I guess that shows. You can a) increase
> > "statuscache", or you can downgrade to 6.1.0. 6.1.0 should be fine, the
> > only thing that changed was improved storage information, but if you
> > have been happy so far, please downgrade. I have an idea how we can keep
> > the information in the cache more efficiently, but that will also
> > require some development effort in LINSTOR itself.
>
> That helped, thank you!

Too much technical details, but we we did the necessary development, a
new LINSTOR RC1 should follow early next week and I will adapt the
plugin during the RC period. So next LINSTOR + linstor-proxmox 7.0.1
should then behave on larger setups again. Sorry for the inconvenience.
Probably also a good call for people to test RCs, I do them for a
reason.

Regards, rck
_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user