Mailing List Archive

CPU Reservations
Hi all,

It's been a bit since I've asked this question, if I have here before.

Do we all run our UC appliances in VMWare with the full CPU MHz and core reservations prescribed by Cisco, in production? Or, if you have information on hand regarding the actual resource usage, have any of you taken on resizing the VM reservations?

The various documents are very much so clear that oversubscription isn't supported, but, it also talks about vCPU to cores which I'm told doesn't really play out in VMWare as it's a MHz reservation that can be scheduled in to available hardware.

There are various statements peppered in about running your own VM environment with best practices - but also the 1:1 pcore:vcore comments.

Is anyone turning these knobs? Has anyone stepped over that pcore:vcore line when it appears there are enough resources?

I'm looking for thoughts or unforeseen consequences that we can use to back somewhat of the case as to why we need to continue to fund hardware at scale which is largely idle.

Adam
Re: CPU Reservations [ In reply to ]
Adam,

Yes, the reservation appears steep and outside monitors appear ripe for harvesting underutilized resources.


And 3 things come to mind:

* ccm.bin at its core is a distributed real time state machine. Delays have serious impacts. Feature are invoked across nodes. See the long history around CCM clustering over the WAN requirements for more background.
* mtp/moh/cfb are all essentially real time for audio.
* some processes are less predictable in their consumption of resources. “Resource Starvation” has many symptoms across features that are difficult to diagnose. This consumes more of your time as implementers, admins, and monitors. And it significantly degrades customer experience of the product and people servicing the product. There is an element of resource allocation that is akin to road infrastructure planning - it may appear idle many times and it can still be under provisioned during peak demand.

Consider for example
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCtu18692

This under provision/over subscription negatively impacted several deployments.

In my experience VOIP has been a bit of a “canary in the coal mine” for IP applications. I see many situations where users are willing to re-send an email. Meanwhile the first call that does not go through, gets dropped, or suffers poor audio is grounds for immediate escalation.

Maybe some things to consider.

-Wes

On Jul 8, 2019, at 11:57 AM, Pawlowski, Adam <ajp26@buffalo.edu<mailto:ajp26@buffalo.edu>> wrote:

Hi all,

It’s been a bit since I’ve asked this question, if I have here before.

Do we all run our UC appliances in VMWare with the full CPU MHz and core reservations prescribed by Cisco, in production? Or, if you have information on hand regarding the actual resource usage, have any of you taken on resizing the VM reservations?

The various documents are very much so clear that oversubscription isn’t supported, but, it also talks about vCPU to cores which I’m told doesn’t really play out in VMWare as it’s a MHz reservation that can be scheduled in to available hardware.

There are various statements peppered in about running your own VM environment with best practices – but also the 1:1 pcore:vcore comments.

Is anyone turning these knobs? Has anyone stepped over that pcore:vcore line when it appears there are enough resources?

I’m looking for thoughts or unforeseen consequences that we can use to back somewhat of the case as to why we need to continue to fund hardware at scale which is largely idle.

Adam


_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip
Re: CPU Reservations [ In reply to ]
Adam,

You'd probably be less likely to have an "issue" if the host's aggregate compute resources are at least 30%-35% below subscribed capacity (but no guarantees). Real time traffic on the software such as mtp, moh,.. etc (I think UCOS even has some files that internally communicate via real time or near real time) can be a diva at times. There are two things that come to mind that I have always seen give issue to real time traffic without fail;


* latency and jitter, albeit spatial or mechanical such as distance or clock cycles
* unreliable or latent synchronization (I'd humbly suggest this is anything over 5 hops from a cesium clock, but Cisco says 3 so we'll go with that)

I've found that if you violate either of these, you can be in for a wild ride. The symptoms are often obscure, inconsistent and seemingly unrelated to any other known issues; a small delay in voice that randomly pops up here or there for one or two phones, a presence HA pair that randomly fails over for no apparent reason, some phones not showing call history, SIP trunks going out of service.. etc. These issues can be very difficult to troubleshoot because there won't "appear" to be anything wrong.

Its "hit or miss" at best with the CPU reservations (on a host that is not already over subscribed) I'd say; if the UC VM has to wait (delay) on cycle time, even for a fraction of a second, it may or may not cause you an issue... just depends on what the server was trying to do at the time, and if it involved real time traffic. If you've got UC VMs in the mix with HA / clustering, then the VMs will be even less tolerant of asking for cycle time rather than it just being available.

The safest path is to guarantee (reserve) the require resources to the UC VM, even though it may not ever (or nearly never) use the full capacity (because having cycle time readily available and having to ask the scheduler for cycle time that is available is not the same thing).

Think of it like insurance, you're not paying for it because you don't need it (actual waste), you're paying because of that one time you do need it and don't know it.

Thanks,

Ryan
________________________________
From: cisco-voip <cisco-voip-bounces@puck.nether.net> on behalf of Pawlowski, Adam <ajp26@buffalo.edu>
Sent: Monday, July 8, 2019 11:57 AM
To: cisco-voip@puck.nether.net
Subject: [cisco-voip] CPU Reservations


Hi all,



It?s been a bit since I?ve asked this question, if I have here before.



Do we all run our UC appliances in VMWare with the full CPU MHz and core reservations prescribed by Cisco, in production? Or, if you have information on hand regarding the actual resource usage, have any of you taken on resizing the VM reservations?



The various documents are very much so clear that oversubscription isn?t supported, but, it also talks about vCPU to cores which I?m told doesn?t really play out in VMWare as it?s a MHz reservation that can be scheduled in to available hardware.



There are various statements peppered in about running your own VM environment with best practices ? but also the 1:1 pcore:vcore comments.



Is anyone turning these knobs? Has anyone stepped over that pcore:vcore line when it appears there are enough resources?



I?m looking for thoughts or unforeseen consequences that we can use to back somewhat of the case as to why we need to continue to fund hardware at scale which is largely idle.



Adam
Re: CPU Reservations [ In reply to ]
“ccm.bin at its core is a distributed real time state machine.”

^^^ this ^^^

This is what I try to tell people when they tell me that they can run 50 machines on the same boxes that I run 10.

And they _still_ don’t get it.


-sent from mobile device-

Lelio Fulgenzi, B.A. | Senior Analyst
Computing and Communications Services | University of Guelph
Room 037 Animal Science & Nutrition Bldg | 50 Stone Rd E | Guelph, ON | N1G 2W1<x-apple-data-detectors://1/0>
519-824-4120 Ext. 56354<tel:519-824-4120;56354> | lelio@uoguelph.ca<mailto:lelio@uoguelph.ca>

www.uoguelph.ca/ccs<http://www.uoguelph.ca/ccs> | @UofGCCS on Instagram, Twitter and Facebook

[University of Guelph Cornerstone with Improve Life tagline]

On Jul 8, 2019, at 7:38 PM, Wes Sisk (wsisk) via cisco-voip <cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>> wrote:

ccm.bin at its core is a distributed real time state machine.
Re: CPU Reservations [ In reply to ]
The VM team here when I told them we had to have resource reservation and no oversubscription always complain. “Your servers don’t consume anywhere near that capacity I can show you the performance stats” “Every vendor asks for resource reservation that comes in”

That is what I always hear from them.

Brian Palmer | VoiceOps | DC6 3 355
904-905-8263 | Internal Ext: 58263

From: cisco-voip <cisco-voip-bounces@puck.nether.net> On Behalf Of Lelio Fulgenzi
Sent: Monday, July 8, 2019 10:42 PM
To: Wes Sisk (wsisk) <wsisk@cisco.com>
Cc: cisco-voip@puck.nether.net; Pawlowski, Adam <ajp26@buffalo.edu>
Subject: Re: [cisco-voip] CPU Reservations



“ccm.bin at its core is a distributed real time state machine.”

^^^ this ^^^

This is what I try to tell people when they tell me that they can run 50 machines on the same boxes that I run 10.

And they _still_ don’t get it.


-sent from mobile device-


Lelio Fulgenzi, B.A. | Senior Analyst
Computing and Communications Services | University of Guelph
Room 037 Animal Science & Nutrition Bldg | 50 Stone Rd E | Guelph, ON | N1G 2W1<x-apple-data-detectors://1/0>
519-824-4120 Ext. 56354<tel:519-824-4120;56354> | lelio@uoguelph.ca<mailto:lelio@uoguelph.ca>

http://defang.bcbsfl.com/defang.php?url=www.uoguelph.ca/ccs<http://defang.bcbsfl.com/defang.php?url=http://www.uoguelph.ca/ccs> | @UofGCCS on Instagram, Twitter and Facebook

[University of Guelph Cornerstone with Improve Life tagline]

On Jul 8, 2019, at 7:38 PM, Wes Sisk (wsisk) via cisco-voip <cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>> wrote:
ccm.bin at its core is a distributed real time state machine.
We comply with applicable Federal civil rights laws and do not discriminate.

You may access the Non-Discrimination and Accessibility Notice here <http://floridablue.com/ndnotice>.

Language Assistance Available:

Español, Kreyol Ayisien, Ti?ng Vi?t, Português, ??, français, Tagalog, ???????, italiano, Deutsche, ???, Polskie, Gujarati, ???, ???????, ???, ????? <http://floridablue.com/languageservices>

Florida Blue is a trade name of Blue Cross and Blue Shield of Florida, Inc. Blue Cross and Blue Shield of Florida, Inc., and its subsidiary and affiliate companies are not responsible for errors or omissions in this e-mail message. Any personal comments made in this e-mail do not reflect the views of Blue Cross and Blue Shield of Florida, Inc. The information contained in this document may be confidential and intended solely for the use of the individual or entity to whom it is addressed. This document may contain material that is privileged or protected from disclosure under applicable law. If you are not the intended recipient or the individual responsible for delivering to the intended recipient, please (1) be advised that any use, dissemination, forwarding, or copying of this document IS STRICTLY PROHIBITED; and (2) notify sender immediately by telephone and destroy the document. THANK YOU.
Re: CPU Reservations [ In reply to ]
My answer back to them is this most of the time:

Me: “Please pick up your phone.”
Them: (they pick up their phone)
Me: “What did you hear?”
Them: “Dial Tone”
Me: _That’s_ why we need resource reservation.

And then I go on to ask them how they’d feel if that dial tone or other features were delayed by a ½ second? 1 second? 2 seconds?

They’re still not happy, but they begin to get the picture.

Of course, my example may not be the best example, but it gets the message across.




---
Lelio Fulgenzi, B.A. | Senior Analyst
Computing and Communications Services | University of Guelph
Room 037 Animal Science & Nutrition Bldg | 50 Stone Rd E | Guelph, ON | N1G 2W1
519-824-4120 Ext. 56354 | lelio@uoguelph.ca<mailto:lelio@uoguelph.ca>

www.uoguelph.ca/ccs<http://www.uoguelph.ca/ccs> | @UofGCCS on Instagram, Twitter and Facebook

[University of Guelph Cornerstone with Improve Life tagline]

From: Palmer, Brian <Brian.Palmer@bcbsfl.com>
Sent: Wednesday, July 10, 2019 9:35 AM
To: Lelio Fulgenzi <lelio@uoguelph.ca>; Wes Sisk (wsisk) <wsisk@cisco.com>
Cc: cisco-voip@puck.nether.net; Pawlowski, Adam <ajp26@buffalo.edu>
Subject: RE: [cisco-voip] CPU Reservations

The VM team here when I told them we had to have resource reservation and no oversubscription always complain. “Your servers don’t consume anywhere near that capacity I can show you the performance stats” “Every vendor asks for resource reservation that comes in”

That is what I always hear from them.

Brian Palmer | VoiceOps | DC6 3 355
904-905-8263 | Internal Ext: 58263

From: cisco-voip <cisco-voip-bounces@puck.nether.net<mailto:cisco-voip-bounces@puck.nether.net>> On Behalf Of Lelio Fulgenzi
Sent: Monday, July 8, 2019 10:42 PM
To: Wes Sisk (wsisk) <wsisk@cisco.com<mailto:wsisk@cisco.com>>
Cc: cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>; Pawlowski, Adam <ajp26@buffalo.edu<mailto:ajp26@buffalo.edu>>
Subject: Re: [cisco-voip] CPU Reservations



“ccm.bin at its core is a distributed real time state machine.”

^^^ this ^^^

This is what I try to tell people when they tell me that they can run 50 machines on the same boxes that I run 10.

And they _still_ don’t get it.


-sent from mobile device-

Lelio Fulgenzi, B.A. | Senior Analyst
Computing and Communications Services | University of Guelph
Room 037 Animal Science & Nutrition Bldg | 50 Stone Rd E | Guelph, ON | N1G 2W1<x-apple-data-detectors://1/0>
519-824-4120 Ext. 56354<tel:519-824-4120;56354> | lelio@uoguelph.ca<mailto:lelio@uoguelph.ca>

http://defang.bcbsfl.com/defang.php?url=www.uoguelph.ca/ccs<http://defang.bcbsfl.com/defang.php?url=http://www.uoguelph.ca/ccs> | @UofGCCS on Instagram, Twitter and Facebook

[University of Guelph Cornerstone with Improve Life tagline]

On Jul 8, 2019, at 7:38 PM, Wes Sisk (wsisk) via cisco-voip <cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>> wrote:
ccm.bin at its core is a distributed real time state machine.

We comply with applicable Federal civil rights laws and do not discriminate.

You may access the Non-Discrimination and Accessibility Notice here<http://floridablue.com/ndnotice>.

Language Assistance Available:

Español, Kreyol Ayisien, Ti?ng Vi?t, Português, ??, français, Tagalog, ???????, italiano, Deutsche, ???, Polskie, Gujarati, ???, ???????, ???, ?????<http://floridablue.com/languageservices>

Florida Blue is a trade name of Blue Cross and Blue Shield of Florida, Inc. Blue Cross and Blue Shield of Florida, Inc., and its subsidiary and affiliate companies are not responsible for errors or omissions in this e-mail message. Any personal comments made in this e-mail do not reflect the views of Blue Cross and Blue Shield of Florida, Inc. The information contained in this document may be confidential and intended solely for the use of the individual or entity to whom it is addressed. This document may contain material that is privileged or protected from disclosure under applicable law. If you are not the intended recipient or the individual responsible for delivering to the intended recipient, please (1) be advised that any use, dissemination, forwarding, or copying of this document IS STRICTLY PROHIBITED; and (2) notify sender immediately by telephone and destroy the document. THANK YOU.
Re: CPU Reservations [ In reply to ]
People view dial tone as if it were in the US Bill of Rights. Let it not be there when expected and see what happens. It’s a fascinating social experience.

Sent from my iPhone

On Jul 10, 2019, at 10:47, Lelio Fulgenzi <lelio@uoguelph.ca<mailto:lelio@uoguelph.ca>> wrote:

My answer back to them is this most of the time:

Me: “Please pick up your phone.”
Them: (they pick up their phone)
Me: “What did you hear?”
Them: “Dial Tone”
Me: _That’s_ why we need resource reservation.

And then I go on to ask them how they’d feel if that dial tone or other features were delayed by a ½ second? 1 second? 2 seconds?

They’re still not happy, but they begin to get the picture.

Of course, my example may not be the best example, but it gets the message across.




---
Lelio Fulgenzi, B.A. | Senior Analyst
Computing and Communications Services | University of Guelph
Room 037 Animal Science & Nutrition Bldg | 50 Stone Rd E | Guelph, ON | N1G 2W1
519-824-4120 Ext. 56354 | lelio@uoguelph.ca<mailto:lelio@uoguelph.ca>

www.uoguelph.ca/ccs<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.uoguelph.ca%2Fccs&data=02%7C01%7C%7C60ae869851e04ced75d808d705458c42%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636983668576148647&sdata=4uXO%2FCOorYmUQ5iy9w0mlXId5NSCsWZ8qgPUGmCCVGs%3D&reserved=0> | @UofGCCS on Instagram, Twitter and Facebook

<image001.png>

From: Palmer, Brian <Brian.Palmer@bcbsfl.com<mailto:Brian.Palmer@bcbsfl.com>>
Sent: Wednesday, July 10, 2019 9:35 AM
To: Lelio Fulgenzi <lelio@uoguelph.ca<mailto:lelio@uoguelph.ca>>; Wes Sisk (wsisk) <wsisk@cisco.com<mailto:wsisk@cisco.com>>
Cc: cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>; Pawlowski, Adam <ajp26@buffalo.edu<mailto:ajp26@buffalo.edu>>
Subject: RE: [cisco-voip] CPU Reservations

The VM team here when I told them we had to have resource reservation and no oversubscription always complain. “Your servers don’t consume anywhere near that capacity I can show you the performance stats” “Every vendor asks for resource reservation that comes in”

That is what I always hear from them.

Brian Palmer | VoiceOps | DC6 3 355
904-905-8263 | Internal Ext: 58263

From: cisco-voip <cisco-voip-bounces@puck.nether.net<mailto:cisco-voip-bounces@puck.nether.net>> On Behalf Of Lelio Fulgenzi
Sent: Monday, July 8, 2019 10:42 PM
To: Wes Sisk (wsisk) <wsisk@cisco.com<mailto:wsisk@cisco.com>>
Cc: cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>; Pawlowski, Adam <ajp26@buffalo.edu<mailto:ajp26@buffalo.edu>>
Subject: Re: [cisco-voip] CPU Reservations



“ccm.bin at its core is a distributed real time state machine.”

^^^ this ^^^

This is what I try to tell people when they tell me that they can run 50 machines on the same boxes that I run 10.

And they _still_ don’t get it.


-sent from mobile device-

Lelio Fulgenzi, B.A. | Senior Analyst
Computing and Communications Services | University of Guelph
Room 037 Animal Science & Nutrition Bldg | 50 Stone Rd E | Guelph, ON | N1G 2W1<x-apple-data-detectors://1/0>
519-824-4120 Ext. 56354<tel:519-824-4120;56354> | lelio@uoguelph.ca<mailto:lelio@uoguelph.ca>

http://defang.bcbsfl.com/defang.php?url=www.uoguelph.ca/ccs<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdefang.bcbsfl.com%2Fdefang.php%3Furl%3Dhttp%3A%2F%2Fwww.uoguelph.ca%2Fccs&data=02%7C01%7C%7C60ae869851e04ced75d808d705458c42%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636983668576158658&sdata=9F2E43%2BcDbwmOzYDjKIBJ3wNX4kpERmB67C4993YYtQ%3D&reserved=0> | @UofGCCS on Instagram, Twitter and Facebook

[University of Guelph Cornerstone with Improve Life tagline]

On Jul 8, 2019, at 7:38 PM, Wes Sisk (wsisk) via cisco-voip <cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>> wrote:
ccm.bin at its core is a distributed real time state machine.

We comply with applicable Federal civil rights laws and do not discriminate.

You may access the Non-Discrimination and Accessibility Notice here<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffloridablue.com%2Fndnotice&data=02%7C01%7C%7C60ae869851e04ced75d808d705458c42%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636983668576168669&sdata=9PS5XJf5SudErPbI8zhGS3KqAemBc2Jk1tkCTQOIPAs%3D&reserved=0>.

Language Assistance Available:

Español, Kreyol Ayisien, Ti?ng Vi?t, Português, ??, français, Tagalog, ???????, italiano, Deutsche, ???, Polskie, Gujarati, ???, ???????, ???, ?????<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Ffloridablue.com%2Flanguageservices&data=02%7C01%7C%7C60ae869851e04ced75d808d705458c42%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636983668576178675&sdata=H0Z3wd3Tvzr66xp9J%2BcUequsQddKhvdG7A86D9owCqk%3D&reserved=0>

Florida Blue is a trade name of Blue Cross and Blue Shield of Florida, Inc. Blue Cross and Blue Shield of Florida, Inc., and its subsidiary and affiliate companies are not responsible for errors or omissions in this e-mail message. Any personal comments made in this e-mail do not reflect the views of Blue Cross and Blue Shield of Florida, Inc. The information contained in this document may be confidential and intended solely for the use of the individual or entity to whom it is addressed. This document may contain material that is privileged or protected from disclosure under applicable law. If you are not the intended recipient or the individual responsible for delivering to the intended recipient, please (1) be advised that any use, dissemination, forwarding, or copying of this document IS STRICTLY PROHIBITED; and (2) notify sender immediately by telephone and destroy the document. THANK YOU.

_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Fcisco-voip&amp;data=02%7C01%7C%7C60ae869851e04ced75d808d705458c42%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636983668576218707&amp;sdata=jIiNZgZqlRTyrp30AdEhVNoZr%2BFHJLujxfdAk0X95Kk%3D&amp;reserved=0