Mailing List Archive

Re: [RFC PATCH 2/2] ARM: SMP support for mach-virt
On Mon, Dec 03, 2012 at 09:55:35PM +0000, Rob Herring wrote:
> On 12/03/2012 11:52 AM, Will Deacon wrote:
> > From: Marc Zyngier <marc.zyngier@arm.com>
> >
> > This patch adds support for SMP to mach-virt.
> >
> > Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> > Signed-off-by: Will Deacon <will.deacon@arm.com>
> >

[...]

> > +/*
> > + * This provides a "holding pen" into which all secondary cores are held
> > + * until we're ready for them to initialise.
> > + */
> > +ENTRY(virt_secondary_startup)
> > + mrc p15, 0, r0, c0, c0, 5
> > + and r0, r0, #15
> > + adr r4, 1f
> > + ldmia r4, {r5, r6}
> > + sub r4, r4, r5
> > + add r6, r6, r4
> > +pen: ldr r7, [r6]
> > + cmp r7, r0
> > + bne pen
>
> Why is the pen is needed? It should only be needed for hotplug on
> systems that can't reset their cores. I'd hope you could design good
> virtual h/w.

It's not so much about designing good virtual h/w as it is avoiding tying
the platform to it. What we don't want is to mandate that in order to boot
this machine, you *must* implement an emulation of some virtual
power-controller or SMP booting device. If we go down that route, there's
less advantage from having the virtual platform in the first place.

There's also less of a problem with the pen approach to booting because
ultimately the virtual CPUs executing there are just pthreads and will be
scheduled appropriately by the hypervisor (in contrast to a real system
where there may be concerns about power consumption and memory bandwidth).

For hotplug, sure, we could have an *optional* virtio-based device for
dealing with that if we want to. We could even have some early probing code
for it and use it for SMP boot if we find a matching DT node, but we'd still
need to keep the pen code lying around as a fallback.

> > +/*
> > + * Enumerate the possible CPU set from the device tree.
> > + */
> > +static void __init virt_smp_init_cpus(void)
> > +{
> > + const char *enable_method;
> > + struct device_node *dn = NULL;
> > + int cpu = 0;
> > + u32 release_addr;
> > +
> > + while ((dn = of_find_node_by_type(dn, "cpu"))) {
> > + if (cpu >= NR_CPUS)
> > + goto next;
> > +
> > + /*
> > + * We currently support only the "spin-table" enable-method.
> > + */
> > + enable_method = of_get_property(dn, "enable-method", NULL);
> > + if (!enable_method || strcmp(enable_method, "spin-table")) {
>
> Are these documented?

It's part of the EPAPR spec iirc and follows the booting protocol used by
arm64.

Will

_______________________________________________
Xen-arm mailing list
Xen-arm@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-arm
Re: [RFC PATCH 2/2] ARM: SMP support for mach-virt [ In reply to ]
On 12/03/2012 11:52 AM, Will Deacon wrote:
> From: Marc Zyngier <marc.zyngier@arm.com>
>
> This patch adds support for SMP to mach-virt.
>
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> Signed-off-by: Will Deacon <will.deacon@arm.com>
> ---
> arch/arm/mach-virt/Makefile | 1 +
> arch/arm/mach-virt/headsmp.S | 38 ++++++++
> arch/arm/mach-virt/platsmp.c | 205 +++++++++++++++++++++++++++++++++++++++++++
> arch/arm/mach-virt/virt.c | 6 ++
> 4 files changed, 250 insertions(+)
> create mode 100644 arch/arm/mach-virt/headsmp.S
> create mode 100644 arch/arm/mach-virt/platsmp.c
>
> diff --git a/arch/arm/mach-virt/Makefile b/arch/arm/mach-virt/Makefile
> index 7ddbfa6..9ce8a28 100644
> --- a/arch/arm/mach-virt/Makefile
> +++ b/arch/arm/mach-virt/Makefile
> @@ -3,3 +3,4 @@
> #
>
> obj-y := virt.o
> +obj-$(CONFIG_SMP) += platsmp.o headsmp.o
> diff --git a/arch/arm/mach-virt/headsmp.S b/arch/arm/mach-virt/headsmp.S
> new file mode 100644
> index 0000000..e27afb0
> --- /dev/null
> +++ b/arch/arm/mach-virt/headsmp.S
> @@ -0,0 +1,38 @@
> +/*
> + * Copyright (c) 2012 ARM Limited
> + * All Rights Reserved
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +#include <linux/linkage.h>
> +#include <linux/init.h>
> +
> + __INIT
> +
> +/*
> + * This provides a "holding pen" into which all secondary cores are held
> + * until we're ready for them to initialise.
> + */
> +ENTRY(virt_secondary_startup)
> + mrc p15, 0, r0, c0, c0, 5
> + and r0, r0, #15
> + adr r4, 1f
> + ldmia r4, {r5, r6}
> + sub r4, r4, r5
> + add r6, r6, r4
> +pen: ldr r7, [r6]
> + cmp r7, r0
> + bne pen

Why is the pen is needed? It should only be needed for hotplug on
systems that can't reset their cores. I'd hope you could design good
virtual h/w.

> +
> + /*
> + * we've been released from the holding pen: secondary_stack
> + * should now contain the SVC stack for this core
> + */
> + b secondary_startup
> +
> + .align
> +1: .long .
> + .long pen_release
> +ENDPROC(virt_secondary_startup)
> diff --git a/arch/arm/mach-virt/platsmp.c b/arch/arm/mach-virt/platsmp.c
> new file mode 100644
> index 0000000..fe02f51
> --- /dev/null
> +++ b/arch/arm/mach-virt/platsmp.c
> @@ -0,0 +1,205 @@
> +/*
> + * Dummy Virtual Machine - does what it says on the tin.
> + *
> + * SMP operations, shamelessly stolen from:
> + * arch/arm64/kernel/smp.c
> + *
> + * Copyright (C) 2012 ARM Ltd
> + * Author: Catalin Marinas <catalin.marinas@arm.com>
> + * Author: Will Deacon <will.deacon@arm.com>
> + * Author: Marc Zyngier <marc.zyngier@arm.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/smp.h>
> +#include <linux/errno.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/jiffies.h>
> +#include <linux/of.h>
> +
> +#include <asm/cacheflush.h>
> +#include <asm/smp_plat.h>
> +#include <asm/hardware/gic.h>
> +
> +extern void virt_secondary_startup(void);
> +
> +static DEFINE_RAW_SPINLOCK(boot_lock);
> +static phys_addr_t cpu_release_addr[NR_CPUS];
> +
> +/*
> + * Write secondary_holding_pen_release in a way that is guaranteed to be
> + * visible to all observers, irrespective of whether they're taking part
> + * in coherency or not. This is necessary for the hotplug code to work
> + * reliably.
> + */
> +static void __cpuinit write_pen_release(int val)
> +{
> + void *start = (void *)&pen_release;
> + unsigned long size = sizeof(pen_release);
> +
> + pen_release = val;
> + smp_wmb();
> + __cpuc_flush_dcache_area(start, size);
> + outer_clean_range(__pa(&pen_release), __pa(&pen_release + 1));
> +}
> +
> +/*
> + * Enumerate the possible CPU set from the device tree.
> + */
> +static void __init virt_smp_init_cpus(void)
> +{
> + const char *enable_method;
> + struct device_node *dn = NULL;
> + int cpu = 0;
> + u32 release_addr;
> +
> + while ((dn = of_find_node_by_type(dn, "cpu"))) {
> + if (cpu >= NR_CPUS)
> + goto next;
> +
> + /*
> + * We currently support only the "spin-table" enable-method.
> + */
> + enable_method = of_get_property(dn, "enable-method", NULL);
> + if (!enable_method || strcmp(enable_method, "spin-table")) {

Are these documented?

> + pr_err("CPU %d: missing or invalid enable-method property: %s\n",
> + cpu, enable_method);
> + goto next;
> + }
> +
> + /*
> + * Determine the address from which the CPU is polling.
> + */
> + if (of_property_read_u32(dn, "cpu-release-addr", &release_addr)) {
> + pr_err("CPU %d: missing or invalid cpu-release-addr property\n",
> + cpu);
> + goto next;
> + }
> +
> + cpu_release_addr[cpu] = release_addr;
> + set_cpu_possible(cpu, true);
> +next:
> + cpu++;
> + }
> +
> + /* sanity check */
> + if (cpu > NR_CPUS)
> + pr_warning("no. of cores (%d) greater than configured maximum of %d - clipping\n",
> + cpu, NR_CPUS);
> +
> + set_smp_cross_call(gic_raise_softirq);
> +}
> +
> +static void __init virt_smp_prepare_cpus(unsigned int max_cpus)
> +{
> + int cpu;
> + void **release_addr;
> + unsigned int ncores = num_possible_cpus();
> +
> + /*
> + * are we trying to boot more cores than exist?
> + */
> + if (max_cpus > ncores)
> + max_cpus = ncores;
> +
> + /*
> + * Initialise the present map (which describes the set of CPUs
> + * actually populated at the present time) and release the
> + * secondaries from the bootloader.
> + */
> + for_each_possible_cpu(cpu) {
> + if (max_cpus == 0)
> + break;
> +
> + if (!cpu_release_addr[cpu])
> + continue;
> +
> + release_addr = __va(cpu_release_addr[cpu]);
> + release_addr[0] = (void *)__pa(virt_secondary_startup);
> + smp_wmb();
> + __cpuc_flush_dcache_area(release_addr, sizeof(release_addr[0]));
> + outer_clean_range(__pa(release_addr), __pa(release_addr+1));
> +
> + set_cpu_present(cpu, true);
> + max_cpus--;
> + }
> +}
> +
> +static int __cpuinit virt_boot_secondary(unsigned int cpu,
> + struct task_struct *idle)
> +{
> + unsigned long timeout;
> +
> + /*
> + * Set synchronisation state between this boot processor
> + * and the secondary one
> + */
> + raw_spin_lock(&boot_lock);
> +
> + /*
> + * Update the pen release flag.
> + */
> + write_pen_release(cpu);
> +
> + /*
> + * Send the secondary CPU a soft interrupt, causing the
> + * secondaries to read pen_release.
> + */
> + gic_raise_softirq(cpumask_of(cpu), 0);
> +
> + timeout = jiffies + (1 * HZ);
> + while (time_before(jiffies, timeout)) {
> + if (pen_release == -1UL)
> + break;
> + udelay(10);
> + }
> +
> + /*
> + * Now the secondary core is starting up let it run its
> + * calibrations, then wait for it to finish
> + */
> + raw_spin_unlock(&boot_lock);
> +
> + return pen_release != -1 ? -ENOSYS : 0;
> +}
> +
> +static void __cpuinit virt_secondary_init(unsigned int cpu)
> +{
> + /*
> + * if any interrupts are already enabled for the primary
> + * core (e.g. timer irq), then they will not have been enabled
> + * for us: do so
> + */
> + gic_secondary_init(0);
> +
> + /*
> + * let the primary processor know we're out of the
> + * pen, then head off into the C entry point
> + */
> + write_pen_release(-1);
> +
> + /*
> + * Synchronise with the boot thread.
> + */
> + raw_spin_lock(&boot_lock);
> + raw_spin_unlock(&boot_lock);
> +}
> +
> +struct smp_operations __initdata virt_smp_ops = {
> + .smp_init_cpus = virt_smp_init_cpus,
> + .smp_prepare_cpus = virt_smp_prepare_cpus,
> + .smp_secondary_init = virt_secondary_init,
> + .smp_boot_secondary = virt_boot_secondary,
> +};
> diff --git a/arch/arm/mach-virt/virt.c b/arch/arm/mach-virt/virt.c
> index 174b9da..d764835 100644
> --- a/arch/arm/mach-virt/virt.c
> +++ b/arch/arm/mach-virt/virt.c
> @@ -20,6 +20,7 @@
>
> #include <linux/of_irq.h>
> #include <linux/of_platform.h>
> +#include <linux/smp.h>
>
> #include <asm/arch_timer.h>
> #include <asm/hardware/gic.h>
> @@ -56,10 +57,15 @@ static struct sys_timer virt_timer = {
> .init = virt_timer_init,
> };
>
> +#ifdef CONFIG_SMP
> +extern struct smp_operations virt_smp_ops;
> +#endif
> +
> DT_MACHINE_START(VIRT, "Dummy Virtual Machine")
> .init_irq = gic_init_irq,
> .handle_irq = gic_handle_irq,
> .timer = &virt_timer,
> .init_machine = virt_init,
> + .smp = smp_ops(virt_smp_ops),
> .dt_compat = virt_dt_match,
> MACHINE_END
>


_______________________________________________
Xen-arm mailing list
Xen-arm@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-arm
Re: [RFC PATCH 2/2] ARM: SMP support for mach-virt [ In reply to ]
On Tue, Dec 04, 2012 at 01:33:26PM +0000, Russell King - ARM Linux wrote:
> On Tue, Dec 04, 2012 at 12:40:47PM +0000, Will Deacon wrote:
> > On Mon, Dec 03, 2012 at 09:55:35PM +0000, Rob Herring wrote:
> > > Why is the pen is needed? It should only be needed for hotplug on
> > > systems that can't reset their cores. I'd hope you could design good
> > > virtual h/w.
> >
> > It's not so much about designing good virtual h/w as it is avoiding tying
> > the platform to it. What we don't want is to mandate that in order to boot
> > this machine, you *must* implement an emulation of some virtual
> > power-controller or SMP booting device. If we go down that route, there's
> > less advantage from having the virtual platform in the first place.
>
> There is actually a bigger problem here. Let's say that you have a
> quad SMP platform. You've arranged for your kernel to boot and only
> bring one of those cores online.
>
> You then kexec() or reboot. As far as the kernel is concerned, those
> other two CPUs are not online and are not running any kernel code;
> however in reality they could be sitting in this 'pen'.
>
> The memory that these 'offline' CPUs is executing then gets overwritten,
> and that's game over for those CPUs.

That's not strictly true. The device-tree passed to the kernel should have a
/memreserve/ entry for the SMP pen to avoid exactly this scenario. In real
hardware, this still sucks because you have spinning CPUs burning up power
but that's not such a problem with a virtual platform.

> So, the 'pen' approach in the kernel is fragile, I'd much rather not
> have it. It was fine in the beginning for the initial ARM Ltd SMP
> platforms but in this modern age it has no place in real platforms where
> there is proper control of the secondary CPUs.

We could have an (optional) virtual device for booting secondary CPUs but I
think the pen should still be there as a default method. Otherwise, we're
forcing a component of the platform to be emulated unnecessarily (the CPU,
vGIC and timers all have hardware-assisted virtualisation and require no
emulation).

Will

_______________________________________________
Xen-arm mailing list
Xen-arm@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-arm
Re: [RFC PATCH 2/2] ARM: SMP support for mach-virt [ In reply to ]
On Tue, Dec 04, 2012 at 02:37:25PM +0000, Russell King - ARM Linux wrote:
> On Tue, Dec 04, 2012 at 01:40:10PM +0000, Will Deacon wrote:
> > On Tue, Dec 04, 2012 at 01:33:26PM +0000, Russell King - ARM Linux wrote:
> > > The memory that these 'offline' CPUs is executing then gets overwritten,
> > > and that's game over for those CPUs.
> >
> > That's not strictly true. The device-tree passed to the kernel should have a
> > /memreserve/ entry for the SMP pen to avoid exactly this scenario. In real
> > hardware, this still sucks because you have spinning CPUs burning up power
> > but that's not such a problem with a virtual platform.
>
> Umm. So let's see. If I'm running v3.6 stock kernel and want to kexec
> into a v3.7 stock kernel. The SMP pen is part of the v3.6 kernel, which
> will be located at 32K into the RAM. The v3.7 kernel will also want to
> occupy the same place. At some point you have to overwrite the v3.6
> kernel with the v3.7 kernel image.

If the 3.6 kernel didn't bring those CPUs online, they will sit in the
bootloader pen (out of the way of the kernel image) rather than the kernel
pen so I don't think there will be a problem.

The problem you're describing actually happens when the 3.6 kernel onlines
all of the CPUs, because now it has no way to hotplug them off safely. This
is also an issue with non-virtualised hardware but we could solve it for the
virtual platform by having a para-virtualised device for doing CPU hotplug.

> That happens _before_ the DT has been parsed, so any memreserve stuff
> will be ignored. And it's at that point that your "offline" secondary
> CPUs will have their instructions overwritten.
>
> That's fine if the pen ends up being at the same place but that's not
> something we guarantee.

Having CPUs in limbo between the bootloader the being online in the kernel
is something we should just avoid. Isn't that pen __init anyway?

Will

_______________________________________________
Xen-arm mailing list
Xen-arm@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-arm
Re: [RFC PATCH 2/2] ARM: SMP support for mach-virt [ In reply to ]
On 12/04/2012 10:11 AM, Will Deacon wrote:
> On Tue, Dec 04, 2012 at 02:37:25PM +0000, Russell King - ARM Linux wrote:
>> On Tue, Dec 04, 2012 at 01:40:10PM +0000, Will Deacon wrote:
>>> On Tue, Dec 04, 2012 at 01:33:26PM +0000, Russell King - ARM Linux wrote:
>>>> The memory that these 'offline' CPUs is executing then gets overwritten,
>>>> and that's game over for those CPUs.
>>>
>>> That's not strictly true. The device-tree passed to the kernel should have a
>>> /memreserve/ entry for the SMP pen to avoid exactly this scenario. In real
>>> hardware, this still sucks because you have spinning CPUs burning up power
>>> but that's not such a problem with a virtual platform.
>>
>> Umm. So let's see. If I'm running v3.6 stock kernel and want to kexec
>> into a v3.7 stock kernel. The SMP pen is part of the v3.6 kernel, which
>> will be located at 32K into the RAM. The v3.7 kernel will also want to
>> occupy the same place. At some point you have to overwrite the v3.6
>> kernel with the v3.7 kernel image.
>
> If the 3.6 kernel didn't bring those CPUs online, they will sit in the
> bootloader pen (out of the way of the kernel image) rather than the kernel
> pen so I don't think there will be a problem.
>
> The problem you're describing actually happens when the 3.6 kernel onlines
> all of the CPUs, because now it has no way to hotplug them off safely. This
> is also an issue with non-virtualised hardware but we could solve it for the
> virtual platform by having a para-virtualised device for doing CPU hotplug.
>
>> That happens _before_ the DT has been parsed, so any memreserve stuff
>> will be ignored. And it's at that point that your "offline" secondary
>> CPUs will have their instructions overwritten.
>>
>> That's fine if the pen ends up being at the same place but that's not
>> something we guarantee.
>
> Having CPUs in limbo between the bootloader the being online in the kernel
> is something we should just avoid. Isn't that pen __init anyway?

Aren't we mixing 2 pens here? You must have some simple bootloader
containing vector table and a pen that the dtb points to, right? The pen
you have in the kernel is only needed when hotplug only does a wfi. As
you don't yet support hotplug, then you can drop all the kernel pen code.

If there is no way to reset the core, then couldn't the hotplug code
tear down the cpu setup and just jump back to 0x0 which then returns to
the bootloader's pen?

Rob

_______________________________________________
Xen-arm mailing list
Xen-arm@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-arm
Re: [RFC PATCH 2/2] ARM: SMP support for mach-virt [ In reply to ]
On Tue, Dec 04, 2012 at 04:45:58PM +0000, Rob Herring wrote:
> On 12/04/2012 10:11 AM, Will Deacon wrote:
> > On Tue, Dec 04, 2012 at 02:37:25PM +0000, Russell King - ARM Linux wrote:
> >> Umm. So let's see. If I'm running v3.6 stock kernel and want to kexec
> >> into a v3.7 stock kernel. The SMP pen is part of the v3.6 kernel, which
> >> will be located at 32K into the RAM. The v3.7 kernel will also want to
> >> occupy the same place. At some point you have to overwrite the v3.6
> >> kernel with the v3.7 kernel image.
> >
> > If the 3.6 kernel didn't bring those CPUs online, they will sit in the
> > bootloader pen (out of the way of the kernel image) rather than the kernel
> > pen so I don't think there will be a problem.
> >
> > The problem you're describing actually happens when the 3.6 kernel onlines
> > all of the CPUs, because now it has no way to hotplug them off safely. This
> > is also an issue with non-virtualised hardware but we could solve it for the
> > virtual platform by having a para-virtualised device for doing CPU hotplug.
> >
> >> That happens _before_ the DT has been parsed, so any memreserve stuff
> >> will be ignored. And it's at that point that your "offline" secondary
> >> CPUs will have their instructions overwritten.
> >>
> >> That's fine if the pen ends up being at the same place but that's not
> >> something we guarantee.
> >
> > Having CPUs in limbo between the bootloader the being online in the kernel
> > is something we should just avoid. Isn't that pen __init anyway?
>
> Aren't we mixing 2 pens here? You must have some simple bootloader
> containing vector table and a pen that the dtb points to, right? The pen
> you have in the kernel is only needed when hotplug only does a wfi. As
> you don't yet support hotplug, then you can drop all the kernel pen code.

Yes, both qemu and kvmtool have bootloader pens outside of the kernel but
since wfi is not trapped by kvm, the secondaries can be released early due
to a spurious wakeup so we need the second pen.

> If there is no way to reset the core, then couldn't the hotplug code
> tear down the cpu setup and just jump back to 0x0 which then returns to
> the bootloader's pen?

I think hotplug really should be implemented with a virtio device. We just
trap back to the emulation and kill the vcpu thread.

Will

_______________________________________________
Xen-arm mailing list
Xen-arm@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-arm
Re: [RFC PATCH 2/2] ARM: SMP support for mach-virt [ In reply to ]
On 12/04/2012 11:16 AM, Will Deacon wrote:
> On Tue, Dec 04, 2012 at 04:45:58PM +0000, Rob Herring wrote:
>> On 12/04/2012 10:11 AM, Will Deacon wrote:
>>> On Tue, Dec 04, 2012 at 02:37:25PM +0000, Russell King - ARM Linux wrote:
>>>> Umm. So let's see. If I'm running v3.6 stock kernel and want to kexec
>>>> into a v3.7 stock kernel. The SMP pen is part of the v3.6 kernel, which
>>>> will be located at 32K into the RAM. The v3.7 kernel will also want to
>>>> occupy the same place. At some point you have to overwrite the v3.6
>>>> kernel with the v3.7 kernel image.
>>>
>>> If the 3.6 kernel didn't bring those CPUs online, they will sit in the
>>> bootloader pen (out of the way of the kernel image) rather than the kernel
>>> pen so I don't think there will be a problem.
>>>
>>> The problem you're describing actually happens when the 3.6 kernel onlines
>>> all of the CPUs, because now it has no way to hotplug them off safely. This
>>> is also an issue with non-virtualised hardware but we could solve it for the
>>> virtual platform by having a para-virtualised device for doing CPU hotplug.
>>>
>>>> That happens _before_ the DT has been parsed, so any memreserve stuff
>>>> will be ignored. And it's at that point that your "offline" secondary
>>>> CPUs will have their instructions overwritten.
>>>>
>>>> That's fine if the pen ends up being at the same place but that's not
>>>> something we guarantee.
>>>
>>> Having CPUs in limbo between the bootloader the being online in the kernel
>>> is something we should just avoid. Isn't that pen __init anyway?
>>
>> Aren't we mixing 2 pens here? You must have some simple bootloader
>> containing vector table and a pen that the dtb points to, right? The pen
>> you have in the kernel is only needed when hotplug only does a wfi. As
>> you don't yet support hotplug, then you can drop all the kernel pen code.
>
> Yes, both qemu and kvmtool have bootloader pens outside of the kernel but
> since wfi is not trapped by kvm, the secondaries can be released early due
> to a spurious wakeup so we need the second pen.

Wouldn't the pen requiring both a valid address (perhaps !0 or !-1) and
a wake-up fix this?

Rob


_______________________________________________
Xen-arm mailing list
Xen-arm@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-arm
Re: [RFC PATCH 2/2] ARM: SMP support for mach-virt [ In reply to ]
On Tue, Dec 04, 2012 at 04:35:55PM +0000, Russell King - ARM Linux wrote:
> On Tue, Dec 04, 2012 at 04:11:13PM +0000, Will Deacon wrote:
> > The problem you're describing actually happens when the 3.6 kernel onlines
> > all of the CPUs, because now it has no way to hotplug them off safely. This
> > is also an issue with non-virtualised hardware but we could solve it for the
> > virtual platform by having a para-virtualised device for doing CPU hotplug.
>
> That situation exists on ARM Ltd platforms where there's no way to
> properly return them back to the boot loader. We should not be forcing
> this ARM Ltd platform deficiency onto other platforms as part of a
> "design", even virtual platforms.
>
> Most other real-world platforms out there have a way to power off the
> unused secondary CPUs - Tegra and OMAP both do.

If a virtual machine powers off a virtual CPU, I doubt we want to power of
its corresponding CPU -- that logic can remain in the host. All we need to
do is kill the virtual CPU thread, which we can do easily enough. Booting is
the more difficult problem because we introduce a reliance on a virtual
device being ready incredibly early, essentially hardcoding part of the
virtual machine.

> As far as virtual platforms go, how secondary CPUs are dealt with should
> already have been solved; I really can't imagine that KVM and XEN on
> other architectures end up with CPUs spinning in a loop inside the guest
> kernel waiting for the guest OS to ask them to boot. Neither can I imagine
> that KVM and XEN end up with CPUs spinning in the guest OS when CPUs are
> asked to be hot-unplugged.

So neither kvmtool or qemu currently support hotplug for kvm guests on any
architectures from what I can tell. Furthermore, kvmtool on ppc (at least)
uses a secondary spinning loop at a fixed offset into the kernel image. I
don't think we should really pay much attention to those other architectures
in this regard!

> > > That happens _before_ the DT has been parsed, so any memreserve stuff
> > > will be ignored. And it's at that point that your "offline" secondary
> > > CPUs will have their instructions overwritten.
> > >
> > > That's fine if the pen ends up being at the same place but that's not
> > > something we guarantee.
> >
> > Having CPUs in limbo between the bootloader the being online in the kernel
> > is something we should just avoid. Isn't that pen __init anyway?
>
> If you have hotplug enabled, all the secondary bringup code should be
> in the __cpuinit and __cpuinitdata sections.

Right, but if booting a !HOTPLUG kernel via kexec, surely we'd have to clear
that pen of CPUs?

Will

_______________________________________________
Xen-arm mailing list
Xen-arm@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-arm
Re: [RFC PATCH 2/2] ARM: SMP support for mach-virt [ In reply to ]
On 04/12/12 17:16, Will Deacon wrote:
> On Tue, Dec 04, 2012 at 04:45:58PM +0000, Rob Herring wrote:
>> On 12/04/2012 10:11 AM, Will Deacon wrote:
>>> On Tue, Dec 04, 2012 at 02:37:25PM +0000, Russell King - ARM Linux wrote:
>>>> Umm. So let's see. If I'm running v3.6 stock kernel and want to kexec
>>>> into a v3.7 stock kernel. The SMP pen is part of the v3.6 kernel, which
>>>> will be located at 32K into the RAM. The v3.7 kernel will also want to
>>>> occupy the same place. At some point you have to overwrite the v3.6
>>>> kernel with the v3.7 kernel image.
>>>
>>> If the 3.6 kernel didn't bring those CPUs online, they will sit in the
>>> bootloader pen (out of the way of the kernel image) rather than the kernel
>>> pen so I don't think there will be a problem.
>>>
>>> The problem you're describing actually happens when the 3.6 kernel onlines
>>> all of the CPUs, because now it has no way to hotplug them off safely. This
>>> is also an issue with non-virtualised hardware but we could solve it for the
>>> virtual platform by having a para-virtualised device for doing CPU hotplug.
>>>
>>>> That happens _before_ the DT has been parsed, so any memreserve stuff
>>>> will be ignored. And it's at that point that your "offline" secondary
>>>> CPUs will have their instructions overwritten.
>>>>
>>>> That's fine if the pen ends up being at the same place but that's not
>>>> something we guarantee.
>>>
>>> Having CPUs in limbo between the bootloader the being online in the kernel
>>> is something we should just avoid. Isn't that pen __init anyway?
>>
>> Aren't we mixing 2 pens here? You must have some simple bootloader
>> containing vector table and a pen that the dtb points to, right? The pen
>> you have in the kernel is only needed when hotplug only does a wfi. As
>> you don't yet support hotplug, then you can drop all the kernel pen code.
>
> Yes, both qemu and kvmtool have bootloader pens outside of the kernel but
> since wfi is not trapped by kvm, the secondaries can be released early due
> to a spurious wakeup so we need the second pen.

Actually, KVM traps WFI and puts the vcpu thread on a wait queue (we are
actually giving more guaranties than the architecture offers here).

We should be able to remove the loop and rely on WFI.

M.
--
Jazz is not dead. It just smells funny...


_______________________________________________
Xen-arm mailing list
Xen-arm@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-arm
Re: [RFC PATCH 2/2] ARM: SMP support for mach-virt [ In reply to ]
On Tue, Dec 04, 2012 at 05:24:57PM +0000, Marc Zyngier wrote:
> On 04/12/12 17:16, Will Deacon wrote:
> > On Tue, Dec 04, 2012 at 04:45:58PM +0000, Rob Herring wrote:
> >> Aren't we mixing 2 pens here? You must have some simple bootloader
> >> containing vector table and a pen that the dtb points to, right? The pen
> >> you have in the kernel is only needed when hotplug only does a wfi. As
> >> you don't yet support hotplug, then you can drop all the kernel pen code.
> >
> > Yes, both qemu and kvmtool have bootloader pens outside of the kernel but
> > since wfi is not trapped by kvm, the secondaries can be released early due
> > to a spurious wakeup so we need the second pen.
>
> Actually, KVM traps WFI and puts the vcpu thread on a wait queue (we are
> actually giving more guaranties than the architecture offers here).

Ok, if we can rely on this behaviour in the future this sounds promising.
Can somebody comment from the Xen side of things please?

> We should be able to remove the loop and rely on WFI.

Yes, we should be able to remove the kernel-side loop. We still won't have
hotplug and friends, but that can come later because there's a discussion
around PSCI vs virtual power controller to be had there.

Will

_______________________________________________
Xen-arm mailing list
Xen-arm@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-arm
Re: [RFC PATCH 2/2] ARM: SMP support for mach-virt [ In reply to ]
On Tue, 4 Dec 2012, Will Deacon wrote:

> On Tue, Dec 04, 2012 at 04:45:58PM +0000, Rob Herring wrote:
> > If there is no way to reset the core, then couldn't the hotplug code
> > tear down the cpu setup and just jump back to 0x0 which then returns to
> > the bootloader's pen?
>
> I think hotplug really should be implemented with a virtio device. We just
> trap back to the emulation and kill the vcpu thread.

Hotplug and secondary boot should really be considered the same and use
the same methods. So if you envision a virtio device for hotplug, then
just use that for secondary boot as well. PSCI might be simpler to
implement though.


Nicolas

_______________________________________________
Xen-arm mailing list
Xen-arm@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-arm
Re: [RFC PATCH 2/2] ARM: SMP support for mach-virt [ In reply to ]
On Tuesday 04 December 2012, Will Deacon wrote:
> >
> > Most other real-world platforms out there have a way to power off the
> > unused secondary CPUs - Tegra and OMAP both do.
>
> If a virtual machine powers off a virtual CPU, I doubt we want to power of
> its corresponding CPU -- that logic can remain in the host. All we need to
> do is kill the virtual CPU thread, which we can do easily enough. Booting is
> the more difficult problem because we introduce a reliance on a virtual
> device being ready incredibly early, essentially hardcoding part of the
> virtual machine.

Powering off a virtual CPU is the same as killing the virtual CPU thread.
Not powering it off would imply that we schedule a host CPU to run an
endless loop on it.

Arnd

_______________________________________________
Xen-arm mailing list
Xen-arm@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-arm