Mailing List Archive

make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs
Hi,

I'm facing a small issue when trying to generate a debball on Ubuntu 20.04
with the latest Xen master (27357c394ba6e1571a89105b840ce1c6f026485c):

ld: xen.efi: warning: section .init: alignment 2**15 not representable
nm -pa --format=sysv ./xen.efi \
| ./tools/symbols --all-symbols --xensyms --sysv --sort >./xen.efi.map
rm -f ./.xen.efi.[0-9]* ./..xen.efi.[0-9]*
/bin/sh ./tools/check-endbr.sh xen.efi
check-endbr.sh xen.efi Error: More valid-addrs than all-addrs
make[3]: *** [arch/x86/Makefile:219: xen.efi] Error 1
make[2]: *** [build.mk:90: xen] Error 2
make[1]: *** [Makefile:567: xen] Error 2


So the script fails at this location:
https://github.com/xen-project/xen/blob/master/xen/tools/check-endbr.sh#L91

Any idea what happened ?

For reference, this is a Dockerfile to repro the issue:
https://gist.github.com/Wenzel/c49e7cf2c5bdbc4a767002b59cd4d772

Thank you in advance

Mathieu Tarral
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
> On 13 Jul 2022, at 15:54, Mathieu Tarral <mathieu.tarral@protonmail.com> wrote:
>
> Hi,
>
> I'm facing a small issue when trying to generate a debball on Ubuntu 20.04
> with the latest Xen master (27357c394ba6e1571a89105b840ce1c6f026485c):
>
> ld: xen.efi: warning: section .init: alignment 2**15 not representable
> nm -pa --format=sysv ./xen.efi \
> | ./tools/symbols --all-symbols --xensyms --sysv --sort >./xen.efi.map
> rm -f ./.xen.efi.[0-9]* ./..xen.efi.[0-9]*
> /bin/sh ./tools/check-endbr.sh xen.efi
> check-endbr.sh xen.efi Error: More valid-addrs than all-addrs
> make[3]: *** [arch/x86/Makefile:219: xen.efi] Error 1
> make[2]: *** [build.mk:90: xen] Error 2
> make[1]: *** [Makefile:567: xen] Error 2
>
>
> So the script fails at this location:
> https://github.com/xen-project/xen/blob/master/xen/tools/check-endbr.sh#L91
>
> Any idea what happened ?
>
> For reference, this is a Dockerfile to repro the issue:
> https://gist.github.com/Wenzel/c49e7cf2c5bdbc4a767002b59cd4d772
>
> Thank you in advance

Hi Mathieu,

Can you print your awk —version? Does the compilation succeeds if you install gawk in the docker file?

Cheers,
Luca

>
> Mathieu Tarral
>
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
Hi Lucas,

>
> Can you print your awk —version? Does the compilation succeeds if you install gawk in the docker file?

My awk version uses a different flag to print its version info:
root@23afdeb64f40:/# awk -W version
mawk 1.3.4 20200120
Copyright 2008-2019,2020, Thomas E. Dickey
Copyright 1991-1996,2014, Michael D. Brennan

random-funcs: srandom/random
regex-funcs: internal
compiled limits:
sprintf buffer 8192
maximum-integer 2147483647

I attempted the build again after inserting gawk into
the apt install list, and now it fails on newlib:

touch mk-headers-x86_64
/usr/bin/false newlib-1.16.0.tar.gz http://xenbits.xen.org/xen-extfiles/newlib-1.16.0.tar.gz
make[1]: *** [Makefile:90: newlib-1.16.0.tar.gz] Error 1
make[1]: Leaving directory '/xen/stubdom'
make: *** [Makefile:151: install-stubdom] Error 2
The command '/bin/sh -c make debball' returned a non-zero code: 2

I hope this helps.

Mathieu Tarral
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
> On 13 Jul 2022, at 17:21, Mathieu Tarral <mathieu.tarral@protonmail.com> wrote:
>
> Hi Lucas,
>
>>
>> Can you print your awk —version? Does the compilation succeeds if you install gawk in the docker file?
>
> My awk version uses a different flag to print its version info:
> root@23afdeb64f40:/# awk -W version
> mawk 1.3.4 20200120
> Copyright 2008-2019,2020, Thomas E. Dickey
> Copyright 1991-1996,2014, Michael D. Brennan
>
> random-funcs: srandom/random
> regex-funcs: internal
> compiled limits:
> sprintf buffer 8192
> maximum-integer 2147483647
>
> I attempted the build again after inserting gawk into
> the apt install list, and now it fails on newlib:
>
> touch mk-headers-x86_64
> /usr/bin/false newlib-1.16.0.tar.gz http://xenbits.xen.org/xen-extfiles/newlib-1.16.0.tar.gz
> make[1]: *** [Makefile:90: newlib-1.16.0.tar.gz] Error 1
> make[1]: Leaving directory '/xen/stubdom'
> make: *** [Makefile:151: install-stubdom] Error 2
> The command '/bin/sh -c make debball' returned a non-zero code: 2
>
> I hope this helps.

Hi Mathieu,

You are missing wget in your packages.

Cheers,
Luca

>
> Mathieu Tarral
>
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
Hi Luca,

> Hi Mathieu,
>
> You are missing wget in your packages.

Indeed, installing wget solved the issue and I was able to generate
a debian package.

Still, we have a build system that silently depends on gawk and wget, and there is no check in the configure script to ensure these tools are installed.

And the errors in the build system are quite obscure, it's hard to understand that you were missing gawk or wget in the first place.

What do you think about this ?

Best,
Mathieu
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
+ CC x86 maintainers

> On 14 Jul 2022, at 10:40, Mathieu Tarral <mathieu.tarral@protonmail.com> wrote:
>
> Hi Luca,
>
>> Hi Mathieu,
>>
>> You are missing wget in your packages.
>
> Indeed, installing wget solved the issue and I was able to generate
> a debian package.
>
> Still, we have a build system that silently depends on gawk and wget, and there is no check in the configure script to ensure these tools are installed.
>
> And the errors in the build system are quite obscure, it's hard to understand that you were missing gawk or wget in the first place.
>
> What do you think about this ?

Hi Mathieu,

I’ve CC the x86 maintainer, probably they know better the reasons.

Cheers,
Luca

>
> Best,
> Mathieu
>
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
On 14.07.2022 12:12, Luca Fancellu wrote:
> + CC x86 maintainers
>
>> On 14 Jul 2022, at 10:40, Mathieu Tarral <mathieu.tarral@protonmail.com> wrote:
>>
>> Hi Luca,
>>
>>> Hi Mathieu,
>>>
>>> You are missing wget in your packages.
>>
>> Indeed, installing wget solved the issue and I was able to generate
>> a debian package.
>>
>> Still, we have a build system that silently depends on gawk and wget, and there is no check in the configure script to ensure these tools are installed.
>>
>> And the errors in the build system are quite obscure, it's hard to understand that you were missing gawk or wget in the first place.
>>
>> What do you think about this ?
>
> I’ve CC the x86 maintainer, probably they know better the reasons.

I'm afraid too much context was lost. There's no configure script which
could check gawk availability for the hypervisor build; the configure
process only applies to the tool stack part of the build. Also I seem
to recall someone else reporting an issue recently with a gawk-ism in
that script, and iirc Andrew had asked to see about finding a generic
replacement. I don't think I've seen any patch yet ...

Jan
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
On 14/07/2022 11:12, Luca Fancellu wrote:
> + CC x86 maintainers
>
>> On 14 Jul 2022, at 10:40, Mathieu Tarral <mathieu.tarral@protonmail.com> wrote:
>>
>> Hi Luca,
>>
>>> Hi Mathieu,
>>>
>>> You are missing wget in your packages.
>> Indeed, installing wget solved the issue and I was able to generate
>> a debian package.
>>
>> Still, we have a build system that silently depends on gawk and wget, and there is no check in the configure script to ensure these tools are installed.
>>
>> And the errors in the build system are quite obscure, it's hard to understand that you were missing gawk or wget in the first place.
>>
>> What do you think about this ?
> Hi Mathieu,
>
> I’ve CC the x86 maintainer, probably they know better the reasons.

These are two different problems.

The configure script does check for WGET (or more generally FETCHER) and
will object if something is not present.  This is only relevant for
tools/ which you should have been forced to run configure for anyway, so
I'm really not sure what went on there.

For valid-addrs vs all-addrs, that's come up on xen-devel before.  I
think the general consensus is that we don't want to silently depend on
gawk.

(We're trying some new project management tools.) Can you try opening a
bug here: https://gitlab.com/xen-project/xen/-/issues about the
check-endbr.sh issue?

This ought to be a 4.17 release blocker seeing as its a regression from
4.16.  (CC'ing Henry as release manager.)

~Andrew
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
Hi Andy,

> On 14 Jul 2022, at 11:33, Andrew Cooper <Andrew.Cooper3@citrix.com> wrote:
>
> On 14/07/2022 11:12, Luca Fancellu wrote:
>> + CC x86 maintainers
>>
>>> On 14 Jul 2022, at 10:40, Mathieu Tarral <mathieu.tarral@protonmail.com> wrote:
>>>
>>> Hi Luca,
>>>
>>>> Hi Mathieu,
>>>>
>>>> You are missing wget in your packages.
>>> Indeed, installing wget solved the issue and I was able to generate
>>> a debian package.
>>>
>>> Still, we have a build system that silently depends on gawk and wget, and there is no check in the configure script to ensure these tools are installed.
>>>
>>> And the errors in the build system are quite obscure, it's hard to understand that you were missing gawk or wget in the first place.
>>>
>>> What do you think about this ?
>> Hi Mathieu,
>>
>> I’ve CC the x86 maintainer, probably they know better the reasons.
>
> These are two different problems.
>
> The configure script does check for WGET (or more generally FETCHER) and
> will object if something is not present. This is only relevant for
> tools/ which you should have been forced to run configure for anyway, so
> I'm really not sure what went on there.
>
> For valid-addrs vs all-addrs, that's come up on xen-devel before. I
> think the general consensus is that we don't want to silently depend on
> gawk.
>
> (We're trying some new project management tools.) Can you try opening a
> bug here: https://gitlab.com/xen-project/xen/-/issues about the
> check-endbr.sh issue?

I will do that for you Luca as you can only do it if you are a project member right now.

>
> This ought to be a 4.17 release blocker seeing as its a regression from
> 4.16. (CC'ing Henry as release manager.)

Ack

Bertrand

>
> ~Andrew
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
On 14.07.2022 12:33, Andrew Cooper wrote:
> On 14/07/2022 11:12, Luca Fancellu wrote:
>> + CC x86 maintainers
>>
>>> On 14 Jul 2022, at 10:40, Mathieu Tarral <mathieu.tarral@protonmail.com> wrote:
>>>
>>> Hi Luca,
>>>
>>>> Hi Mathieu,
>>>>
>>>> You are missing wget in your packages.
>>> Indeed, installing wget solved the issue and I was able to generate
>>> a debian package.
>>>
>>> Still, we have a build system that silently depends on gawk and wget, and there is no check in the configure script to ensure these tools are installed.
>>>
>>> And the errors in the build system are quite obscure, it's hard to understand that you were missing gawk or wget in the first place.
>>>
>>> What do you think about this ?
>> Hi Mathieu,
>>
>> I’ve CC the x86 maintainer, probably they know better the reasons.
>
> These are two different problems.
>
> The configure script does check for WGET (or more generally FETCHER) and
> will object if something is not present.  This is only relevant for
> tools/ which you should have been forced to run configure for anyway, so
> I'm really not sure what went on there.
>
> For valid-addrs vs all-addrs, that's come up on xen-devel before.  I
> think the general consensus is that we don't want to silently depend on
> gawk.
>
> (We're trying some new project management tools.) Can you try opening a
> bug here: https://gitlab.com/xen-project/xen/-/issues about the
> check-endbr.sh issue?
>
> This ought to be a 4.17 release blocker seeing as its a regression from
> 4.16.  (CC'ing Henry as release manager.)

Except that 4.16 and older then already have regressed as well, due to
the backports they've received.

Jan
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
> On 14 Jul 2022, at 11:33, Andrew Cooper <Andrew.Cooper3@citrix.com> wrote:
>
> On 14/07/2022 11:12, Luca Fancellu wrote:
>> + CC x86 maintainers
>>
>>> On 14 Jul 2022, at 10:40, Mathieu Tarral <mathieu.tarral@protonmail.com> wrote:
>>>
>>> Hi Luca,
>>>
>>>> Hi Mathieu,
>>>>
>>>> You are missing wget in your packages.
>>> Indeed, installing wget solved the issue and I was able to generate
>>> a debian package.
>>>
>>> Still, we have a build system that silently depends on gawk and wget, and there is no check in the configure script to ensure these tools are installed.
>>>
>>> And the errors in the build system are quite obscure, it's hard to understand that you were missing gawk or wget in the first place.
>>>
>>> What do you think about this ?
>> Hi Mathieu,
>>
>> I’ve CC the x86 maintainer, probably they know better the reasons.
>
> These are two different problems.
>
> The configure script does check for WGET (or more generally FETCHER) and
> will object if something is not present. This is only relevant for
> tools/ which you should have been forced to run configure for anyway, so
> I'm really not sure what went on there.
>
> For valid-addrs vs all-addrs, that's come up on xen-devel before. I
> think the general consensus is that we don't want to silently depend on
> gawk.
>
> (We're trying some new project management tools.) Can you try opening a
> bug here: https://gitlab.com/xen-project/xen/-/issues about the
> check-endbr.sh issue?

https://gitlab.com/xen-project/xen/-/issues/26

Please complete if needed.

@marc: We have a first example of something to be used as a potential release
blocker. Also the template is quite extensive and hard to fill, we might want to simplify it.

Cheers
Bertrand

>
> This ought to be a 4.17 release blocker seeing as its a regression from
> 4.16. (CC'ing Henry as release manager.)
>
> ~Andrew
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
On 14/07/2022 11:39, Jan Beulich wrote:
> On 14.07.2022 12:33, Andrew Cooper wrote:
>> On 14/07/2022 11:12, Luca Fancellu wrote:
>>> + CC x86 maintainers
>>>
>>>> On 14 Jul 2022, at 10:40, Mathieu Tarral <mathieu.tarral@protonmail.com> wrote:
>>>>
>>>> Hi Luca,
>>>>
>>>>> Hi Mathieu,
>>>>>
>>>>> You are missing wget in your packages.
>>>> Indeed, installing wget solved the issue and I was able to generate
>>>> a debian package.
>>>>
>>>> Still, we have a build system that silently depends on gawk and wget, and there is no check in the configure script to ensure these tools are installed.
>>>>
>>>> And the errors in the build system are quite obscure, it's hard to understand that you were missing gawk or wget in the first place.
>>>>
>>>> What do you think about this ?
>>> Hi Mathieu,
>>>
>>> I’ve CC the x86 maintainer, probably they know better the reasons.
>> These are two different problems.
>>
>> The configure script does check for WGET (or more generally FETCHER) and
>> will object if something is not present.  This is only relevant for
>> tools/ which you should have been forced to run configure for anyway, so
>> I'm really not sure what went on there.
>>
>> For valid-addrs vs all-addrs, that's come up on xen-devel before.  I
>> think the general consensus is that we don't want to silently depend on
>> gawk.
>>
>> (We're trying some new project management tools.) Can you try opening a
>> bug here: https://gitlab.com/xen-project/xen/-/issues about the
>> check-endbr.sh issue?
>>
>> This ought to be a 4.17 release blocker seeing as its a regression from
>> 4.16.  (CC'ing Henry as release manager.)
> Except that 4.16 and older then already have regressed as well, due to
> the backports they've received.

Luckily not.

Remember that we decided to only backport the less invasive parts of
CET-IBT for the security fix, which is "endbr64 on all functions".

When every function is open to (ab)use, there's no point searching for
misaligned ones too.

~Andrew
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
> > > (We're trying some new project management tools.) Can you try opening a
> > > bug here: https://gitlab.com/xen-project/xen/-/issues about the
> > > check-endbr.sh issue?

I'm getting a Gitlab 404 page when trying to access this link.
Are there access restrictions in place ?

And same for:
https://gitlab.com/xen-project/xen/-/issues/26

I see the fix is already on its way !
Thank you for the great support.

Mathieu
Re: make debball fails on Ubuntu 20.04: More valid-addrs than all-addrs [ In reply to ]
Hi Matthieu,

> On 15 Jul 2022, at 09:53, Mathieu Tarral <mathieu.tarral@protonmail.com> wrote:
>
>>>> (We're trying some new project management tools.) Can you try opening a
>>>> bug here: https://gitlab.com/xen-project/xen/-/issues about the
>>>> check-endbr.sh issue?
>
> I'm getting a Gitlab 404 page when trying to access this link.
> Are there access restrictions in place ?
>
> And same for:
> https://gitlab.com/xen-project/xen/-/issues/26

Yes there are at the moment.
@marc: is there a way to let everybody read the tickets ?

Cheers
Bertrand

>
> I see the fix is already on its way !
> Thank you for the great support.
>
> Mathieu
>