Mailing List Archive

On Wed, 6 Feb 2002 16:30:25 +0100, Josef Siemes <jsiemes@web.de> wrote:


>- Configuration directly burned into the rom - no easy way to 'just change' the configuration.
If nic is pxe you can avoid this by downloading etherboot-rom via network.
The other problems you listed remain.

>BTW, would it really be possible to have some extension for etherboot
>to support PXE booting (e.g. via pxelinux)? So could etherboot create
>some PXE stack that other bootloaders would use? Did anybody already
They don't like pxe, so they did the other way around (pxe boot etherboot).

--
giulioo@pobox.com

> > In way way better and easier? When I have criticised pxelinux I think

Or "what way"... finger trouble, sorry

> I've looked into etherboot - my problems were:
> - Network card must be supported, and should (must?) be ethernet.
> I am booting token-ring clients with pxelinux.

Fair enough. It would be nice to have Etherboot ignore PXE's DHCP/TFTP
but use UNDI. I don't think there's inherently anything restricted to
ethernet, despite the name; IEEE 802 addressing, perhaps, and the
availability of polling drivers.

> - Configuration directly burned into the rom - no easy way to 'just change' the configuration.
> - The etherboot rom is card specific - no 'all purpose'-driver available

As mentioned, Etherboot can be pxe'd, and booted from other media; it's
pretty configurable via DHCP; indeed that made me prefer it to pxelinux.

> These come in part from the purpose it's designed for - it's designed
> as a boot rom, not a bootloader. In some part one could argue about
> doing things different in etherboot. I personally don't like the concept
> of putting together one file that contains everything (configuration,
> kernel, initrd etc.pp), but like to have these in separate files as in pxelinux.

I concede that etherboot's origins as a rom image still show through,
but "not a bootloader"? OK, to be accurate, NBI is the OS specific
loader which knows about Linux or DOS, not Etherboot itself. Etherboot
loads an NBI image with OS specific bootloader, kernel and initrd; I
perceive that as a *good* thing. Together they start Linux with no other
help, so that sounds like a bootloader to me.

Etherboot and NBI could be extended to tftp a separate initrd, but
equally, pxelinux could be ported to Etherboot.

> BTW, would it really be possible to have some extension for etherboot
> to support PXE booting (e.g. via pxelinux)? So could etherboot create
> some PXE stack that other bootloaders would use? Did anybody already
> try this?

I don't know if anyone has tried (Ken Yap would probably know). If NILO
can be bugfixed and ported to Etherboot's dhcp, tftp and drivers, why
not? It could give a degree of control over PXE not normally possible
with Intel derived clients "free" with nic roms and wouldn't be as
broken. No need for pxelinux as such, since Etherboot is perfectly
capable of initiating Linux directly by NBI. It might also be
interesting for the 'doze users - e.g. an image with the pxe stack and
NTLOADER in one (no reason to go through the PXE cycle if you don't need
to), maybe BIOS code for a ramdisk and an "initrd", or even a "hard
vmware" virtual machine.

On Wed, 06 Feb 2002 19:04:23 +0000, Peter Lister <P.Lister@sychron.com>
wrote:

> it's
>pretty configurable via DHCP; indeed that made me prefer it to pxelinux.

How is it more configurable than pxelinux?
Don't you think that having a config file that is not dhcpd.conf is better?

--
giulioo@pobox.com

> They don't like pxe, so they did the other way around (pxe boot etherboot).

It was a bit more complex than just hating PXE. :)

Yes, I realised quite soon that I didn't like pxe (distinct from
pxelinux), mainly because of the time wasted to do an extra tftp cycle.
We do kernel development, so we reboot systems frequently, and wanted a
fast, easily configured way to select kernels at boot time. We also have
many machines without net boots of any sort (and no prospect of adding
it); these we could Etherboot from LILO (i.e. we could initiate a
network boot from a hard disk when needed). That said, we have systems
where PXE is the only net boot option, and customers will use it, so
bootable kits need to be PXE aware. Etherboot has now been ported to
LinuxBIOS, so now I want to get LinuxBIOS supported hardware for our
next hardware purchase if at all possible.

So - I perceived benefits in using Etherboot, but a need to cope with
PXE. I'll admit to some naivete: when I first asked about chaining
Etherboot from PXE, it hadn't really occurred to me that it might be
hard. But a colleague who is conversant with x86 assembler had a look at
Etherboot, pxelinux and the spec, and ended up writing the first working
PXE to Etherboot code in a few hours. Now I can boot a new system out of
the box with vendor PXE, flash a nic with Etherboot at my convenience to
speed up the boot, then maybe flash LinuxBIOS when I have the hardware
and time to build and test; this would get boots down to 3 seconds. My
Etherboot config remains unchanged; this exercise has without doubt
saved us more time that was spent writing it.

And - the "high level" bits of Etherboot are written in C, so I have a
chance of adding to or fixing it if necessary (e.g. I'd like to help
improve the multiple NIC support).

> Don't you think that having a config file that is not dhcpd.conf is better?

No.

I do not find the pxelinux configuration style easy to use. If a minimal
boot code written in assembler has to parse a text file, then the
language won't be terribly powerful. I was not used to syslinux (other
than indirectly in Red Hat kits), having only ever used lilo for disks,
so I had no experience of the style. Yes, ISC dhcpd configuration
language is clunky - v3 is an improvement on v2, but simultaneously too
complex to be simple, and not complex enough to be powerful.

We have to configure our own product on many machines from a central
place, which I've done with DHCP and a home grown perl DHCP client (it's
not ready for the big time, it only does INFORM/ACK, not the full lease
exchange); I know of someone else working on a Perl DHCP server. Until
that's done, I generate my dhcpd.conf from perl scripts and a
configuration which knows far more than just Etherboot and IP addresses.

DHCP (distinct from ISC dhcpd and its language) is better than tftping a
text file because the non-trivial config (i18n, conditional behaviour,
maybe generated from a customer db) is server side; clients see just a
string of easily parsed data in a well known extensible format with
vendor hooks - and there is almost certainly a DHCP client in the boot
rom anyway.

This means that (a) I don't mind fairly complex DHCP config and (b) I
know there's a better future.

Peter Lister wrote:

>
> DHCP (distinct from ISC dhcpd and its language) is better than tftping a
> text file because the non-trivial config (i18n, conditional behaviour,
> maybe generated from a customer db) is server side; clients see just a
> string of easily parsed data in a well known extensible format with
> vendor hooks - and there is almost certainly a DHCP client in the boot
> rom anyway.
>
> This means that (a) I don't mind fairly complex DHCP config and (b) I
> know there's a better future.
>

On PXELINUX, all configuration information is server side in the form of
(optionally!) per-client configuration files. You don't have to interrupt
a common server (dhcpd) to install a new configuration either. Obviously,
once you're server side, you can create arbitrary complex configuration.

So what was the point of this again?

-hpa

> > This means that (a) I don't mind fairly complex DHCP config and (b) I
> > know there's a better future.
> >
>
> On PXELINUX, all configuration information is server side in the form of
> (optionally!) per-client configuration files. You don't have to interrupt
> a common server (dhcpd) to install a new configuration either. Obviously,
> once you're server side, you can create arbitrary complex configuration.

Granted; my point is that we (Sychron) have crossed over into the realm
of getting our hands dirty with DHCP anyway, and we need to maintain a
server side db of all our hardware; we do clusterwide resource control,
so how a node should boot is all tied in with e.g. speed, memory rather
than it's NICs MAC address, so it makes sense to do so in one place.
This is an argument for better a DHCP server which can create options on
the fly.

pxelinux's ability to have different configs based on IP address is
neat, but I really want choices based on arbitrary information (e.g.
hardware that is aware of the rack/chassis/slot where it lives) - we
also may have systems with no fixed IP address.

> So what was the point of this again?

My reason for asking the original question was curiosity - I'd like to
give a balanced view to anything I contribute on the pxe howto, so when
someone expresses a view I find counterintuitive, I'd like to know why.
And, I guess a bit of crude "market research", since we may provide our
software with a bootable kit.

When someone else commented that pxe chaining to etherboot was written
because we "don't like pxe", I thought I should explain that there was
more to it than just pxe hatred.

Likewise, when asked about preferring DHCP to a config file it seemed
reasonable to explain why (issues of ISC dhcpd apart) DHCP seems to be a
much better long term solution. For what it's worth, I think that
Etherboot's DHCP could do with restructuring.

Hi,

Peter Lister <P.Lister@sychron.com> schrieb am 06.02.02:
> > They don't like pxe, so they did the other way around (pxe boot etherboot).
>
> It was a bit more complex than just hating PXE. :)
>
> Yes, I realised quite soon that I didn't like pxe (distinct from
> pxelinux), mainly because of the time wasted to do an extra tftp cycle.

Why is this 'wasted time'? It leaves the decision to you if you
like etherboot, grub, pxelinux or some NT Remote Installer or ...
as the bootloader. For loading a diskless client this is a second tftp,
but we talk about seconds, not microseconds while booting. Even this
tftp is quite small (<= 32 K? At least <64K), so this won't hurt
on boot time. Compare this to a kernel with 7xx K, and an additional
initrd.

BTW: On Intel eepro 100 cards (Bootix rom)the dhcp needs 10 seconds
(standard IBM AIX dhcp server without binld), tftp of pxelinux needs
<1 second, getting the default config and showing the boot:-prompt
needs less than 1 second. These are 9 tftp requests by pxelinux for
the config file. If you like you can give the config file with
some dhcp option, so this would be one tftp instead of 9.

So what could be heavily speeded up here is the rom doing the dhcp.
Maybe one could implement some rom that takes the first dhcp answer
if it contains a special option, and don't wait for any special
pxe dhcp offers.

The need here is to have the dhcp and kernel configuration split into
different files: The dhcp config remains unchanged and is administered
by one group. The other group administers the kernel and the
filesystems. BTW, we boot some 1.500 clients via pxelinux.

> And - the "high level" bits of Etherboot are written in C, so I have a
> chance of adding to or fixing it if necessary (e.g. I'd like to help
> improve the multiple NIC support).

That's a different point - if the high level bits in pxelinux were in
C I'd have done something there. Since I only can read a bit assembler
but not write it I can see how it works, maybe why, but can't extend
it to my needs. I'd for example like to have blksize in pxelinux,
but since it's too complicated for me to implement and hpa doesn't
have the time to do it I'll go on without it. Maybe sometime parts
of syslinux/pxelinux/isolinux will be available in C so I can do
something about it.

> I do not find the pxelinux configuration style easy to use. If a minimal
> boot code written in assembler has to parse a text file, then the
> language won't be terribly powerful. I was not used to syslinux (other
> than indirectly in Red Hat kits), having only ever used lilo for disks,
> so I had no experience of the style. Yes, ISC dhcpd configuration
> language is clunky - v3 is an improvement on v2, but simultaneously too
> complex to be simple, and not complex enough to be powerful.

Consider booting without ISC dhcpd - take some IBM AIX or Sun Solaris
dhcp server, that the management refuses to change with ISC dhcpd.
The reason here is that there's a service with the server manufacturer
(here IBM), and if you take some other dhcp server you're on your own
if some update breaks your dhcp server or the dhcp server somehow
refuses to work.

> DHCP (distinct from ISC dhcpd and its language) is better than tftping a
> text file because the non-trivial config (i18n, conditional behaviour,
> maybe generated from a customer db) is server side; clients see just a
> string of easily parsed data in a well known extensible format with
> vendor hooks - and there is almost certainly a DHCP client in the boot
> rom anyway.

The standard today is PXE - quite a lot of cards are capable of pxe
booting a client out of the box. And please don't tell that the
PXE option extensions are easy - they are quite a mess if you have
to use them. The whole DHCP(with PXE extensions and all the funny stuff)
protocol is a set of extended extensions that were further extended. I
sometimes suspect that vendors didn't implement some parts of these,
since they didn't understand this part.

> This means that (a) I don't mind fairly complex DHCP config and (b) I
> know there's a better future.

For now there's DHCP/PXE for netbooting a client. Bootp is obsolete,
and I don't think that there is much need for the 'Etherboot' protocol.

Everything you get with etherboot may be get with portions
of specialized code derived from etherboot/pxelinux. So the low-level
code may come from etherboot (the rom code itself), giving PXE support
and maybe for backward compatibility support for the Etherboot style
loading some tagged image. Then there's pxelinux (or a conglomerate
of the etherboot and pxelinux code) which parses the dhcp options
and maybe loads some config file. Then there's some 'Kernel bootloader'
that tftp's a kernel and an initrd, and spits this things into the
correct memory position. I consider memdisk as part of a bootloader,
for loading (or better: reconfiguring memory for getting this as 'A:')
the floppy image.

Don't take this too serious, it's just some thought how things could
be split up in some specialized package.

So I agree with HPA that there's no need for one big package, but
some free implementation of a pxe rom which in turn uses some
pxe bootloader.

Regards,

Josef



______________________________________________________________________________
Lotto online tippen! Egal zu welcher Zeit, egal von welchem Ort.
Mit dem WEB.DE Lottoservice. http://tippen2.web.de/?x=5

Peter Lister wrote:

>
> Granted; my point is that we (Sychron) have crossed over into the realm
> of getting our hands dirty with DHCP anyway, and we need to maintain a
> server side db of all our hardware; we do clusterwide resource control,
> so how a node should boot is all tied in with e.g. speed, memory rather
> than it's NICs MAC address, so it makes sense to do so in one place.
> This is an argument for better a DHCP server which can create options on
> the fly.
>
> pxelinux's ability to have different configs based on IP address is
> neat, but I really want choices based on arbitrary information (e.g.
> hardware that is aware of the rack/chassis/slot where it lives) - we
> also may have systems with no fixed IP address.
>


Please do note that PXELINUX can have the config file specified or even
synthesized by the DHCP server. If you really want to do DHCP-centric
configuration you can do it by specifying abstract configuration.

The IP-based default is a convenience, but isn't the only option.

> Likewise, when asked about preferring DHCP to a config file it seemed
> reasonable to explain why (issues of ISC dhcpd apart) DHCP seems to be a
> much better long term solution. For what it's worth, I think that
> Etherboot's DHCP could do with restructuring.


I would have to disagree with that. DHCP has a fair number of problems,
one of them being that you simply can't specify abitrary amounts of data,
since it needs to fit in a single UDP packet *and* fit in the buffers on
the target system -- in practice (meaning: this is how real life
implementations work, like it or not) that means it has to fit in an
single Ethernet packet. A configuration file can be arbitrarily long.

There is also the particular issue with Etherboot that it interprets just
about the entire site-specific set of codes as its own, without having any
kind of magic key or anything. PXELINUX has a pretty small set (what it
needs *before* the configuration file loads) of site-specific codes, and
they're keyed. You basically has to do something like that, especially in
a PXE environment.

-hpa

Josef Siemes wrote:

>
> That's a different point - if the high level bits in pxelinux were in
> C I'd have done something there. Since I only can read a bit assembler
> but not write it I can see how it works, maybe why, but can't extend
> it to my needs. I'd for example like to have blksize in pxelinux,
> but since it's too complicated for me to implement and hpa doesn't
> have the time to do it I'll go on without it. Maybe sometime parts
> of syslinux/pxelinux/isolinux will be available in C so I can do
> something about it.
>


At some point I might do a C-based bootloader -- probably an actually
working implementation of Genesis -- but the *LINUX series are designed to
be as small as possible, which means that going to C, using a 16-bit C
compiler, isn't really practical -- free 16-bit compilers are uniformly
nonoptimizing.

blksize is complicated because the TFTP spec for blksize is broken --
"blksize 1024" doesn't mean "give me a 1024-byte block if you can", rather
it means "give me any block size 1024 bytes or less." Dealing with
block sizes that are not multiples of 1024 means doing some very
sophisticated buffering, which impacts memory management. Unlike an
on-disk filesystem we can't afford to flush a pending connection should we
need the space; the connection needs to be maintained.

I actually have a "blksize2" option in tftp-hpa that resolves this
problem, however, if PXELINUX ever used it it would only work against
tftp-hpa.

Since you have to turn off blksize anyway if you want to support early PXE
stacks, how much does this matter?

I suspect the right way to do this is to negotiate "blksize 1024", set
SECTOR_SIZE in PXELINUX to 1024, and if the server comes back with blksize
other than 512 or 1024, EOPTNEG the connection and try again without
blksize -- if block size is 512 then we would always treat blocks in
pairs, which isn't a problem, obviously.

Now, questions:

a) Is this worthwhile enough that I should spend time on it? How much do
you think it will buy you?

b) Should we aim for an even larger blksize by default? What about PXE
stacks that don't handle defragmentation correctly?

Either way, this will have to wait until I get 1.70/2.00 out (haven't
decided yet if I should bump it up to 2.00) -- this is the code
restructuring to support multifile initramfs.

-hpa

Hi,

"H. Peter Anvin" <hpa@zytor.com> schrieb am 07.02.02:
> Josef Siemes wrote:
>
> >
> > That's a different point - if the high level bits in pxelinux were in
> > C I'd have done something there.
>
> At some point I might do a C-based bootloader -- probably an actually
> working implementation of Genesis -- but the *LINUX series are designed to
> be as small as possible, which means that going to C, using a 16-bit C
> compiler, isn't really practical -- free 16-bit compilers are uniformly
> nonoptimizing.

It's just some idea to have it C-based. I think of it as some asm
routines with C interface, so if you want to deal with A20 handling
or some other complicated task you'd still have asm, some high-level
processing (like e.g. parsing the config file) would be in C.
I think this is how etherboot handles this.

> Since you have to turn off blksize anyway if you want to support early PXE
> stacks, how much does this matter?

It would be nice to have this option (e.g. set via a DHCP option),
usually this would be turned off, and only activated for clients were
one knows that it works.

> Now, questions:
>
> a) Is this worthwhile enough that I should spend time on it? How much do
> you think it will buy you?

Since we have some of the clients booting over an 1 MBit line it
would save some seconds booting these. It's just an idea and
nice to have, but it also works without blksize. Since you already
told me a while ago that blksize support is quite much work I think
it isn't needed. Maybe you could keep blksize in mind while
restructuring the code.

> b) Should we aim for an even larger blksize by default? What about PXE
> stacks that don't handle defragmentation correctly?

I think that the standard behaviour should stay with 512 Bytes blksize
(read: without the blksize option at all). This works reliably with
almost all PXE stacks.

We made some measurements about different blksizes at different bandwidth,
and it turned out that you gain few with LAN lines, but the lower the bandwith the more you gain with higher blksize. So higher blksize is only needed if you have some WAN line with low bandwidth.

Regards,

Josef
________________________________________________________________
Keine verlorenen Lotto-Quittungen, keine vergessenen Gewinne mehr!
Beim WEB.DE Lottoservice: http://tippen2.web.de/?x=13

Josef Siemes wrote:

>
> We made some measurements about different blksizes at different bandwidth,
> and it turned out that you gain few with LAN lines, but the lower the bandwith the more you gain with higher blksize. So higher blksize is only needed if you have some WAN line with low bandwidth.
>


I suspect you actually mean "high latency" rather than "low bandwidth"
since the main benefit is to reduce ping-pong. This actually begs the
question if a pretransmit option (send a packet before it gets the
preceeding ACK) on the server would improve performance -- it would help
latency somewhat, but may increase retransmits.

Over high-latency links TFTP basically blows. I was thinking about
doing a "TFTP tcp option" (basically, if the client knows how to open a
TCP socket, it could request the tcp option and get back a port number
instead of a full TFTP connection.) However, it's unclear to me how
well this would work with PXE -- it's clear that PXE doesn't contain a
TCP stack; what isn't clear is if it is possible to use the IP layer in
PXE without in effect reimplementing it; it doesn't have a direct API,
but there are some hints that it might be accessible via the UNDI API...

*Sigh*

-hpa