Mark Knecht posted on Tue, 25 Jun 2013 15:51:14 -0700 as excerpted:
> This is related to my thread from a few days ago about the
> disappointing speed of my RAID6 root partition. The goal here is to get
> the machine booting from an SSD so that I can free up my five hard
> drives to play with.
FWIW, this post covers a lot of ground, too much I think to really cover
in one post. Which is why I've delayed replying until now. I expect
I'll punt on some topics this first time thru, but we'll see how it
goes...
> SHORT SUMMATION: I've tried noninitrd and noraid in the kernel line of
> grub.conf but I keep booting from old RAID instead of the new SSD.
> What am I doing wrong?
>
> What I've done so far:
>
> 1) I've removed everything relatively non-essential from the HDD-based
> RAID6. It's still a lot of data (40GB) but my Windows VMs are moved to
> an external USB drive as is all the video content which is on a second
> USB drive so the remaining size is pretty manageable.
OK...
> 2) In looking around for ways to get / copied to the SDD I ran across
> this Arch Linux page called "Full System Backup with rsync":
>
> https://wiki.archlinux.org/index.php/Full_System_Backup_with_rsync
> Basically it boiled down to just a straight-forward rsync command, but
> what I liked about the description what that it can be done on a live
> system. The command in the page is
>
> rsync -aAXv /* /path/to/backup/folder
> --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost
+found}
>
> which I have modified to
>
> rsync -avx /* /path/to/backup/folder
> --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost
+found}
>
> because I don't use (AFAICT) any of the ACL stuff and the command simply
> wouldn't do anything.
For ACL you're probably correct. But you might be using xattributes. Do
you have any of the following USE flags turned on: caps xattr filecaps ?
(Without going into an explanation of the specific distinction between
the USE flags above, particularly caps and filecaps.) What filesystem do
you use on / (and /usr if separate), and if appropriate, are the extended
attributes and security label kernel options enabled for it?
For example, here I have ext4, reiserfs and btrfs enabled and use or have
used them on my various root filesystems, as well as tmpfs with the
appropriate options since I have PORTAGE_TMPDIR pointed at tmpfs (and
also devtmpfs needs some of the options):
zgrep 'REISER\|EXT4\|TMPFS\|BTRFS' /proc/config.gz
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT23=y
# CONFIG_EXT4_FS_POSIX_ACL is not set
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_REISERFS_FS=y
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
CONFIG_REISERFS_FS_XATTR=y
# CONFIG_REISERFS_FS_POSIX_ACL is not set
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_BTRFS_FS=y
# CONFIG_BTRFS_FS_POSIX_ACL is not set
# CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set
# CONFIG_BTRFS_FS_RUN_SANITY_TESTS is not set
# CONFIG_BTRFS_DEBUG is not set
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
tmpfs only has ACL on for devtmpfs (and I'm not sure I need that, but to
avoid both security issues and broken device functionality...). The
others don't have that on, but where appropriate, they have XATTR on, as
well as FS_SECURITY. (Again, this is really only surface coverage,
here. TBH I don't fully understand the depths myself, certainly not well
enough to be comfortable discussing it in depth, tho I'm reasonably sure
I have the options I want enabled, here.)
The deal is that file capacities in one form or another can be used to
avoid having to SETUID root various executables that would otherwise need
it, which is a good thing since that reduces the security vulnerability
window that SETUID root otherwise opens, often necessarily.
And these file capacities are implemented using xattrs. So if your
system is setup to use them, a good thing from a security perspective but
somewhat complicated by the kernel config requirements in addition to the
USE flags, you'll probably want to use the -X option, tho you should
still be safe without -A (tho it shouldn't hurt).
However, the penalty for NOT using -X, provided you're not using xattrs
for anything else, should simply be that you'll need to become root to
run some commands that would otherwise be runnable without root (with the
corresponding open security window, should it be possible for a cracker
to get those commands running as root to do something unintended). So
the potential cost of getting it wrong is actually quite limited, unless
you happen to be the target of a cracker with both good timing and a
reasonable skill level, as well.
And of course if you have only part of the pieces above enabled, say the
appropriate filesystem options in the kernel but not the USE flags, or
the reverse, then you're not covered and the rsync options won't matter
in any case.
But the -AX options shouldn't do any harm in any case, so here I'd have
just left them on, making it -avxAX.
Meanwhile, while I always see people worried about copying a live
filesystem around, I've never had a problem here simply doing a
cp --archive, or the equivalent in mc (midnight commander, ncurses-based
commander-style dual-pane file manager).
What I do for root is use a root-bind script:
#!/bin/bash
me=${0##*/}
case $me in
rootbind) mount --bind / /mnt/rootbind;;
rootbindu) umount /mnt/rootbind;;
*) echo rootbind: bad call; exit 1;;
esac
(That allows the script to be called rootbind, with a symlink to it
called rootbindu, that does the corresponding umount.)
What a bind-mount does is mount an already mounted filesystem at a
different mountpoint. In particular, it does NOT do recursive mounts
(tho there's another mount option that copies the full mount tree, it's
just not what I want here), so what I'm using it for here is to get a
"clean" copy of the rootfs, WITHOUT other filesystems such as /dev and
/home mounted on top.
Then I can do a nice clean cp --archive of my rootfs to a (normally
freshly formated, so cp and rsync would accomplish the same thing) backup
root, and that's what I've used for backup, for years.
And I test those backups too, and occasionally reboot to the backup and
do a clean mkfs and copy back from the backup to the normally working
copy too, just to take care of fragmentation and any possibility of
unresolved filesystem damage or bitrot that might have set in, as well as
ensuring that I can switch to the backups for operational use by actually
doing so. So I know the technique works for me.
Now if I was running some active database that was continuing to update
as I did my copy, yes, that would be a problem, and I'd want to do a
snapshot or otherwise "freeze" the live filesystem in ordered to get a
good self-consistent copy. But, for root anyway, unless I'm trying to do
an emerge --update in the background or something at the same time (and
why would I, both the copy and the update could be trying to access the
filesystem at once, slowing both down, and it needlessly complicates
things, so there's no purpose to doing so), a simple cp --archive of the
live filesystem, from the bind-mount so I get JUST the root filesystem,
no more no less, is sufficient.
For /home, there's a /bit/ more concern, say with the firefox sqlite
databases, if I'm browsing at the same time I'm trying to do the backup.
However, that's simple enough to avoid. Just don't do anything that's
going to be actively changing the filesystem at the same time I'm trying
to make an accurate backup of it.
Of course with your VMs that's a bit different story, rather like the
active database case. A snapshotting filesystem (like btrfs) or sub-
filesystem block-device layer (like lvm2) can be used here, taking the
snapshot and copying it while the activity continues on the live
filesystem, or, likely a simpler solution for those where it's possible,
just do the copy when the database/vms aren't active and in use.
But unless your vms/databases are files on your rootfs, that shouldn't be
a problem with the rootfs backup, in any case. And if they are and you
can't shut down the vms/databases for long enough to do a backup, I'd
personally have questions of the strategy that put them on rootfs to
begin with, but whatever, THAT is when you'd need to worry about taking
an accurate copy of the live rootfs, but ideally, that's not a case you
need to worry about, and indeed, from what I've read it's not a problem
in your case at all. =:^)
> I ran this command the first time to get 98% of everything copied while
> in KDE, but before I moved forward I exited KDE, stopped X and ran it as
> root from the console. After the second run it didn't pick up any new
> file changes so I suspect it's pretty close to what I'd get dealing with
> a Live CD boot. (COMMENTS?)
As the above commentary should indicate, if anything I think you're being
overly cautious. In the vast majority of cases, a simple cp --archive,
or your equivalent rsync, should be fine. The caveat would be if you
were trying to backup the vms while they were in operation, but you've
taken care of that separately, so (with the possible caveat about file
capacities and xattrs) I believe you're good to go.
> 3) I added a new boot options in grub.conf:
(Long) Note in passing: You should probably look into upgrading to grub2
at some point. Now may not be a good time, as you've got a lot on your
plate right now as it is, but at some point. Because while there's a bit
of a learning curve to getting up and running on grub2, it's a lot more
flexible than grub1, with a lot more troubleshooting possible if you're
not getting the boot you expect, and direct support of all sorts of fancy
stuff like mdadm, lvm2, btrfs, zfs, etc, as well as an advanced command-
line shell much like sh/bash itself, so it's very possible to browse your
whole filesystem directly from inside grub, as I said, making
troubleshooting **MUCH** easier. Plus its scripting (including if/then
conditionals and variable handling much like bash) and menu system make
all sorts of advanced boot configs possible.
And while I'm at it, I'd strongly recommend switching to gpt partitioning
from the old mbr style partitions, either before switching to grub2 or at
the same time. GPT is more reliable (checksummed partitioned table with
two copies, one at the beginning and one at the end of the device, unlike
mbr with just one, with no checking, that if it goes bad...) and less
complicated (no primary/secondary/logical partition distinction, up to
128 partitions handled by default, with a possibility for even more if
you setup a larger gpt than mbr. Plus, it allows partition labels much
like the filesystem labels people already use, only on the partitions
themselves, so they don't change with the filesystem. That in itself
makes things much easier, since with labels it's much easier to keep
track of what each partition is actually for.
The reason I recommend switching to gpt before switching to grub2, is
that gpt has a special BIOS-reserved partition type, that grub2 can make
use of to store its core (like grub1's stage-1.5 and 2), making the grub2
administration and updates less problematic than they might be
otherwise. (This of course assumes bios, not efi, but grub2 works with
efi as well, I'm just not familiar with its efi operation, and besides,
efi folks are likely to already be running grub2 or something else,
instead of legacy grub1, so it's likely a reasonably safe assumption.)
Actually, when I switched to gpt here, while still on grub1, I was
forward thinking enough to setup both a bios-reserved partition, and an
efi-reserved partition, even tho neither one was used at the time. They
were small (a couple MB for the BIOS partition, just under 128 MB for the
efi partition, so the both fit in 128 MB, an eighth of a gig). Then I
upgraded to grub2 and it found and used the gpt bios partition without
issue, instead of having to worry about fitting it in slack space before
the first partition or whatever. The efi-reserved partition is still
unused, but it's there in case I upgrade to efi on this machine (I doubt
I will as I have no reason to), or decide to fit the existing disk into a
new machine at some point, without full repartitioning.
(FWIW, I use gptfdisk, aka gdisk, as my gpt-partitioner analogous to
fdisk. However, gparted has supported gpt for awhile, and even standard
fdisk, from the util-linux package, has (still experimental) gpt support
now. Tho the cfdisk variant (also from util-linux) doesn't have gpt
support yet, but cgdisk, from the gptfdisk package, does, and that's the
executable from the gptfdisk package I tend to use here. (I use gdisk -l
to spit out the partition list on the commandline, similar to cat-ing a
file. That's about it. I use cgdisk for actual gpt partition table
editing.))
It's just that reading your post, I'm translating to grub2 in my head,
and thinking how much simpler grub2 makes troubleshooting, when you can
effectively browse all hard drives in read-only mode directly from grub,
not only browsing around to know for sure that a particular partition is
the one you need, but paging thru various files in the kernel
Documentation dir, for instance, to get options to plug in on the kernel
commandline in grub, etc. It really does make troubleshooting early boot
problems MUCH easier, because grub2 simply gives you far more to work
with in terms of troubleshooting tools available to use at the grub
prompt.
The one caveat for gpt is for people multi-booting to other than Linux.
From what I've read, MS does support GPT, but with substantially less
flexibility (especially for XP, 7 is better) than Linux. I think it can
install to either, but switching from one to the other without
reinstalling is problematic, or something like that, whereas with Linux
it's simply ensuring the appropriate support is configured into (or
available as modules if you're running an initr*) the kernel. (I have
little idea how Apple or the BSDs work with GPT.)
But while you do MS, AFAIK it's all in VMs, so that shouldn't be a
problem for you, so gpt should be fine.
And of course grub2 should be fine as well, gpt or not, but based on my
experience, gpt makes the grub2 upgrade far easier, at least as long as
there's a bios-reserved partition setup in gpt already, as there was here
when I did my grub2 upgrade, since I'd already done the gpt upgrade
previously.
But as I said, now may not be the best time to think about that as you
have enough on your plate ATM. Maybe something for later, tho... Or
maybe consider doing gpt now, since you're repartitioning now, and grub2
later...
(grub1 menu entries:)
> title fastVM 3.8.13-gentoo using LABEL (SSD, initramfs in kernel)
> root (hd5,0)
> kernel (hd0,0)/boot/bzImage-3.8.13-gentoo root=LABEL=fastVM video=vesafb
> vga=0x307title
>
> fastVM 3.8.13-gentoo using LABEL (SSD, initramfs in kernel)
> root (hd5,0)
> kernel (hd0,0)/boot/bzImage-3.8.13-gentoo root=/dev/sda1 video=vesafb
> vga=0x307
I'll assume that "vga=0x307title" is a typo, and that "title" starts the
second menu entry...
... Making the difference between the two entries the root=LABEL=fastVM,
vs root=/dev/sda1
> I am relatively confident that (hd5,0) is the SSD. I have 6 drives in
> the system - the 5 HDDs and the SSD. The 5 hard drives all have multiple
> partitions which is what grub tells me using tab completion for the line
>
> root(hdX,
>
> Additionally the SDD has a single partition to tab completion on
> root(hd5 finishes with root(hd5,0). I used /dev/sda as that's how it's
> identified when I boot using RAID.
This is actually what triggered the long grub2 note above. "Relatively
confident", vs. knowing, because with grub2's mdadm support, you can
(read-only) browse all the filesystems in the raid, etc (lvm2, etc, if
you're using that...), as well. So you know what's what, because you can
actually browse it, direct from the grub2 boot prompt.
However, while my grub1 knowledge is getting a bit rusty now, I think
you're mixing up grub's root(hdX,Y) notation, which can be thought of as
sort of like a cd in bash, simply changing the location you're starting
from if you don't type in the full path, with the kernel's root=
commandline option.
Once the kernel loads (from hd0,0 in both entries), its root= line may
have an entirely DIFFERENT device ordering, depending on the order in
which it loaded its (sata chipset, etc) drivers and the order the devices
came back in the device probes it did as it loaded them.
That's actually why kernel devs and udev folks plus many distros tend to
recommend the LABEL= (or alternatively UUID=) option for the kernel's
root= commandline option, these days, instead of the old /dev/sdX style,
because in theory at least, the numbering of /dev/sdX devices can change
arbitrarily. In fact, on most home systems with a consistent set of
devices appearing at boot, the order seldom changes, and it's *OFTEN* the
same as the order as seen by grub, but that doesn't HAVE to be the case.
Of course the monkey wrench in all this is that as far as I'm aware
anyway, the LABEL= and UUID= variants of the root= kernel commandline
option *REQUIRE* an initr* with working udev or similar (I'm not sure if
busybox's mdev supports LABEL=/UUID= or not), which might well be a given
on binary-based distros that handle devices using kernel modules instead
of custom built-in kernel device support, and thus require an initr* to
handle the modules load anyway, but it's definitely *NOT* a given on a
distro like gentoo, which strongly encourages building from source and
where many, perhaps most users, use a custom-built kernel with the
drivers necessary to boot builtin, and thus may well not require an initr*
at all. For initr*-less boots, AFAIK root=/dev/* is the only usable
alternative, because the /dev/disk/by-*/ subdirs that LABEL= and UUID=
depends on are udev userspace created, and those won't be available for
rootfs mount in an initr*-less boot.
> Now, the kernel has the initrd built into it so if it cannot be
> turned off I guess I'll try building a new kernel without it. However I
> found a few web pages that also said RAID could be disabled using a
> 'noraid' option which I thought should stop the system from finding the
> exiting RAID6 but no luck.
FWIW, the best reference for kernel commandline options is the kernel
documentation itself. Sometimes you need more, but that's always the
place to look first.
Specifically, $KERNELDIR/Documentation/kernel-parameters.txt , for the
big list in one place, with additional documentation often provided in
the various individual files documenting specific features.
kernel-parameters.txt lists noinitrd:
noinitrd [RAM] Tells the kernel not to load any configured
initial RAM disk.
So that should work. It doesn't say anything about it not working with
a built-in initramfs, either, so if it doesn't, there's a bug in that it
either should say something about it, or it should work.
FWIW, depending on what initramfs creation script you're using and its
content, you should be able to tell whether the initramfs activated or
not.
Here, I /just/ /recently/ started using dracut, since it seems multi-
device btrfs as root doesn't work reliably otherwise, and that's what I'm
using as my rootfs now (btrfs raid1 mode on dual SSDs, I could only get
it to mount the dual-device btrfs raid1 in degraded mode, seeing only one
of the two devices, without the btrfs device scan in the initramfs, tho a
google says some people have it working, <shrug>).
But even booting to my more traditional reiserfs rootfs backups on the
"spinning rust", where booting from the initramfs isn't mandatory, I can
tell whether the initramfs was loaded or not by the boot-time console
output. Among other things, if the initramfs is loaded and run, then
/proc and /run are already loaded when the openrc service that would
normally mount them gets run, because the initramfs mounted them. But
apparently the initramfs mounts at least /run with different permissions,
so openrc mentions that it's changing permissions on /run when it runs
after the initramfs, but simply mounts it with the permissions it wants,
when the initramfs hasn't run.
But unfortunately, I've not actually tried the noinitrd kernel commandline
option, so I can't VERIFY that it works here, with my now builtin
initramfs. I'll have to reboot to try that, and will try to get back to
you on that. (Note to self. Test the root=LABEL with initramfs-less
boot too, while I'm at it.)
If you're using a dracut-created initr*, then there's several other
helpful kernel commandline options that it hooks. See the
dracut.comandline manpage for the full list, but rd.break and its
rd.break=<brkpoint> variants allow dropping to the initr*'s builtin shell
(AFAIK dash by default for dracut, but bash is an option... which I've
enabled) at various points, say right before or right after the initr*'s
udev runs, right before mounting the real rootfs, or right before the
final switchroot and start of the init on the real rootfs. If you're
using some other initr* creator, obviously you'd check its documentation
for similar options.
I know rd.break works here, as I tested it while I was figuring out how
to work this new-to-me initramfs thing. And it's obvious that I'm in the
initr*'s bash, because its shell prompt isn't anything like my customized
shell prompt.
Meanwhile, I DO NOT see "noraid" listed in kernel-parameters.txt, altho
that doesn't mean it doesn't (or didn't at some point) exist. I DO see a
couple raid-options, md= and raid=, however, with references to
Documentation/md.txt.
Based on the md.txt file, it appears raid=noautodetect is the option
you're looking for. This also matches my now slightly rusty recollection
from when I ran mdraid before. noraid didn't look quite right, but
raid=noautodetect looks much closer to what I remember.
(If you're using dracut-based initr*, there's a similar option for it,
rd.auto, rd.auto=(0|1), that defaults off with current versions of dracut,
according to the dracut.cmdline manpage. That governs autoassembly of
raid, lvm, etc. But since it already defaults off, unless you're running
an old version where that defaulted on, or have it as part of your
builtin commandline as configured in your kernel or something, that
shouldn't be your problem.)
> Does anyone here have any ideas? fdisk info follows at the end.
> Ask for anything else you want to see.
>
> If I can get to booting off the SSD then for the next few days I
> could build different RAIDs and do some performance testing.
Hmm... This didn't turn out to be so hard to reply to after all. Maybe
because I kept my initr* remarks to dracut-based, which is all I know
anyway...
Some other remarks...
FWIW, if you're running an md-raid rootfs, at least with gpt and a
dedicated bios partition, installing grub2 is easier than installing or
updating grub1, as well. I remember the pain it was to install grub1 to
each of the four drives composing my raid, back when I had that setup, in
particular, the pain of trying to be sure I was installing to the
physical drive I /thought/ I was installing to, while at the same time
ensuring it was actually pointed at the /boot on the same drive, not at
the /boot on a different drive, so that if that drive was the only one I
had left, I could still boot from it. The problem was that because I was
on mdraid, grub1 was detecting that and I had to specify the physical
device one way to tell it where to install stage1, and a different way to
tell it where to put stage2 in /boot.
With grub2, things were so much easier that I had trouble believing I'd
actually installed it already. But I rebooted and it worked just fine,
so I had. Same thing when I switched to the pair of ssds with btrfs in
raid1 mode as my rootfs. I installed to the first one... and thought
surely there was another step I had missed, but there it was. After
reboot to test, I installed to the second one, and rebooted to it (using
the boot selector in the BIOS) to test. All fine. =:^)
Of course part of that, again, is due to using gpt with the reserved bios
partition for grub to put its stage2 core in, quite apart from what it
puts in /boot. I suppose I'd have had a similar problem as I did with
grub1, if I was still using mbr or didn't have a reserved bios partition
in my gpt layout, and grub had to stick the stage2 core either in slack
space before the first partition (if there was room to do so), or in
/boot itself, and hope the filesystem didn't move things around afterward
(which reiserfs did do a couple of times to me back with grub1, tho it
wasn't usually a problem).
I glossed over what to do with non-dracut-based initr*, as I've not used
anything other than dracut and direct no-initr*, and dracut's only very
recently. However, I'd be quite surprised if others didn't have
something similar to dracuts rd.break options, etc, and you certainly
should be able to tell whether the initr* is running or not, based on the
early-boot console output. Of course, whether you're /familiar/ enough
with that output or not to tell what's initr* and what's not, is an
entirely different question, but if you know well what one looks like,
the other should be enough different that you can tell, if you look
closely.
From the initrd, it should be possible to mount something else other than
the old raid as rootfs, and by that time, you'll have the kernel
populated /dev tree to work with as well as possibly the udev populated
disk/by-* trees, so finding the right one to mount shouldn't be an issue
-- no worries about kernel device order not matching grub device order,
because you're past grub and using the kernel already, by that point.
That was definitely one of the things I tested on my dracut-based initr*,
that from within the initr* (using for instance rd.break=pre-mount to
drop to a shell before the mount), I could find and mount backup root
filesystems, should it be necessary.
From within the initrd, you should be able to mount using label, uuid or
device, any of the three, provided of course that udev has populated the
disk/by-label and by-uuid trees, and I could certainly mount with either
label or device (I didn't try uuid), using my dracut-based initramfs,
here.
So really, you shouldn't need the noinitrd option for that. Tho as long
as your selected rootfs doesn't /require/ an initr* to boot (as my multi-
device btrfs rootfs seems to here), you should be able to boot to it with
an appropriate kernel commandline root= option, with or without the
initr*.
> c2RAID6 ~ # fdisk -l
>
> Disk /dev/sda: 128.0 GB [snip]
>
> Device Boot Start End Blocks Id System
> /dev/sda1 2048 250069679 125033816 83 Linux
>
> Disk /dev/sdb: 500.1 GB
>
> Device Boot Start End Blocks Id System
> /dev/sdb1 * 63 112454 56196 83 Linux
> /dev/sdb2 112455 8514449 4200997+ 82 Linux swap
> /dev/sdb3 8594775 976773167 484089196+ fd Linux raid
>
> Disk /dev/sdc: 500.1 GB
>
> Device Boot Start End Blocks Id System
> /dev/sdc1 * 63 112454 56196 83 Linux
> /dev/sdc2 112455 8514449 4200997+ 82 Linux swap
> /dev/sdc3 8594775 976773167 484089196+ fd Linux raid
>
> Disk /dev/sde: 500.1 GB
>
> Device Boot Start End Blocks Id System
> /dev/sde1 2048 8594774 4296363+ 83 Linux
> /dev/sde3 8594775 976773167 484089196+ fd Linux raid
>
> Disk /dev/sdf: 500.1 GB
>
> Device Boot Start End Blocks Id System
> /dev/sdf1 2048 8594774 4296363+ 83 Linux
> /dev/sdf3 8595456 976773167 484088856 fd Linux raid
>
> Disk /dev/sdd: 500.1 GB
>
> Device Boot Start End Blocks Id System
> /dev/sdd1 * 63 112454 56196 83 Linux
> /dev/sdd2 112455 8514449 4200997+ 82 Linux swap
> /dev/sdd3 8594775 976773167 484089196+ fd Linux raid
>
> Disk /dev/md3: 1487.1 GB
>
> c2RAID6 ~ #
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman