Following some very nasty experiences with Veritas (vxvm & vxfs)
on one of our systems, we are currently in paranoia mode about
our ability to restore from backups, and have been doing more
than usually extensive tests.
I have come across a serious problem with ONTAP restore, but I am
in communication with NetApp about that and it's not the subject
of this message. Maybe later on ... :-)
NetApp have said on a number of occasions that it's a design feature
of their dump format that it is a compatible extension of good old
BSD dump format, so that if all else fails one can feed them to a
BSD-type restore program (losing ACLs and such-like info, of course).
Solaris "ufsrestore" is usually explicitly or implicitly mentioned.
So I have been testing that, and have fallen over a problem that
I knew about (at least since May 2000, as I see I mentioned it in
passing on toasters then) but have been ignoring. Maybe it's time
to do something about it! What happens is that ufsrestore says
write error extracting inode NNNNN, name ./path/name/to/file
write: Bad address
and gives up. One can see that it has half-written the file involved.
It seems to happen only on files that have holes (often many of them)
at odd multiples of 4K: in our case they are usually *.pag files that
are part of (n)dbm databases.
Now I very strongly suspect that this is a bug in Solaris ufsrestore,
not in the NetApp dump contents, and I would like to be able to report
it to Sun and get it fixed. But it seems to be very difficult to
reproduce with a small example: it's far from the case that every
file with oddly-aligned holes causes a problem, or even that a file
with exactly the same hole pattern will provoke the bug if it occurs
at a different point in the dump.
Even worse, it apparently depends on what sort of filing system
ufsrestore is restoring into: I have never had it happen if it is
a local ufs filing system, but often if it was an nfs one (usually
on a NetApp filer, of course). If all these variables are reproduced,
though, the effect is repeatable.
It's possible that Sun would reject such a bug report unless one
could show that it failed on a dump generated by Solaris ufsdump.
Solaris ufs filing systems have always been blocked at 8K by default
(so that hole boundaries must be on multiples of 8K). On UltraSPARC
(sun4u) systems one can't even mount ones that are blocked at 4K any
longer. I tried making a ufsdump of a filing system blocked at 4K on
a SPARCstation 5 (sun4m) - sometimes it's useful to have such an out
of date machine as one's personal workstation! - with a suitably holey
file in it, but I couldn't get the ufsrestore bug to show up... :-(
If anyone else has ever come across this problem, and/or has any
suggestions on how to proceed with homing in on the bug, I would
very much like to hear from them. The latest experiments were done
with Solaris 8 ufsrestore as patched by 109091-05, but as I said
above I believe the bug has been there for many years.
Chris Thompson University of Cambridge Computing Service,
Email: cet1@ucs.cam.ac.uk New Museums Site, Cambridge CB2 3QH,
Phone: +44 1223 334715 United Kingdom.
on one of our systems, we are currently in paranoia mode about
our ability to restore from backups, and have been doing more
than usually extensive tests.
I have come across a serious problem with ONTAP restore, but I am
in communication with NetApp about that and it's not the subject
of this message. Maybe later on ... :-)
NetApp have said on a number of occasions that it's a design feature
of their dump format that it is a compatible extension of good old
BSD dump format, so that if all else fails one can feed them to a
BSD-type restore program (losing ACLs and such-like info, of course).
Solaris "ufsrestore" is usually explicitly or implicitly mentioned.
So I have been testing that, and have fallen over a problem that
I knew about (at least since May 2000, as I see I mentioned it in
passing on toasters then) but have been ignoring. Maybe it's time
to do something about it! What happens is that ufsrestore says
write error extracting inode NNNNN, name ./path/name/to/file
write: Bad address
and gives up. One can see that it has half-written the file involved.
It seems to happen only on files that have holes (often many of them)
at odd multiples of 4K: in our case they are usually *.pag files that
are part of (n)dbm databases.
Now I very strongly suspect that this is a bug in Solaris ufsrestore,
not in the NetApp dump contents, and I would like to be able to report
it to Sun and get it fixed. But it seems to be very difficult to
reproduce with a small example: it's far from the case that every
file with oddly-aligned holes causes a problem, or even that a file
with exactly the same hole pattern will provoke the bug if it occurs
at a different point in the dump.
Even worse, it apparently depends on what sort of filing system
ufsrestore is restoring into: I have never had it happen if it is
a local ufs filing system, but often if it was an nfs one (usually
on a NetApp filer, of course). If all these variables are reproduced,
though, the effect is repeatable.
It's possible that Sun would reject such a bug report unless one
could show that it failed on a dump generated by Solaris ufsdump.
Solaris ufs filing systems have always been blocked at 8K by default
(so that hole boundaries must be on multiples of 8K). On UltraSPARC
(sun4u) systems one can't even mount ones that are blocked at 4K any
longer. I tried making a ufsdump of a filing system blocked at 4K on
a SPARCstation 5 (sun4m) - sometimes it's useful to have such an out
of date machine as one's personal workstation! - with a suitably holey
file in it, but I couldn't get the ufsrestore bug to show up... :-(
If anyone else has ever come across this problem, and/or has any
suggestions on how to proceed with homing in on the bug, I would
very much like to hear from them. The latest experiments were done
with Solaris 8 ufsrestore as patched by 109091-05, but as I said
above I believe the bug has been there for many years.
Chris Thompson University of Cambridge Computing Service,
Email: cet1@ucs.cam.ac.uk New Museums Site, Cambridge CB2 3QH,
Phone: +44 1223 334715 United Kingdom.