Mailing List Archive

NDMP backup speeds
Toasters,
What is the general consensus for NDMP backup speeds? I have four older
F760Cs that are getting about 5-6 MB/s to local tape drives. The problem is
that I also have another nine filers (1 F630, 1 F720, 1 F740, 6 F760C) that are
backing up via 3-way NDMP and we are lucky to see 4-5 MB/s; most are in the 1-2
MB/s range. And don't even get me started on the Solaris/SunOS/Linux/HP-UX/DEC
clients that backup through the drives attached to the filers (0.1 - 2 MB/s).

We were looking at upgrading the tape drives and such, but we have done a
few test dumps from filer to filer, filer to LTO-2, and filer to LTO-2 attached
to a Sun, and the numbers didn't improve much (definitely not enough to justify
buying a new tape library / tape technology).

Just a little background:
We are using DLT7000 tape drives, which should have a maximum through put of
about 10 MB/s (2:1 compression), contained in a Quantum P3000 library. We have
all our filers on two separate networks; one for general data sharing and a
private one for our backups. The private network is all GbE (fibre) using the
Alteon GbE cards (Gigabit Ethernet Controller II).

We are doing qtree dumps, and while some of the filers have only four qtrees
per volume, others have over 40. Each qtree is generally capped at 200 GB max,
and for those filers that have lots of qtrees, we still try to limit each
saveset to around 200 GB.

So, I guess what I'm really asking is 1) "What kind of speeds are you
seeing?" and 2) "Am I doing something wrong?"

Many thanks,

Geoff Hardin
geoff.hardin@dalsemi.com
Difficulty is the excuse history never accepts.
Re: NDMP backup speeds [ In reply to ]
On Tue, May 18, 2004 at 04:53:07PM -0500, Geoff Hardin wrote:
> So, I guess what I'm really asking is 1) "What kind of speeds are you
> seeing?" and 2) "Am I doing something wrong?"
>
We're running AIT-2 and AIT-3 drives for our backups. Our F820 which
has 2 AIT-2 drives direct attached gets 5-7MB/s on the tape drives.
Our 3-way NDMP backups get 4-6MB/s. We have separate networks too but
ours are 100Mbit (not GigE like yours).

We have an AIT-3 drive attached to a Solaris host running 3-way NDMP
backups and on that one we see 5-7MBs. We should be seeing better
perfomance on this one, but we have other configuration issues to
sort out (all of our AIT-3 drives are getting effectively AIT-2
performance even when direct attach).

DLT drives are notorious for shoe-shining, 1-2MB/s would be expected
if you're not streaming the drives.

--
Jeff Bryer bryer@sfu.ca
Systems Administrator (604) 291-4935
Academic Computing, Simon Fraser University
RE: NDMP backup speeds [ In reply to ]
On Tue, May 18, 2004 at 04:53:07PM -0500, Geoff Hardin wrote:
> So, I guess what I'm really asking is 1) "What kind of speeds are you
> seeing?" and 2) "Am I doing something wrong?"

We get about 20MB/s to GigE SDLT drives. The drives have a built in NDMP
server and are on a private GigE network with the filers. Our FAS960s can
push 6+ streams at 20MB/s each, the F880 maxes out at about 5-6 streams.
Our
F760 used to manage 4 streams at 15+MB/s.

We are switching over to LTO drives connected to DinoStor Tape Servers
(http://www.dinostor.com) which do the same job as the built in NDMP server
that our SDLT drives have.

In our experience, and that of our NetApp SE, direct attached tape drives
have worse
performance then network attached drives. NetApps are optimized for sending
data
out of the network, not for dealing with tape drives.

John
Re: NDMP backup speeds [ In reply to ]
I am getting 10MB/sec from a FAS250 back to a SUN
V480 with netvault (all 1000BaseT) to local disk
(staging). Then 50MB/sec off the disk to LTO II drives.

That is backing up off the mirror targets, not the
original filers so they are pretty idle. But I can
run the backups during the day off 3 filers at a time
so so far I am happy.

Early days for this configuration though.

On Tue, May 18, 2004 at 04:53:07PM -0500, Geoff Hardin wrote:
> Toasters, What is the general consensus for NDMP backup speeds? I
> have four older F760Cs that are getting about 5-6 MB/s to local
> tape drives. The problem is that I also have another nine filers
> (1 F630, 1 F720, 1 F740, 6 F760C) that are backing up via 3-way
> NDMP and we are lucky to see 4-5 MB/s; most are in the 1-2 MB/s
> range. And don't even get me started on the
> Solaris/SunOS/Linux/HP-UX/DEC clients that backup through the
> drives attached to the filers (0.1 - 2 MB/s).
>

Regards,
pdg

--

See mail headers for contact information.
Re: NDMP backup speeds [ In reply to ]
Geoff wrote: "

Toasters,
What is the general consensus for NDMP backup speeds? .....

So, I guess what I'm really asking is 1) "What kind of speeds are you
seeing?" and 2) "Am I doing something wrong?"

"

We have F820, with three DLT8000 and two are direct attach to Filer. I see
the most of my NDMP back up is 12MB/s. The average is for both running at
the same time is about 8-10 MB/s. We do not have money to test out with
NDMP over the network instead. Meaning redo out back up solutions. This is
cost us on tapes and time, so we are happy with what we have for now.

Hope that help.

C-
RE: NDMP backup speeds [ In reply to ]
I must apologise for being half-asleep and therefor overlooked the
significance of the request ;-).

Pse note, the following are real results obtained while testing with a
representative mix of user data.
There were a few restrictions at the time: I was only able to use a
single FC port on an F880 and the 4 LTO 1 tape drives were attached to
two FC-SCSI routers.
Networker 6.1.3, Ontap 6.3. Administration = History


Backup Data Including Excluding
Operations Volume Administration Administration
MB/sec GB/Hr MB/sec GB/hr
1 122 22,3 80 25,1
90
2 244 38,4 138 47,3
170
3 366 51,7 186 63,5
229
4 488 59,8 215 77,5
279

The following assumption was made based on backup and recover tests:
backup +- 50 GB/Hr, recovery +- 25GB/Hr

Operations Backup Recovery
Estimated Measured Estimated Measured
GByte/Hr GByte/Hr GByte/Hr GByte/Hr
1 50 80 25 62
2 100 138 50 120
3 150 186 75 144
4 200 215 100 160

Btw. I just remembered, the Networker server was a PC with a 200 Mhz
Pentium, 128 Mbyte RAM.

Bye
Ernie


-----Original Message-----
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com]
On Behalf Of Geoff Hardin
Sent: Tuesday, May 18, 2004 11:53 PM
To: toasters
Subject: NDMP backup speeds


Toasters,
What is the general consensus for NDMP backup speeds? I have four
older
F760Cs that are getting about 5-6 MB/s to local tape drives. The
problem is
that I also have another nine filers (1 F630, 1 F720, 1 F740, 6 F760C)
that are
backing up via 3-way NDMP and we are lucky to see 4-5 MB/s; most are in
the 1-2
MB/s range. And don't even get me started on the
Solaris/SunOS/Linux/HP-UX/DEC
clients that backup through the drives attached to the filers (0.1 - 2
MB/s).

We were looking at upgrading the tape drives and such, but we have
done a
few test dumps from filer to filer, filer to LTO-2, and filer to LTO-2
attached
to a Sun, and the numbers didn't improve much (definitely not enough to
justify
buying a new tape library / tape technology).

Just a little background:
We are using DLT7000 tape drives, which should have a maximum
through put of
about 10 MB/s (2:1 compression), contained in a Quantum P3000 library.
We have
all our filers on two separate networks; one for general data sharing
and a
private one for our backups. The private network is all GbE (fibre)
using the
Alteon GbE cards (Gigabit Ethernet Controller II).

We are doing qtree dumps, and while some of the filers have only
four qtrees
per volume, others have over 40. Each qtree is generally capped at 200
GB max,
and for those filers that have lots of qtrees, we still try to limit
each
saveset to around 200 GB.

So, I guess what I'm really asking is 1) "What kind of speeds are
you
seeing?" and 2) "Am I doing something wrong?"

Many thanks,

Geoff Hardin
geoff.hardin@dalsemi.com
Difficulty is the excuse history never accepts.
RE: NDMP backup speeds [ In reply to ]
You should see at least native rates on you backups.
I would do s sysstat -s -c 100 1 and see what CPU and network speeds are.

For AIT-2 I get 6-12 MB/s
For LTO it was Closer to 15 MB/s
This was three way backup with Sun 420R or Windows 2002 (DL580/DL580 G2)

How many filers are backing up at once?
You may need multiple GE networks to take advantage of the speed.
Also Contention on the volumes that house qtree could also slow you down.
-----Original Message-----
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On
Behalf Of Geoff Hardin
Sent: Tuesday, May 18, 2004 5:53 PM
To: toasters
Subject: NDMP backup speeds


Toasters,
What is the general consensus for NDMP backup speeds? I have four older

F760Cs that are getting about 5-6 MB/s to local tape drives. The problem is

that I also have another nine filers (1 F630, 1 F720, 1 F740, 6 F760C) that
are
backing up via 3-way NDMP and we are lucky to see 4-5 MB/s; most are in the
1-2
MB/s range. And don't even get me started on the
Solaris/SunOS/Linux/HP-UX/DEC
clients that backup through the drives attached to the filers (0.1 - 2
MB/s).

We were looking at upgrading the tape drives and such, but we have done
a
few test dumps from filer to filer, filer to LTO-2, and filer to LTO-2
attached
to a Sun, and the numbers didn't improve much (definitely not enough to
justify
buying a new tape library / tape technology).

Just a little background:
We are using DLT7000 tape drives, which should have a maximum through
put of
about 10 MB/s (2:1 compression), contained in a Quantum P3000 library. We
have
all our filers on two separate networks; one for general data sharing and a
private one for our backups. The private network is all GbE (fibre) using
the
Alteon GbE cards (Gigabit Ethernet Controller II).

We are doing qtree dumps, and while some of the filers have only four
qtrees
per volume, others have over 40. Each qtree is generally capped at 200 GB
max,
and for those filers that have lots of qtrees, we still try to limit each
saveset to around 200 GB.

So, I guess what I'm really asking is 1) "What kind of speeds are you
seeing?" and 2) "Am I doing something wrong?"

Many thanks,

Geoff Hardin
geoff.hardin@dalsemi.com
Difficulty is the excuse history never accepts.
RE: NDMP backup speeds [ In reply to ]
If you are doing lots of small files, it will suck
anyway. My move to ndmp (3-way) using veritas yielded
about double my performance from just nfs backups via
solaris machines. It is, however, still slow. i.e. I
have 3 mil files in only 18GB and in one backup...
it's not pretty. My average is only about 4.9
Megabytes per second. On systems with SAN attached
storage I am able to get 24 Megabytes per second as an
example. (still sucking over the gig-e).

--- "Blake, Delroy" <delroy.blake@gs.com> wrote:
> You should see at least native rates on you backups.
> I would do s sysstat -s -c 100 1 and see what CPU
> and network speeds are.
>
> For AIT-2 I get 6-12 MB/s
> For LTO it was Closer to 15 MB/s
> This was three way backup with Sun 420R or Windows
> 2002 (DL580/DL580 G2)
>
> How many filers are backing up at once?
> You may need multiple GE networks to take advantage
> of the speed.
> Also Contention on the volumes that house qtree
> could also slow you down.
> -----Original Message-----
> From: owner-toasters@mathworks.com
> [mailto:owner-toasters@mathworks.com] On
> Behalf Of Geoff Hardin
> Sent: Tuesday, May 18, 2004 5:53 PM
> To: toasters
> Subject: NDMP backup speeds
>
>
> Toasters,
> What is the general consensus for NDMP backup
> speeds? I have four older
>
> F760Cs that are getting about 5-6 MB/s to local tape
> drives. The problem is
>
> that I also have another nine filers (1 F630, 1
> F720, 1 F740, 6 F760C) that
> are
> backing up via 3-way NDMP and we are lucky to see
> 4-5 MB/s; most are in the
> 1-2
> MB/s range. And don't even get me started on the
> Solaris/SunOS/Linux/HP-UX/DEC
> clients that backup through the drives attached to
> the filers (0.1 - 2
> MB/s).
>
> We were looking at upgrading the tape drives and
> such, but we have done
> a
> few test dumps from filer to filer, filer to LTO-2,
> and filer to LTO-2
> attached
> to a Sun, and the numbers didn't improve much
> (definitely not enough to
> justify
> buying a new tape library / tape technology).
>
> Just a little background:
> We are using DLT7000 tape drives, which should
> have a maximum through
> put of
> about 10 MB/s (2:1 compression), contained in a
> Quantum P3000 library. We
> have
> all our filers on two separate networks; one for
> general data sharing and a
> private one for our backups. The private network is
> all GbE (fibre) using
> the
> Alteon GbE cards (Gigabit Ethernet Controller II).
>
> We are doing qtree dumps, and while some of the
> filers have only four
> qtrees
> per volume, others have over 40. Each qtree is
> generally capped at 200 GB
> max,
> and for those filers that have lots of qtrees, we
> still try to limit each
> saveset to around 200 GB.
>
> So, I guess what I'm really asking is 1) "What
> kind of speeds are you
> seeing?" and 2) "Am I doing something wrong?"
>
> Many thanks,
>
> Geoff Hardin
> geoff.hardin@dalsemi.com
> Difficulty is the excuse history never accepts.





__________________________________
Do you Yahoo!?
Friends. Fun. Try the all-new Yahoo! Messenger.
http://messenger.yahoo.com/
Re: NDMP Backup speeds [ In reply to ]
How about this for the win?

https://docs.netapp.com/ontap-9/topic/com.netapp.doc.dot-cm-ptbrg/GUID-6EC88448-13B9-41E9-A8BD-218E5343D7DF.html

--tmac

*Tim McCarthy, **Principal Consultant*

*Proud Member of the #NetAppATeam <https://twitter.com/NetAppATeam>*

*I Blog at TMACsRack <https://tmacsrack.wordpress.com/>*


On Wed, Jun 2, 2021 at 9:41 AM Heino Walther <hw@beardmann.dk> wrote:

> Hi there
>
>
>
> We are doing NDMP backups from out FAS2750 with NL-SAS disks onto LTO8
> tape drives via NetBackup.
>
> For other reasons we are doing this via 10G Ethernet, and not directly
> connected tapedrives to the NetApp controllers.
>
> We are seeing backup speeds of about 40-50MB/sec. which is kinda slow, the
> backup is a Windows host that is doing nothing… no CPU or memory load as we
> backup.
>
> The LTO8 drives should be able to do much better (350MB/sec. before
> compression)…
>
> The aggregates are RAID-TEC with 17 drives in it. We are on ONTAP 9.7.
> We are not seeing a lot of load as we are running the backups…
>
> Could it be that the protocol itself it as cause here?
>
> In the “old days” we did NDMP backup directly from the controllers to the
> tape drives.
>
> We are now thinking of converting two 10G ports to fibrechannel ports… if
> possible…
>
> So I have two questions ????
>
>
>
> The FAS2750 is configured with RJ45 10G ports, but would it be possible to
> use FCoE and somehow connect this to a FC/Ethernet switch and be able to
> connect it to the tape drive which is 8G FC (LC Connectors) ?
>
> (I have not worked that much with FCoE… if the FAS2750 was configured with
> CNA-Ports, it would have been easy to convert that into FC)
>
>
>
> Has anyone tried to backup NL-SAS volumes similar to this over directly
> connected NDMP, and what kind of speeds are you seeing?
>
>
>
> Finally if anyone have any idea if the NetBackup setup can be tweaked
> somehow, that would also be useful… (we have tweaked it a bit)
>
>
>
> /Heino
> _______________________________________________
> Toasters mailing list
> Toasters@teaparty.net
> https://www.teaparty.net/mailman/listinfo/toasters
Re: NDMP Backup speeds [ In reply to ]
Another thing we’ve observed is that since NDMP has to enumerate the entire
file system, backups of locations with lots of inodes can take a long time
and drive high levels of CPU and memory utilization.

On Wed, Jun 2, 2021 at 3:45 PM tmac <tmacmd@gmail.com> wrote:

> How about this for the win?
>
>
> https://docs.netapp.com/ontap-9/topic/com.netapp.doc.dot-cm-ptbrg/GUID-6EC88448-13B9-41E9-A8BD-218E5343D7DF.html
>
> --tmac
>
> *Tim McCarthy, **Principal Consultant*
>
> *Proud Member of the #NetAppATeam <https://twitter.com/NetAppATeam>*
>
> *I Blog at TMACsRack <https://tmacsrack.wordpress.com/>*
>
>
> On Wed, Jun 2, 2021 at 9:41 AM Heino Walther <hw@beardmann.dk> wrote:
>
>> Hi there
>>
>>
>>
>> We are doing NDMP backups from out FAS2750 with NL-SAS disks onto LTO8
>> tape drives via NetBackup.
>>
>> For other reasons we are doing this via 10G Ethernet, and not directly
>> connected tapedrives to the NetApp controllers.
>>
>> We are seeing backup speeds of about 40-50MB/sec. which is kinda slow,
>> the backup is a Windows host that is doing nothing… no CPU or memory load
>> as we backup.
>>
>> The LTO8 drives should be able to do much better (350MB/sec. before
>> compression)…
>>
>> The aggregates are RAID-TEC with 17 drives in it. We are on ONTAP 9.7.
>> We are not seeing a lot of load as we are running the backups…
>>
>> Could it be that the protocol itself it as cause here?
>>
>> In the “old days” we did NDMP backup directly from the controllers to the
>> tape drives.
>>
>> We are now thinking of converting two 10G ports to fibrechannel ports…
>> if possible…
>>
>> So I have two questions ????
>>
>>
>>
>> The FAS2750 is configured with RJ45 10G ports, but would it be possible
>> to use FCoE and somehow connect this to a FC/Ethernet switch and be able to
>> connect it to the tape drive which is 8G FC (LC Connectors) ?
>>
>> (I have not worked that much with FCoE… if the FAS2750 was configured
>> with CNA-Ports, it would have been easy to convert that into FC)
>>
>>
>>
>> Has anyone tried to backup NL-SAS volumes similar to this over directly
>> connected NDMP, and what kind of speeds are you seeing?
>>
>>
>>
>> Finally if anyone have any idea if the NetBackup setup can be tweaked
>> somehow, that would also be useful… (we have tweaked it a bit)
>>
>>
>>
>> /Heino
>> _______________________________________________
>> Toasters mailing list
>> Toasters@teaparty.net
>> https://www.teaparty.net/mailman/listinfo/toasters
>
> _______________________________________________
> Toasters mailing list
> Toasters@teaparty.net
> https://www.teaparty.net/mailman/listinfo/toasters
SV: NDMP Backup speeds [ In reply to ]
Hi Tim



That is exactly one of the tuning parameters we have set &#128521;

Not that it made much difference anyway…

But I have found out that NetBackup has an option called NDMP Accelerated backup, which may make sense for us because we want to do incremental forever (more or less).

The backup server keeps a list of the backed up files, and it then determines which new or changed files needs to be backed up…. It is supposed to be faster than NDMP Level backups…

We will test this once our current monthly backups have completed, which will be in a week or so &#128521;



/Heino



Fra: tmac <tmacmd@gmail.com>
Dato: onsdag, 2. juni 2021 kl. 21.43
Til: Heino Walther <hw@beardmann.dk>
Cc: toasters@teaparty.net <toasters@teaparty.net>
Emne: Re: NDMP Backup speeds

How about this for the win?



https://docs.netapp.com/ontap-9/topic/com.netapp.doc.dot-cm-ptbrg/GUID-6EC88448-13B9-41E9-A8BD-218E5343D7DF.html"]https://docs.netapp.com/ontap-9/topic/com.netapp.doc.dot-cm-ptbrg/GUID-6EC88448-13B9-41E9-A8BD-218E5343D7DF.html




--tmac



Tim McCarthy, Principal Consultant

Proud Member of the https://twitter.com/NetAppATeam"]#NetAppATeam

I Blog at https://tmacsrack.wordpress.com/"]TMACsRack





On Wed, Jun 2, 2021 at 9:41 AM Heino Walther <hw@beardmann.dk> wrote:


Hi there



We are doing NDMP backups from out FAS2750 with NL-SAS disks onto LTO8 tape drives via NetBackup.

For other reasons we are doing this via 10G Ethernet, and not directly connected tapedrives to the NetApp controllers.

We are seeing backup speeds of about 40-50MB/sec. which is kinda slow, the backup is a Windows host that is doing nothing… no CPU or memory load as we backup.

The LTO8 drives should be able to do much better (350MB/sec. before compression)…

The aggregates are RAID-TEC with 17 drives in it. We are on ONTAP 9.7. We are not seeing a lot of load as we are running the backups…

Could it be that the protocol itself it as cause here?

In the “old days” we did NDMP backup directly from the controllers to the tape drives.

We are now thinking of converting two 10G ports to fibrechannel ports… if possible…

So I have two questions &#128522;



The FAS2750 is configured with RJ45 10G ports, but would it be possible to use FCoE and somehow connect this to a FC/Ethernet switch and be able to connect it to the tape drive which is 8G FC (LC Connectors) ?

(I have not worked that much with FCoE… if the FAS2750 was configured with CNA-Ports, it would have been easy to convert that into FC)



Has anyone tried to backup NL-SAS volumes similar to this over directly connected NDMP, and what kind of speeds are you seeing?



Finally if anyone have any idea if the NetBackup setup can be tweaked somehow, that would also be useful… (we have tweaked it a bit)



/Heino

_______________________________________________
Toasters mailing list
Toasters@teaparty.net
https://www.teaparty.net/mailman/listinfo/toasters"]https://www.teaparty.net/mailman/listinfo/toasters
SV: NDMP Backup speeds [ In reply to ]
Hi Basil



Yes we are also aware of this.  But this happens at the start of the NDMP dump where is dumps the file and directory structure first, and then it dumps the data as the last part… and even when the system is running the last part of the backup we do not see speeds over 60MB/sec…   and yes we are sure it uses the 10G NICs and the Intercluster NICs etc.

If I really have a lot of spare time on my hands I might just open a case with Veritas and/or NetApp to figure out if it is possible to speed it up.



/Heino



Fra: Basil <basilberntsen@gmail.com>
Dato: onsdag, 2. juni 2021 kl. 21.53
Til: tmac <tmacmd@gmail.com>
Cc: Heino Walther <hw@beardmann.dk>, toasters@teaparty.net <toasters@teaparty.net>
Emne: Re: NDMP Backup speeds

Another thing we’ve observed is that since NDMP has to enumerate the entire file system, backups of locations with lots of inodes can take a long time and drive high levels of CPU and memory utilization.



On Wed, Jun 2, 2021 at 3:45 PM tmac <tmacmd@gmail.com> wrote:


How about this for the win?





https://docs.netapp.com/ontap-9/topic/com.netapp.doc.dot-cm-ptbrg/GUID-6EC88448-13B9-41E9-A8BD-218E5343D7DF.html"]https://docs.netapp.com/ontap-9/topic/com.netapp.doc.dot-cm-ptbrg/GUID-6EC88448-13B9-41E9-A8BD-218E5343D7DF.html




--tmac



Tim McCarthy, Principal Consultant

Proud Member of the https://twitter.com/NetAppATeam"]#NetAppATeam

I Blog at https://tmacsrack.wordpress.com/"]TMACsRack





On Wed, Jun 2, 2021 at 9:41 AM Heino Walther <hw@beardmann.dk> wrote:


Hi there



We are doing NDMP backups from out FAS2750 with NL-SAS disks onto LTO8 tape drives via NetBackup.

For other reasons we are doing this via 10G Ethernet, and not directly connected tapedrives to the NetApp controllers.

We are seeing backup speeds of about 40-50MB/sec. which is kinda slow, the backup is a Windows host that is doing nothing… no CPU or memory load as we backup.

The LTO8 drives should be able to do much better (350MB/sec. before compression)…

The aggregates are RAID-TEC with 17 drives in it. We are on ONTAP 9.7. We are not seeing a lot of load as we are running the backups…

Could it be that the protocol itself it as cause here?

In the “old days” we did NDMP backup directly from the controllers to the tape drives.

We are now thinking of converting two 10G ports to fibrechannel ports… if possible…

So I have two questions &#128522;



The FAS2750 is configured with RJ45 10G ports, but would it be possible to use FCoE and somehow connect this to a FC/Ethernet switch and be able to connect it to the tape drive which is 8G FC (LC Connectors) ?

(I have not worked that much with FCoE… if the FAS2750 was configured with CNA-Ports, it would have been easy to convert that into FC)



Has anyone tried to backup NL-SAS volumes similar to this over directly connected NDMP, and what kind of speeds are you seeing?



Finally if anyone have any idea if the NetBackup setup can be tweaked somehow, that would also be useful… (we have tweaked it a bit)



/Heino

_______________________________________________
Toasters mailing list
Toasters@teaparty.net
https://www.teaparty.net/mailman/listinfo/toasters"]https://www.teaparty.net/mailman/listinfo/toasters


_______________________________________________
Toasters mailing list
Toasters@teaparty.net
https://www.teaparty.net/mailman/listinfo/toasters"]https://www.teaparty.net/mailman/listinfo/toasters