Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Gentoo>
  3. SPARC
64-bit gentoo-sparc, how to make it happen?
zhangweiwu at realss
Jun 24, 2006, 9:01 PM
Post #1 of 11 (4562 views)
Permalink
I have been using gentoo linux on my ultra-sparc U2 and U5 for two
years, untill very recently I learnt the gentoo Linux is actually not
64-bit (it is 32-bit).

The U2 gets broken after I upgraded OBP (when I tries Solaris 10, it
prompted OBP is 32-bit OBP, I must upgrade to 64bit, then I did it, then
this U2 never boots up again). I have the remaining U5 box serving as
print server, it simply run out of CPU power (a single A4 brochure
printing uses up CPU and caused very slow printing). I think I can run
the machine in 64-bit mode, recompile related packages in 64-bit
executable to regain some speed.

I tried to adjust C-flags:

CFLAGS="-O2 -m64 -mcpu=ultrasparc -fomit-frame-pointer -pipe"
(added -m64)

then emerge a package, get this error:

checking whether the C compiler (gcc -O2 -m64 -mcpu=ultrasparc
-fomit-frame-pointer -pipe ) works... no

I think perhaps it is not possible to compile anything in 64-bit
executable when the kernel itself is running in 32-bit mode. Perhaps it
is not possible to run 64-bit code on 32-bit kernal at all (if this
sounds obvious, I admit I am a newbie). So what should I do? Should I
try to compile a kernel with -m64 first? Then there comes another
question: can 32-bit OBP loads 64bit kernel? I don't know if I have 64
OBP at all (and don't know how to tell it), my other U2 has had 32bit
OBP, that's for sure.

and even if I try to compile a 64-bit kernel, that may as well not
possible because it seems gcc refuse to run with -m64 (the error message
above).

Besides, is 64-big executable much faster then 32-bit executable?

P.S. I bet there must be an article somewhere on the Internet telling
how to get 64-bit linux running on sparcs, only that I didn't find it.

--
gentoo-sparc@gentoo.org mailing list
Re: 64-bit gentoo-sparc, how to make it happen? [ In reply to ]
zhangweiwu at realss
Jun 25, 2006, 12:29 AM
Post #2 of 11 (4486 views)
Permalink
在 2006-06-25日的 12:01 +0800，Zhang Weiwu写道：
> I have been using gentoo linux on my ultra-sparc U2 and U5 for two
> years, untill very recently I learnt the gentoo Linux is actually not
> 64-bit (it is 32-bit).
>
> The U2 gets broken after I upgraded OBP (when I tries Solaris 10, it
> prompted OBP is 32-bit OBP, I must upgrade to 64bit, then I did it, then
> this U2 never boots up again). I have the remaining U5 box serving as
> print server, it simply run out of CPU power (a single A4 brochure
> printing uses up CPU and caused very slow printing). I think I can run
> the machine in 64-bit mode, recompile related packages in 64-bit
> executable to regain some speed.
>
> I tried to adjust C-flags:
>
> CFLAGS="-O2 -m64 -mcpu=ultrasparc -fomit-frame-pointer -pipe"
> (added -m64)
>
> then emerge a package, get this error:
>
> checking whether the C compiler (gcc -O2 -m64 -mcpu=ultrasparc
> -fomit-frame-pointer -pipe ) works... no
>
> I think perhaps it is not possible to compile anything in 64-bit
> executable when the kernel itself is running in 32-bit mode. Perhaps it
> is not possible to run 64-bit code on 32-bit kernal at all (if this
> sounds obvious, I admit I am a newbie). So what should I do? Should I
> try to compile a kernel with -m64 first? Then there comes another
> question: can 32-bit OBP loads 64bit kernel? I don't know if I have 64
> OBP at all (and don't know how to tell it), my other U2 has had 32bit
> OBP, that's for sure.
>
> and even if I try to compile a 64-bit kernel, that may as well not
> possible because it seems gcc refuse to run with -m64 (the error message
> above).
>
> Besides, is 64-big executable much faster then 32-bit executable?
>
> P.S. I bet there must be an article somewhere on the Internet telling
> how to get 64-bit linux running on sparcs, only that I didn't find it.
>

an example of trying to run gcc with -m64 on 32bit system:

sappho ~ # touch abc #creates 0byte file abc
sappho ~ # gcc -O2 -m64 -mcpu=ultrasparc -fomit-frame-pointer -pipe abc
/usr/lib/gcc/sparc-unknown-linux-gnu/3.4.4/../../../crt1.o(.text+0x10):
In function `_start':
: undefined reference to `main'
/usr/lib/gcc/sparc-unknown-linux-gnu/3.4.4/../../../crt1.o(.text+0x1c):
In function `_start':
: undefined reference to `main'
collect2: ld returned 1 exit status


--
gentoo-sparc@gentoo.org mailing list
Re: 64-bit gentoo-sparc, how to make it happen? [ In reply to ]
zhangweiwu at realss
Jun 25, 2006, 12:43 AM
Post #3 of 11 (4492 views)
Permalink
在 2006-06-25日的 15:29 +0800，Zhang Weiwu写道：

> an example of trying to run gcc with -m64 on 32bit system:
>
> sappho ~ # touch abc #creates 0byte file abc
> sappho ~ # gcc -O2 -m64 -mcpu=ultrasparc -fomit-frame-pointer -pipe abc
> /usr/lib/gcc/sparc-unknown-linux-gnu/3.4.4/../../../crt1.o(.text+0x10):
> In function `_start':
> : undefined reference to `main'
> /usr/lib/gcc/sparc-unknown-linux-gnu/3.4.4/../../../crt1.o(.text+0x1c):
> In function `_start':
> : undefined reference to `main'
> collect2: ld returned 1 exit status
>

How stupid. I pasted the result without thinking. My idea is first to
test empty file to see if gcc can actually run instead of quitting
immediatly. The output is actually very sane, empty files don't have
main(). Shame! Hope my CS lecturer (when I was a CS student) didn't see
this post.

Here is a not-so-stupid test:
sappho tmp # gcc -O2 -m64 -mcpu=ultrasparc -fomit-frame-pointer -pipe
helloworld.c
helloworld.c:1: error: -m64 is not supported by this configuration
helloworld.c:1: error: -mlong-double-64 not allowed with -m64


--
gentoo-sparc@gentoo.org mailing list
Re: 64-bit gentoo-sparc, how to make it happen? [ In reply to ]
shrdlu at unlimitedmail
Jun 25, 2006, 8:18 AM
Post #4 of 11 (4491 views)
Permalink
On Sunday 25 June 2006 06:01, Zhang Weiwu wrote:

> Besides, is 64-big executable much faster then 32-bit executable?
>
> P.S. I bet there must be an article somewhere on the Internet telling
> how to get 64-bit linux running on sparcs, only that I didn't find it.

Gentoo on sparc64 currently uses a 64 bit kernel and a 32 bit userland.
The kernel is compiled with a different compiler than userland apps.
See

http://www.ultralinux.org/faq.html#q_1_5
http://devmanual.gentoo.org/archs/sparc/index.html
http://marc.theaimsgroup.com/?l=gentoo-sparc&m=110277902832636&w=2
http://marc.theaimsgroup.com/?l=gentoo-sparc&m=110281060602052&w=2
http://marc.theaimsgroup.com/?l=gentoo-sparc&m=109545333801428&w=2
--
gentoo-sparc@gentoo.org mailing list
Re: 64-bit gentoo-sparc, how to make it happen? [ In reply to ]
zhangweiwu at realss
Jun 25, 2006, 10:02 AM
Post #5 of 11 (4479 views)
Permalink
在 2006-06-25日的 17:18 +0200，Etaoin Shrdlu写道：
> On Sunday 25 June 2006 06:01, Zhang Weiwu wrote:
>
> > Besides, is 64-big executable much faster then 32-bit executable?
> >
> > P.S. I bet there must be an article somewhere on the Internet telling
> > how to get 64-bit linux running on sparcs, only that I didn't find it.
>
> Gentoo on sparc64 currently uses a 64 bit kernel and a 32 bit userland.
> The kernel is compiled with a different compiler than userland apps.
> See
>
> http://www.ultralinux.org/faq.html#q_1_5
> http://devmanual.gentoo.org/archs/sparc/index.html
> http://marc.theaimsgroup.com/?l=gentoo-sparc&m=110277902832636&w=2
> http://marc.theaimsgroup.com/?l=gentoo-sparc&m=110281060602052&w=2
> http://marc.theaimsgroup.com/?l=gentoo-sparc&m=109545333801428&w=2

Thank you for the answer. Just another smaller question: from pruely a
user's view (who simply wish to use it, rather then from developer view
who might gain knowledge and fun doing it), is it worthy to try it
(64bit sparc)?

The original idea is the system as a print server is too slow (capable
of doing fast dithering for the inkjet printer, but not fast enough to
do real-time even-tone dithering to server the printer, usually cause
printer to wait for server), a rough gain of 20% to 30% CPU power, by my
guessing of observing top(1), should be enough for the printer.
Dithering is always a CPU intensive job and we do a lot of printing.

I observed kernel is taking only a very small percentage of CPU usage,
if I try hours to get 64-bit kernel running but only obtain 5% speed
gain, that may not be worth. Or should I try Solaris 10? Should Solaris
provide better performance in this very case? (how much performance gain
might happen? 10%? 20%?) I don't need a very close estimation, just if
someone can give me a very rough estimation is very helpful, for I am
not techincally able to do the decision:)

If switch to Solaris for performance (no offensive, I am a 5 year's
hardcore Linux user), generally speaking, is Solaris very compatible
with PCI cards like NEC USB controller, Promise IDE controller and
ethernet cards?


And thank you a lot for you time,
Etaoin Shrdlu

Attached Files:

Re: 64-bit gentoo-sparc, how to make it happen? [ In reply to ]
fmccor at gentoo
Jun 25, 2006, 10:21 AM
Post #6 of 11 (4490 views)
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 26 Jun 2006, å¼ é~_¡æ­¦ wrote:

> 在 2006-06-25日的 17:18 +0200，Etaoin Shrdlu写道：
>> On Sunday 25 June 2006 06:01, Zhang Weiwu wrote:
>>
>
> Thank you for the answer. Just another smaller question: from pruely a
> user's view (who simply wish to use it, rather then from developer view
> who might gain knowledge and fun doing it), is it worthy to try it
> (64bit sparc)?
>
> The original idea is the system as a print server is too slow (capable
> of doing fast dithering for the inkjet printer, but not fast enough to
> do real-time even-tone dithering to server the printer, usually cause
> printer to wait for server), a rough gain of 20% to 30% CPU power, by my
> guessing of observing top(1), should be enough for the printer.
> Dithering is always a CPU intensive job and we do a lot of printing.
>
> I observed kernel is taking only a very small percentage of CPU usage,
> if I try hours to get 64-bit kernel running but only obtain 5% speed
> gain, that may not be worth.

I think there is a miscommunication here. On sparc64 you are running the
kernel on 64-bit mode. It's user programs which run only in 32-bit mode
(without a fair amount of effort, that is). Since 32-bit user mode is
FASTER than 64-bit user mode, for your application, you don't want 64-bit
user mode in any case.

> Or should I try Solaris 10? Should Solaris
> provide better performance in this very case? (how much performance gain
> might happen? 10%? 20%?) I don't need a very close estimation, just if
> someone can give me a very rough estimation is very helpful, for I am
> not techincally able to do the decision:)
>
> If switch to Solaris for performance (no offensive, I am a 5 year's
> hardcore Linux user), generally speaking, is Solaris very compatible
> with PCI cards like NEC USB controller, Promise IDE controller and
> ethernet cards?
>
>
> And thank you a lot for you time,
> Etaoin Shrdlu
>

I can't comment meaningfully on your other questions.

Regards,
Ferris

- --
Ferris McCormick (P44646, MI) <fmccor@gentoo.org>
Developer, Gentoo Linux (Devrel, Sparc)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2-ecc0.1.6 (GNU/Linux)

iD8DBQFEnsYBQa6M3+I///cRAo1SAJ0VAsIfETO/Mrt9HuyPRNA0YtovZACeOEQH
m4Iu5es1q6rQFKU7ogbmUxI=
=ZeNA
-----END PGP SIGNATURE-----
Re: 64-bit gentoo-sparc, how to make it happen? [ In reply to ]
shrdlu at unlimitedmail
Jun 25, 2006, 10:50 AM
Post #7 of 11 (4487 views)
Permalink
On Sunday 25 June 2006 19:02, 张韡武 wrote:

> Thank you for the answer. Just another smaller question: from pruely a
> user's view (who simply wish to use it, rather then from developer
> view who might gain knowledge and fun doing it), is it worthy to try
> it (64bit sparc)?
>
> The original idea is the system as a print server is too slow (capable
> of doing fast dithering for the inkjet printer, but not fast enough to
> do real-time even-tone dithering to server the printer, usually cause
> printer to wait for server), a rough gain of 20% to 30% CPU power, by
> my guessing of observing top(1), should be enough for the printer.
> Dithering is always a CPU intensive job and we do a lot of printing.
>
> I observed kernel is taking only a very small percentage of CPU usage,
> if I try hours to get 64-bit kernel running but only obtain 5% speed
> gain, that may not be worth. Or should I try Solaris 10? Should
> Solaris provide better performance in this very case? (how much
> performance gain might happen? 10%? 20%?) I don't need a very close
> estimation, just if someone can give me a very rough estimation is
> very helpful, for I am not techincally able to do the decision:)
>
> If switch to Solaris for performance (no offensive, I am a 5 year's
> hardcore Linux user), generally speaking, is Solaris very compatible
> with PCI cards like NEC USB controller, Promise IDE controller and
> ethernet cards?

Well, I'm not a sparc dev so probably I cannot give you the most accurate
answer you could get; moreover, I cannot speak about solaris since I
don't know it. Anyway...
From what I've read, the main reason gentoo stays 64 bit kernel/32 bit
userland is that switching to a full 64 bit system (perhaps with
multilib) will not be a considerable improvement, but rather just a
waste of memory. This is different from what happens in the x86 world,
where being 64-bit means getting a lot of benefits (like, for example,
many more cpu registers). In the sparc world, for an userland app "being
64-bit" means, basically, "wasting more memory" (due to larger
executables binaries). In short, I think that (from the pure user's view
you were talking about), it's not worth trying.
However, I believe that a (highly experimental) userland 64-bit profile
exists (but of course don't expect that to work flawlessly or to be
usable in a production environment).
Read this thread on the forums, which explains all:

http://forums.gentoo.org/viewtopic-t-442194.html

btw, AFAIK solaris uses a similar 64bit-kernel/32bit-userland approach
(or, at least, *most* of the userland is 32bit).

Hope this helps (and please somebody correct me if what I said is wrong).

--
gentoo-sparc@gentoo.org mailing list
Re: 64-bit gentoo-sparc, how to make it happen? [ In reply to ]
zhangweiwu at realss
Jun 25, 2006, 11:32 AM
Post #8 of 11 (4480 views)
Permalink
在 2006-06-25日的 19:50 +0200，Etaoin Shrdlu写道：
> On Sunday 25 June 2006 19:02, 张韡武 wrote:
>
> > Thank you for the answer. Just another smaller question: from pruely a
> > user's view (who simply wish to use it, rather then from developer
> > view who might gain knowledge and fun doing it), is it worthy to try
> > it (64bit sparc)?
> >
> > The original idea is the system as a print server is too slow (capable
> > of doing fast dithering for the inkjet printer, but not fast enough to
> > do real-time even-tone dithering to server the printer, usually cause
> > printer to wait for server), a rough gain of 20% to 30% CPU power, by
> > my guessing of observing top(1), should be enough for the printer.
> > Dithering is always a CPU intensive job and we do a lot of printing.
> >
> > I observed kernel is taking only a very small percentage of CPU usage,
> > if I try hours to get 64-bit kernel running but only obtain 5% speed
> > gain, that may not be worth. Or should I try Solaris 10? Should
> > Solaris provide better performance in this very case? (how much
> > performance gain might happen? 10%? 20%?) I don't need a very close
> > estimation, just if someone can give me a very rough estimation is
> > very helpful, for I am not techincally able to do the decision:)
> >
> > If switch to Solaris for performance (no offensive, I am a 5 year's
> > hardcore Linux user), generally speaking, is Solaris very compatible
> > with PCI cards like NEC USB controller, Promise IDE controller and
> > ethernet cards?
>
> Well, I'm not a sparc dev so probably I cannot give you the most accurate
> answer you could get; moreover, I cannot speak about solaris since I
> don't know it. Anyway...
> From what I've read, the main reason gentoo stays 64 bit kernel/32 bit
> userland is that switching to a full 64 bit system (perhaps with
> multilib) will not be a considerable improvement, but rather just a
> waste of memory. This is different from what happens in the x86 world,
> where being 64-bit means getting a lot of benefits (like, for example,
> many more cpu registers). In the sparc world, for an userland app "being
> 64-bit" means, basically, "wasting more memory" (due to larger
> executables binaries). In short, I think that (from the pure user's view
> you were talking about), it's not worth trying.
> However, I believe that a (highly experimental) userland 64-bit profile
> exists (but of course don't expect that to work flawlessly or to be
> usable in a production environment).
> Read this thread on the forums, which explains all:
>
> http://forums.gentoo.org/viewtopic-t-442194.html
>
> btw, AFAIK solaris uses a similar 64bit-kernel/32bit-userland approach
> (or, at least, *most* of the userland is 32bit).
>
> Hope this helps (and please somebody correct me if what I said is wrong).

Blame me for being stupid, but I simply cannot understand. Truly in x86
world 64-bit is often marketed as much faster then 32-bit applications,
if this is not ture in sparc system, the question is why people bother
to build 64-bit stuff, being slower and wastes more memory at all? For
sure I am not an arch guru, but there must be a good reason behind it.

I am one of stupid graduated CS student the modern universities
produced, I can understand perfectly well that 32-bit binary
applications perform equally on 32-bit or 64-bit platform, but I was
tought that is because the binary is 32-bit: if the source is compiled
targeted to be 64-bit binary, usually it is faster especially for CPU
intensive tasks, the compiler optimize the source and use 64-bit when
possible (perhaps especially for float point). In my case I can
re-compile gutenprint or cups to generate 64-bit binary. Especially the
job like dithering for the printer. In my understanding dithering is
purely computation of how to mix colors, few I/O and only some memory,
that looks really like something can be optimized by using 64-bit.

For the suggestions I can easily take all of them, just now I am
curious:) If 64-bit is truly considered an enhancement for CPU, there
must be some use of it, and if purely CPU intensive tasks (dithering)
cannot benifit from enhancement for CPU, whatelse can take the advantage
of 64-bit? (Okay, now I am not in purely user's view, I wish to dig into
it a little bit because that makes me really curious).

--
gentoo-sparc@gentoo.org mailing list
Re: 64-bit gentoo-sparc, how to make it happen? [ In reply to ]
zhangweiwu at realss
Jun 25, 2006, 11:57 AM
Post #9 of 11 (4488 views)
Permalink
在 2006-06-25日的 19:50 +0200，Etaoin Shrdlu写道：

> btw, AFAIK solaris uses a similar 64bit-kernel/32bit-userland approach
> (or, at least, *most* of the userland is 32bit).

Ah, I get a good idea (good in sense of a newbie) If most of userland is
32bit on solaris too, the *some* applications in 64 bit are there
perhaps because they perform better in 64-bit. If gimp-print/gutenprint
package on solaris is 64-bit, that might shows Sun people think it can
be in 64-bit mode. Sadly I cannot google out the package filename for
gimp-print on solaris, usually in Linux binary package name often
indicate if it is 32-bit or 64-bit
(e.g. /suse/x86_64/libgimpprint-4.2.7-62.x86_64.rpm)

--
gentoo-sparc@gentoo.org mailing list
Re: 64-bit gentoo-sparc, how to make it happen? [ In reply to ]
javier.augusto at gmx
Jun 25, 2006, 2:20 PM
Post #10 of 11 (4486 views)
Permalink
[..]
>
> Ah, I get a good idea (good in sense of a newbie) If most of userland is
> 32bit on solaris too, the *some* applications in 64 bit are there
> perhaps because they perform better in 64-bit. If gimp-print/gutenprint
> package on solaris is 64-bit, that might shows Sun people think it can
> be in 64-bit mode. Sadly I cannot google out the package filename for
> gimp-print on solaris, usually in Linux binary package name often
> indicate if it is 32-bit or 64-bit
> (e.g. /suse/x86_64/libgimpprint-4.2.7-62.x86_64.rpm)

You can try installing Solaris 10 Community Release (SXCR) on your box, then go after GCC-4.1 from Sun Downloads. This includes SPARC optimizations out of the box. Don't forget to read GCC docs about compiling on SPARC for better understanding flags differences.
For compiling, install automake autogen gmake gmk4 autoconf from blastwave.org thru `pkg-get`.
In your environment, set this flags before start `./configure`ing :

--
PATH=/opt/SUNWspro/bin:/opt/onbld/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:\
/usr/sfw/bin:/usr/ccs/bin:/usr/local/enlightenment/bin:/opt/csw/bin:/usr/X11/bin:/usr/openwin/bin :/usr/ccs/bin

CC=/opt/gcc/bin/gcc

CFLAGS='-I/usr/local/include -I/usr/sfw/include -I/opt/csw/include -m32 \
-mcpu=ultrasparc -mtune=ultrasparc -O2' <- safe mode

LDFLAGS='-L/usr/local/lib -L/usr/sfw/lib -L/opt/csw/lib -R/usr/local/lib -R/usr/sfw/lib -R/opt/csw/lib'

alias tar='gtar'

alias make='gmake'
--

Don't forget to share your results.

Hope this helps.. Good luck.

--
Javier O. Augusto - <javier.augusto@gmx.net>


"Feel free" – 10 GB Mailbox, 100 FreeSMS/Monat ...
Jetzt GMX TopMail testen: http://www.gmx.net/de/go/topmail
--
gentoo-sparc@gentoo.org mailing list
Re: 64-bit gentoo-sparc, how to make it happen? [ In reply to ]
zhangweiwu at realss
Jun 25, 2006, 6:34 PM
Post #11 of 11 (4482 views)
Permalink
在 2006-06-25日的 23:20 +0200，Javier O. Augusto写道：
> [..]
> >
> > Ah, I get a good idea (good in sense of a newbie) If most of userland is
> > 32bit on solaris too, the *some* applications in 64 bit are there
> > perhaps because they perform better in 64-bit. If gimp-print/gutenprint
> > package on solaris is 64-bit, that might shows Sun people think it can
> > be in 64-bit mode. Sadly I cannot google out the package filename for
> > gimp-print on solaris, usually in Linux binary package name often
> > indicate if it is 32-bit or 64-bit
> > (e.g. /suse/x86_64/libgimpprint-4.2.7-62.x86_64.rpm)
>
> You can try installing Solaris 10 Community Release (SXCR) on your box, then go after GCC-4.1 from Sun Downloads. This includes SPARC optimizations out of the box. Don't forget to read GCC docs about compiling on SPARC for better understanding flags differences.
> For compiling, install automake autogen gmake gmk4 autoconf from blastwave.org thru `pkg-get`.
> In your environment, set this flags before start `./configure`ing :
>
> --
> PATH=/opt/SUNWspro/bin:/opt/onbld/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:\
> /usr/sfw/bin:/usr/ccs/bin:/usr/local/enlightenment/bin:/opt/csw/bin:/usr/X11/bin:/usr/openwin/bin :/usr/ccs/bin
>
> CC=/opt/gcc/bin/gcc
>
> CFLAGS='-I/usr/local/include -I/usr/sfw/include -I/opt/csw/include -m32 \
> -mcpu=ultrasparc -mtune=ultrasparc -O2' <- safe mode
>
> LDFLAGS='-L/usr/local/lib -L/usr/sfw/lib -L/opt/csw/lib -R/usr/local/lib -R/usr/sfw/lib -R/opt/csw/lib'
>
> alias tar='gtar'
>
> alias make='gmake'
> --
>
> Don't forget to share your results.
>
> Hope this helps.. Good luck.

Thank you! I would like to try it on July when I got more free time.
Besides, now I am thinking people's suggestions are making good sense,
probably it's right the print servre would not be faster after upgraded
to 64-bit, because I took a glance into gutenprint source, the very part
of eventone dithering (which from my limited knowledge is what computer
busy doing for the printer) is about 700 lines of C source code, most
variables are defined integer, which seems would not benifit from 64-bit
processing. I would like to try 64-bit anyway.

Thank you for all the help.

Attached Files:

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal