Mailing List Archive

Living in NGL: was: NAS and replacing with larger drives
On 12/18/22 10:38, Mark Knecht wrote:
> Wipe the machine and start over with Gentoo from scratch...
> Wipe the machine and start over with Gentoo from scratch...
> Humm, I think you should...
> Wipe the machine and start over with Gentoo from scratch...

First there was Linux from Scratch.

Next came Beyond Linux from Scratch.

Then there was Gentoo.

Now, the latest, greatest installment: Gentoo from Scratch.
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
On Sun, Dec 18, 2022 at 8:49 AM Jack <ostroffjh@users.sourceforge.net>
wrote:
>
> On 12/18/22 10:38, Mark Knecht wrote:
> > Wipe the machine and start over with Gentoo from scratch...
> > Wipe the machine and start over with Gentoo from scratch...
> > Humm, I think you should...
> > Wipe the machine and start over with Gentoo from scratch...
>
> First there was Linux from Scratch.
>
> Next came Beyond Linux from Scratch.
>
> Then there was Gentoo.
>
> Now, the latest, greatest installment: Gentoo from Scratch.
>

Gawd, that's funny! Thanks for making me smile, assuming I found what
you're talking about:

https://wiki.gentoo.org/wiki/Gentoo_ebuild_tree_from_scratch

Just what every Gentoo user needs. More management!

I really have fallen off the deep end thinking computers are just tools to
get a job done. I'm ashamed of myself...

In the really early days of Gentoo circa 2003 when I started there was some
choice about regular Gentoo or a really low level install. I failed with
the low level one but soon learned that every package on my machine was
going to get rebuilt anyway so why bother?

:-( Sad Mark. I'm a putz...
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
On 2022.12.18 11:17, Mark Knecht wrote:
> On Sun, Dec 18, 2022 at 8:49 AM Jack <ostroffjh@users.sourceforge.net>
> wrote:
> >
> > On 12/18/22 10:38, Mark Knecht wrote:
> > > Wipe the machine and start over with Gentoo from scratch...
> > > Wipe the machine and start over with Gentoo from scratch...
> > > Humm, I think you should...
> > > Wipe the machine and start over with Gentoo from scratch...
> >
> > First there was Linux from Scratch.
> >
> > Next came Beyond Linux from Scratch.
> >
> > Then there was Gentoo.
> >
> > Now, the latest, greatest installment: Gentoo from Scratch.
> >
>
> Gawd, that's funny! Thanks for making me smile, assuming I found what
> you're talking about:
>
> https://wiki.gentoo.org/wiki/Gentoo_ebuild_tree_from_scratch
I actually hadn't seen that, but it fits.
>
> Just what every Gentoo user needs. More management!
>
> I really have fallen off the deep end thinking computers are just
> tools to get a job done. I'm ashamed of myself...
I've long said that many folks have computers to play games, but for
me, the computer IS the game.
>
> In the really early days of Gentoo circa 2003 when I started there
> was some choice about regular Gentoo or a really low level install. I
> failed with the low level one but soon learned that every package on
> my machine was going to get rebuilt anyway so why bother?
>
> :-( Sad Mark. I'm a putz...
Jack
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
Mark Knecht wrote:
>
>
> On Sun, Dec 18, 2022 at 8:49 AM Jack <ostroffjh@users.sourceforge.net
> <mailto:ostroffjh@users.sourceforge.net>> wrote:
> >
> > On 12/18/22 10:38, Mark Knecht wrote:
> > > Wipe the machine and start over with Gentoo from scratch...
> > > Wipe the machine and start over with Gentoo from scratch...
> > > Humm, I think you should...
> > > Wipe the machine and start over with Gentoo from scratch...
> >
> > First there was Linux from Scratch.
> >
> > Next came Beyond Linux from Scratch.
> >
> > Then there was Gentoo.
> >
> > Now, the latest, greatest installment: Gentoo from Scratch.
> >
>
> Gawd, that's funny! Thanks for making me smile, assuming I found what
> you're talking about:
>
> https://wiki.gentoo.org/wiki/Gentoo_ebuild_tree_from_scratch
>
> Just what every Gentoo user needs. More management!
>
> I really have fallen off the deep end thinking computers are just
> tools to get a job done. I'm ashamed of myself...
>
> In the really early days of Gentoo circa 2003 when I started there was
> some choice about regular Gentoo or a really low level install. I
> failed with the low level one but soon learned that every package on
> my machine was going to get rebuilt anyway so why bother?
>
> :-( Sad Mark. I'm a putz...

It is connected properly.  Everything I have is 1GB now.  When I got
fiber internet, I had to upgrade everything.  With the old DSL, well,
even a 10MB card would have worked.  lol 

See other reply that has more info.  I'm pretty sure it is the
encryption maxing out the CPU.  It's really the only thing that is
working hard.  Everything else seems to be doing more waiting than
anything. 

I just might find a simple distro that I can install on that thing. 
Honestly, I don't really care what is on it as long as it does what I
want and I can figure out how to make it work easily enough.  Still, I
wouldn't want Gentoo on a rig that slow.  I wouldn't have GUI stuff like
KDE, Firefox, libreoffice on it so that would save a lot of time but
still, even a headless machine would require some compile times.  Heck,
Ubuntu, Arch or any number of other distros would likely work just
fine.  Mostly, I need a better CPU.  If I encrypt anyway. 

This will work for the time being.  Now that I figured out how to make
it work.  ROFL

Dale

:-) 
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
On Sun, Dec 18, 2022 at 12:08 PM Dale <rdalek1967@gmail.com> wrote:
<SNIP>
> I just might find a simple distro that I can install on that thing.
Honestly, I don't really care what is on it as long as it does what I want
and I can figure out how to make it work easily enough.
<SNIP>

Consider Ubuntu Server. It's text only and will likely have everything you
need to start other than the NFS server which is easy enough to install.
ssh is on it by default,

You will need a monitor to install.

Update after install is probably two commands which is more or less
equivalent to emerge -DuN world but takes only a couple of minutes.

sudo apt update
sudo apt upgrade

There are lots of web pages that will tell you how to install and configure
the NFS server. It's not hard and only necessary if you actually want to
create mounts. Not necessary for rsync.
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
Am Sun, Dec 18, 2022 at 01:07:43PM -0600 schrieb Dale:

> See other reply that has more info.  I'm pretty sure it is the
> encryption maxing out the CPU.

The most simple benchmark is dd: unlock the LUKS layer on your HDD. then
first read from the HDD directly and then from the LUKS device:

dd if=<path/to/raw/HDD> of=/dev/null bs=1M count=1000
dd if=<path/to/cryptdevice> of=/dev/null bs=1M count=1000

It will tell you the transfer rate in MB/s.

> Mostly, I need a better CPU.  If I encrypt anyway. 

Did you ever tell us the exact CPU you have in there? All I can remember is
it has 4 cores. And some AMD processor with a II in its name, but that was
you main rig, right?

--
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

It's not a bug, it's tradition!
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
On Sun, Dec 18, 2022 at 2:30 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:
>
> Am Sun, Dec 18, 2022 at 01:07:43PM -0600 schrieb Dale:
>
> > Mostly, I need a better CPU. If I encrypt anyway.
>
> Did you ever tell us the exact CPU you have in there? All I can remember is
> it has 4 cores. And some AMD processor with a II in its name, but that was
> you main rig, right?

What encryption algorithm are you using? You should see if this is
hardware-accelerated in the kernel for your CPU, or if not if there is
another strong algorithm which is. Most newer CPUs will tend to have
hardware support for algorithms like AES, and the kernel will use
this. This will greatly improve CPU performance.

I've run into this issue with zfs on Raspberry Pis. ZFS does the
encryption internally, and the openzfs code didn't have support for
ARM hardware encryption the last time I checked (this could have
changed). I found that dm-crypt works MUCH better on Pis as a result,
as the kernel does have ARM encryption hardware support.

Again, this all depends on the algorithm. If you're using something
exotic odds are the hardware won't handle it natively.

--
Rich
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
On Sun, Dec 18, 2022 at 1:07 PM Rich Freeman <rich0@gentoo.org> wrote:
>
> On Sun, Dec 18, 2022 at 2:30 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:
> >
> > Am Sun, Dec 18, 2022 at 01:07:43PM -0600 schrieb Dale:
> >
> > > Mostly, I need a better CPU. If I encrypt anyway.
> >
> > Did you ever tell us the exact CPU you have in there? All I can
remember is
> > it has 4 cores. And some AMD processor with a II in its name, but that
was
> > you main rig, right?
>
> What encryption algorithm are you using? You should see if this is
> hardware-accelerated in the kernel for your CPU, or if not if there is
> another strong algorithm which is. Most newer CPUs will tend to have
> hardware support for algorithms like AES, and the kernel will use
> this. This will greatly improve CPU performance.
>
> I've run into this issue with zfs on Raspberry Pis. ZFS does the
> encryption internally, and the openzfs code didn't have support for
> ARM hardware encryption the last time I checked (this could have
> changed). I found that dm-crypt works MUCH better on Pis as a result,
> as the kernel does have ARM encryption hardware support.
>
> Again, this all depends on the algorithm. If you're using something
> exotic odds are the hardware won't handle it natively.
>
> --
> Rich

Great background info Rich. Thanks.

If Dale would supply us with the first few lines of the following command I
think it would help

mark@truenas1:~ $ sysctl hw
hw.machine: amd64
hw.model: Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz
hw.ncpu: 4


Note that hw.ncpu isn't actually cores but rather threads. My
processor has just 2 cores.

I don't know how to get the CPU flags on FreeBSD nor
how to determine if encryption is hardware or software
based on TrueNAS. Given some time I might Google
that.

Cheers,
Mark
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
<SNIP>

sysctl hw

SNIP>

sudo dmidecode -t processor -t cache
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
Am Sun, Dec 18, 2022 at 01:30:45PM -0700 schrieb Mark Knecht:

> I don't know how to get the CPU flags on FreeBSD nor
> how to determine if encryption is hardware or software
> based on TrueNAS. Given some time I might Google
> that.

Wikipedia has lists of practically everything:
https://en.wikipedia.org/wiki/List_of_AMD_processors
(We just need to know on which to click ;-) )

Another good source for such stuff is https://www.cpu-world.com/ .

--
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

Remember: now are the good old times
which you will be raving about in ten year’s time.
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
Frank Steinmetzger wrote:
> Am Sun, Dec 18, 2022 at 01:07:43PM -0600 schrieb Dale:
>
>> See other reply that has more info.  I'm pretty sure it is the
>> encryption maxing out the CPU.
> The most simple benchmark is dd: unlock the LUKS layer on your HDD. then
> first read from the HDD directly and then from the LUKS device:
>
> dd if=<path/to/raw/HDD> of=/dev/null bs=1M count=1000
> dd if=<path/to/cryptdevice> of=/dev/null bs=1M count=1000
>
> It will tell you the transfer rate in MB/s.

One thing about it, the speed is likely what its going to be.  Whether
it is the encryption or whatever, it is what it is.  It did improve a
little with the 1GB network card but not a whole lot.

>> Mostly, I need a better CPU.  If I encrypt anyway. 
> Did you ever tell us the exact CPU you have in there? All I can remember is
> it has 4 cores. And some AMD processor with a II in its name, but that was
> you main rig, right?
>


It took some digging around but I found out it is a AMD Phenom 9750 quad
core.  It lists some extensions but I don't see any that are related to
encryption.  No AES or whatever for sure.  Unless I missed it.  I used
the command cpuid to get this info on BSD.

My main rig has a AMD FX-8350 Eight-Core Processor and 32GBs of memory. 
I'm thinking about a new rig eventually.  Rig is getting a little age on
it.  ;-) 

Dale

:-)  :-) 
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
Am Sun, Dec 18, 2022 at 03:53:28PM -0600 schrieb Dale:

> > Did you ever tell us the exact CPU you have in there? All I can remember is
> > it has 4 cores. And some AMD processor with a II in its name, but that was
> > you main rig, right?
>
> It took some digging around but I found out it is a AMD Phenom 9750 quad
> core.

Great! That’s a model from 2008:
https://en.wikipedia.org/wiki/List_of_AMD_Phenom_processors#%22Agena%22_(B2/B3,_65_nm,_Quad-core)

> It lists some extensions but I don't see any that are related to
> encryption.  No AES or whatever for sure.  Unless I missed it.  I used

The instruction set’s original proposal is about as young as your processor:
https://en.wikipedia.org/wiki/AES_instruction_set

> My main rig has a AMD FX-8350 Eight-Core Processor and 32GBs of memory. 
> I'm thinking about a new rig eventually.  Rig is getting a little age on
> it.  ;-) 

At least it has AES ;-)
https://en.wikipedia.org/wiki/List_of_AMD_FX_processors#Piledriver_Core_(Vishera,_32_nm)

I’ve been thinking about a new build for about a year now, aiming at a
power-efficient Ryzen 5700G. But I can’t let go of my trusty old (8 years)
i5-4590.

--
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.
So this unvaccinated man comes in no bar ...
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
Frank Steinmetzger wrote:
> Am Sun, Dec 18, 2022 at 03:53:28PM -0600 schrieb Dale:
>
>>> Did you ever tell us the exact CPU you have in there? All I can remember is
>>> it has 4 cores. And some AMD processor with a II in its name, but that was
>>> you main rig, right?
>> It took some digging around but I found out it is a AMD Phenom 9750 quad
>> core.
> Great! That’s a model from 2008:
> https://en.wikipedia.org/wiki/List_of_AMD_Phenom_processors#%22Agena%22_(B2/B3,_65_nm,_Quad-core)
>
>>  It lists some extensions but I don't see any that are related to
>> encryption.  No AES or whatever for sure.  Unless I missed it.  I used
> The instruction set’s original proposal is about as young as your processor:
> https://en.wikipedia.org/wiki/AES_instruction_set
>
>> My main rig has a AMD FX-8350 Eight-Core Processor and 32GBs of memory. 
>> I'm thinking about a new rig eventually.  Rig is getting a little age on
>> it.  ;-) 
> At least it has AES ;-)
> https://en.wikipedia.org/wiki/List_of_AMD_FX_processors#Piledriver_Core_(Vishera,_32_nm)
>
> I’ve been thinking about a new build for about a year now, aiming at a
> power-efficient Ryzen 5700G. But I can’t let go of my trusty old (8 years)
> i5-4590.
>


What worries me, a cap on the mobo deciding to take a long break and let
the stink out.  I checked when I bought it, they do use high dollar caps
but still, they age no matter how well they are made.  I do have a
backup mobo tho.  Current is a Gigabyte 970 but backup is a 870.  It
will even run the same kernel I think. 

I just need to make time to find out where I get the most bang for the
buck.  I read a few weeks ago that Intel is the most cost effective
now.  It used to be that power compared to money that AMD was a much
better deal.  Things have changed.  That's all for another day tho. 
Gotta save up some money.  Likely need about $1,000 or maybe even more. 
I could reuse some things tho.  Mobo, CPU and memory being the big
things I'd need to even start.  Maybe a power supply too.  My P/S is
only a few years old tho.  Fairly modern.  Plenty of watts. 

One of these days. 

Dale

:-)  :-) 
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
On Sun, Dec 18, 2022 at 4:53 PM Dale <rdalek1967@gmail.com> wrote:
>
> It took some digging around but I found out it is a AMD Phenom 9750 quad
> core.

You might want to hook a Kill-a-watt to that and see how much power it
uses. Those older AMD processors were really inefficient.

One thing I've come to appreciate with anything that runs 24x7 is to
consider the energy bill to run things. That's why the bulk of my
storage is running on ARM SBCs. Those things use almost no power when
idle, while even a modern amd64 system will often use more at idle
than an ARM will at full load.

Most of the newer CPUs are MUCH better in performance per watt as
well. A server that uses just 50W 24x7 costs about $65/yr (and I have
pretty cheap electricity due to a 1yr contract - odds are it costs
quite a bit more for most of this list). That's a fairly significant
amount of money so it can pay to optimize things, especially if you
replace what was a power-hungry performance CPU from 10 years ago with
a lower-end CPU from today that probably has double the performance
for 10% of the power draw. Obviously if you want bleeding edge the
electricity bill won't cover the up-front costs, but lower end CPUs
are very cost-effective and efficient compared to really old systems
that people tend to use for these kinds of projects.

--
Rich
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
Rich Freeman wrote:
> On Sun, Dec 18, 2022 at 4:53 PM Dale <rdalek1967@gmail.com> wrote:
>> It took some digging around but I found out it is a AMD Phenom 9750 quad
>> core.
> You might want to hook a Kill-a-watt to that and see how much power it
> uses. Those older AMD processors were really inefficient.
>
> One thing I've come to appreciate with anything that runs 24x7 is to
> consider the energy bill to run things. That's why the bulk of my
> storage is running on ARM SBCs. Those things use almost no power when
> idle, while even a modern amd64 system will often use more at idle
> than an ARM will at full load.
>
> Most of the newer CPUs are MUCH better in performance per watt as
> well. A server that uses just 50W 24x7 costs about $65/yr (and I have
> pretty cheap electricity due to a 1yr contract - odds are it costs
> quite a bit more for most of this list). That's a fairly significant
> amount of money so it can pay to optimize things, especially if you
> replace what was a power-hungry performance CPU from 10 years ago with
> a lower-end CPU from today that probably has double the performance
> for 10% of the power draw. Obviously if you want bleeding edge the
> electricity bill won't cover the up-front costs, but lower end CPUs
> are very cost-effective and efficient compared to really old systems
> that people tend to use for these kinds of projects.
>


My UPS has a power meter thing on it which is pretty close.  It shows a
little over 100 watts for rig and monitor to when running.  Given it is
winter time here, I don't mind the extra heat.  It's either the puter or
a heater.  Given the forecast, I'm considering a emerge -e world pretty
soon.  lol  This NAS box won't run 24/7 tho.  I'll run it when I update
backups then shut it down and maybe even move it to a out building. 
Once I build a Raspberry thingy, I may have one as a 24/7 NAS box and
take some load off my main rig.  That I would like to have more
efficiency with.  That's for later tho.  First, I want to build a small
one that will fit in my fire safe.  Of course, I still want a larger
safe so that maybe I can have a 6 bay NAS box.  I wish I could find a
commercial or bank type safe that is used.  If it isn't to heavy. 

I still remember my old AMD 2500+ single core rig.  That thing pulled
over 400 watts idle.  If I was compiling a lot and all the fans were
running plus hard drives were busy, it would pull even more, close to
500 watts or something.  It's been a while but I remember winters were
nice but summers made the A/C run a good bit longer.  Thing is, my
current rig is much faster, likely 10 times faster, and pulls less than
200 watts even if at full power plus I have a LOT more hard drives.  It
goes to show, you can't measure computer power by what it pulls from the
wall.  :-)

If I like these Raspberry things, may make a media box out of one.  I'd
like to have a remote tho.  ;-)

Dale

:-)  :-) 
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
On Mon, Dec 19, 2022 at 12:11 AM Dale <rdalek1967@gmail.com> wrote:
>
> If I like these Raspberry things, may make a media box out of one. I'd
> like to have a remote tho. ;-)

So, I've done that. Honestly, these days a Roku is probably the
better option, or something like a Google Chromecast or the 47 other
variations on this them. Keep in mind that low-powered devices like
ARM SBCs (or Rokus) are very picky about codecs. If you're running
something like Plex the software will offload the transcoding to a
server when the codec isn't supported by the player, but if you're
doing something more DIY then you need to ensure your library conforms
to the specs or your player will just die. Even 1080p can be pushing
it for software decoding on an ARM SBC, and forget 4k. Heck, realtime
software transcoding 4k is pretty hard on most amd64 servers as well -
they will do it but it will use a significant amount of CPU just for
one stream.

These days I'm using Pis just for moosefs storage, and all the
compute-heavy stuff is on amd64. For players I just use TV clients or
Rokus plus Plex server on amd64. I don't really get any benefit out
of completely DIYing the client side and LIRC is a PITA.

--
Rich
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
On 19/12/2022 12:00, Rich Freeman wrote:
> On Mon, Dec 19, 2022 at 12:11 AM Dale<rdalek1967@gmail.com> wrote:
>> If I like these Raspberry things, may make a media box out of one. I'd
>> like to have a remote tho. ????

> So, I've done that. Honestly, these days a Roku is probably the
> better option, or something like a Google Chromecast or the 47 other
> variations on this them.

Where do you put that 2TB drive on your Roku or Chromecast?

I'm thinking of building a media server, not to drive the TV, but to
record and store. I thought that was what a media server was!

Cheers,
Wol
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
On Mon, Dec 19, 2022 at 7:51 AM Wols Lists <antlists@youngman.org.uk> wrote:
>
> On 19/12/2022 12:00, Rich Freeman wrote:
> > On Mon, Dec 19, 2022 at 12:11 AM Dale<rdalek1967@gmail.com> wrote:
> >> If I like these Raspberry things, may make a media box out of one. I'd
> >> like to have a remote tho. ????
>
> > So, I've done that. Honestly, these days a Roku is probably the
> > better option, or something like a Google Chromecast or the 47 other
> > variations on this them.
>
> Where do you put that 2TB drive on your Roku or Chromecast?
>
> I'm thinking of building a media server, not to drive the TV, but to
> record and store. I thought that was what a media server was!

So, he said "media box," which I assumed meant the client that
attaches to the TV. There are some canned solutions for media servers
- I think the NVidia Shield can run Plex server for example. However,
in general server-side I'd go amd64.

My current solution is:
1. Moosefs for storage: amd64 container for the master, and ARM SBCs
for the chunkservers which host all the USB3 hard drives. With a
modest number of them performance is very good, though certainly not
as good as Ceph or local storage. (I do have moosefs in my overlay -
might try to get that into the main repo when I get a chance.)
2. Plex server in a container on amd64 (looking to migrate this to k8s
over the holiday).
3. Rokus or TV apps for the clients.

--
Rich
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
Hi Rich

On Mon, Dec 19, 2022 at 6:30 AM Rich Freeman <rich0@gentoo.org> wrote:
<SNIP>
> My current solution is:
> 1. Moosefs for storage: amd64 container for the master, and ARM SBCs
> for the chunkservers which host all the USB3 hard drives.

I'm trying to understand the form factor of what you are mentioning above.
Presumably the chunkservers aren't sitting on a lab bench with USB
drives hanging off of them. Can you point me toward and example of
what you are using?

I've been considering some of these new mini-computers that have
a couple of 2.5Gb/S Ethernet ports and 3 USB 3 ports but haven't
moved forward because I want it packaged in a single case.

Where does the master reside? In a container on your desktop
machine or is that another element on your network?

<SNIP>

> 2. Plex server in a container on amd64 (looking to migrate this to k8s
> over the holiday).

Why Kubernetes? Is the Plex server safer when not being used? How
long does it take to spin up an instance and do the TV apps understand
this operation? Or would it be up and running all the time?

Thanks,
Mark
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
On Mon, Dec 19, 2022 at 11:43 AM Mark Knecht <markknecht@gmail.com> wrote:
>
> On Mon, Dec 19, 2022 at 6:30 AM Rich Freeman <rich0@gentoo.org> wrote:
> <SNIP>
> > My current solution is:
> > 1. Moosefs for storage: amd64 container for the master, and ARM SBCs
> > for the chunkservers which host all the USB3 hard drives.
>
> I'm trying to understand the form factor of what you are mentioning above.
> Presumably the chunkservers aren't sitting on a lab bench with USB
> drives hanging off of them. Can you point me toward and example of
> what you are using?

Well, a few basically are just sitting on a bench, but most are in
half-decent cases (I've found that Pi4s really benefit from a decent
case as they will thermal throttle otherwise). I then have USB3 hard
drives attached. I actually still have one RockPro64 with an LSI HBA
on a riser card but I'm going to be moving those drives to USB3-SATA
adapters because dealing with the kernel patches needed to fix the
broken PCIe driver is too much fuss, and the HBA uses a TON of power
which I didn't anticipate when I bought it (ugh, server gear).

Really at this point for anything new 2GB Pi4s are my preferred go-to
with Argon ONE v2 cases. Then I just get USB3 hard drives on sale at
Best Buy for ~$15/TB if possible. USB3 will handle a few hard drives
depending on how much throughput they're getting, but this setup is
more focused on capacity/cost than performance anyway.

The low memory requirement for the chunkservers is a big part of why I
went with MooseFS instead of Ceph. The OSDs for Ceph recommend
something like 4GB per hard drive which adds up very fast.

The USB3 hard drives do end up strewn about a fair bit, but they have
their own enclosures anyway. I just label them well.

>
> I've been considering some of these new mini-computers that have
> a couple of 2.5Gb/S Ethernet ports and 3 USB 3 ports but haven't
> moved forward because I want it packaged in a single case.

Yeah, better ethernet would definitely be on my wish list. I'll
definitely take a look at the state of those the next time I add a
node.

> Where does the master reside? In a container on your desktop
> machine or is that another element on your network?

In an nspawn container on one of my servers. It is pretty easy to set
up or migrate so it can go anywhere, but it does benefit from a bit
more CPU/RAM. Running it in a container creates obvious dependency
challenges if I want to mount moosefs on the same server - that can be
solved with systemd dependencies, but it won't figure that out on its
own.

> > 2. Plex server in a container on amd64 (looking to migrate this to k8s
> > over the holiday).
>
> Why Kubernetes?

I run it 24x7. This is half an exercise to finally learn and grok
k8s, and half an effort to just develop better container practices in
general. Right now all my containers run in nspawn which is actually
a very capable engine, but it does nothing for image management, so my
containers are more like pets than cattle. I want to get to a point
where everything is defined by a few trivially backed-up config files.

One thing I do REALLY prefer with nspawn is its flexibility around
networking. An nspawn container can use a virtual interface attached
to any bridge, which means you can give them their own IPs, routes,
gateways, VLANs, and so on. Docker and k8s are pretty decent about
giving containers a way to listen on the network for connection
(especially k8s ingress or load balancers), but they do nothing really
to manage the outbound traffic, which just uses the host network
config. On a multi-homed network or when you want to run services for
VLANs and so on it seems like a lot of trouble. Sure, you can go
crazy with iptables and iproute2 and so on, but I used to do that with
non-containerized services and hated it. With nspawn it is pretty
trivial to set that stuff up and give any container whatever set of
interfaces you want bridged however you want them. I actually fussed
a little with running a k8s node inside an nspawn container so that I
could just tie pods to nodes to do exotic networking but clusters
inside containers (using microk8s which runs in snapd) was just a
bridge too far...

--
Rich
Re: Living in NGL: was: NAS and replacing with larger drives [ In reply to ]
On 19/12/22 21:30, Rich Freeman wrote:
> On Mon, Dec 19, 2022 at 7:51 AM Wols Lists <antlists@youngman.org.uk> wrote:
>> On 19/12/2022 12:00, Rich Freeman wrote:
>>> On Mon, Dec 19, 2022 at 12:11 AM Dale<rdalek1967@gmail.com> wrote:
>>>> If I like these Raspberry things, may make a media box out of one. I'd
>>>> like to have a remote tho. ????
>>> So, I've done that. Honestly, these days a Roku is probably the
>>> better option, or something like a Google Chromecast or the 47 other
>>> variations on this them.
>> Where do you put that 2TB drive on your Roku or Chromecast?
>>
>> I'm thinking of building a media server, not to drive the TV, but to
>> record and store. I thought that was what a media server was!
> So, he said "media box," which I assumed meant the client that
> attaches to the TV. There are some canned solutions for media servers
> - I think the NVidia Shield can run Plex server for example. However,
> in general server-side I'd go amd64.
>
> My current solution is:
> 1. Moosefs for storage: amd64 container for the master, and ARM SBCs
> for the chunkservers which host all the USB3 hard drives. With a
> modest number of them performance is very good, though certainly not
> as good as Ceph or local storage. (I do have moosefs in my overlay -
> might try to get that into the main repo when I get a chance.)
> 2. Plex server in a container on amd64 (looking to migrate this to k8s
> over the holiday).
> 3. Rokus or TV apps for the clients.

Very similar to what I have (intel/arm for moosefs) - I am effectively
using moosefs as a distributed NAS (fuse mount onto whatever system(s) I
am using) with built in data protection and redundancy.  LVM and similar
pooling is discouraged as it defeats some of the built in data
protection. To increase storage, just add a disk, format, add to the
config and reload - it automatically redistributes the data.  Similarly,
you can add/remove storage or whole storage systems while live with no
risk to your data (within limits!!!) With LVM, if a drive fails, you are
SOL and offline until you can recover and restore the data.  On a recent
holiday, an SD card failed and a moosefs arm SBC in AU went offline -
discovered the next morning when doing status checks from a ship in the
Mediterranean(!) - it had already backfilled and protection was back at
normal, moosefs was just missing 2Tb of storage space.  5 weeks later
when I got home, I replaced the SD card, rebooted and readded the system
all with no risk to the data.

Dale, I was where you are about 10 or so years ago and was forced to
move on when that design hit its limits - forget LVM etc, these days
there are lots of better ways to do what you want with less risk to your
data.  Another factor is power - moosefs is currently 1 intel and 7 arm
SBC's that use 90-110w (most of which is due to using ancient WD and
Seagate hard drives) - where as my intel desktop is 90w when idle, or
over 300 w when compiling etc. so its off unless its being used. Power
is important to me as its expensive!!

BillK