Mailing List Archive

mythth 32
This is not a troll but a genuine question/observation.

I am biased, I do not like ubuntu because of the endless 'You shall not ... " eg dmesg eg mythtv password.

After some fumbling I installed 22.04, built from scratch, using anisible, (32-fixes uptodate). I have issues ie playing while recording ... glitch back to beginning ... later ... pause of 15 secs ...

I installed opensuse leap 15.4: much better but still ocassional (once-a-week) ... pause cica 15 secs ...

Using the same hardware with fixes-31I've never had any glitches, ever.

So it seems ubuntu 22.04 is a tad flakey (there were other minor gotchas) and it seems 32-fixes is less robust than 31-fixes.

This machine runs my development VMs so it is a 4 core i7 NUC with 24G RAM and 8T flash. The only spinning rust is for backups.

I do not think io is an issue
/dev/nvme0n1p5:
Timing cached reads: 37824 MB in 2.00 seconds = 18942.04 MB/sec
Timing buffered disk reads: 7850 MB in 3.00 seconds = 2616.18 MB/sec
(I know to take hdparn with a pinch)

Can anybody comment on 22.04
Can anybody comment on 32 vs 31

Thanks
James
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythth 32 [ In reply to ]
On Fri, 8 Jul 2022 23:08:57 +0800, you wrote:

>This is not a troll but a genuine question/observation.
>
>I am biased, I do not like ubuntu because of the endless 'You shall not ... " eg dmesg eg mythtv password.
>
>After some fumbling I installed 22.04, built from scratch, using anisible, (32-fixes uptodate). I have issues ie playing while recording ... glitch back to beginning ... later ... pause of 15 secs ...
>
>I installed opensuse leap 15.4: much better but still ocassional (once-a-week) ... pause cica 15 secs ...
>
>Using the same hardware with fixes-31I've never had any glitches, ever.
>
>So it seems ubuntu 22.04 is a tad flakey (there were other minor gotchas) and it seems 32-fixes is less robust than 31-fixes.
>
>This machine runs my development VMs so it is a 4 core i7 NUC with 24G RAM and 8T flash. The only spinning rust is for backups.
>
>I do not think io is an issue
>/dev/nvme0n1p5:
> Timing cached reads: 37824 MB in 2.00 seconds = 18942.04 MB/sec
> Timing buffered disk reads: 7850 MB in 3.00 seconds = 2616.18 MB/sec
>(I know to take hdparn with a pinch)
>
>Can anybody comment on 22.04
>Can anybody comment on 32 vs 31
>
>Thanks
>James

Ubuntu versions are often a little flakey when they first come out.
That is one reason why the LTS upgrades are not recommended until the
.1 version has been released, and you have to manually override the
system to do an LTS upgrade to a .0 version.

When running any system in a VM, unless you have one that is fully set
up for real-time operation, there can be times when the main system is
busy and delays the operation of the VM. It should not be enough to
cause problems with recordings though. A 15 second pause is way
bigger than should ever happen.

Ubuntu packages can also have customisations of the code for things
that Ubuntu does a bit differently. So when you compile from the
original source without the custom patches, you will not be getting
that code. It is better to just install the Ubuntu packages from the
Mythbuntu v32-fixes repository. But again, that should not cause 15
second pauses.

I have been running v32-fixes for quite a while now, and have not
found it to be any less (or more) reliable than v31-fixes.

The timings you posted for I/O are for reads. NVME disks are
extremely fast always for reads. Where they can be much slower is for
writes. That will happen (as on any flash memory) if they run out of
erased blocks and have to erase one before they can write. Recording
TV to an SSD causes large numbers of SSD blocks to be written to, so
it is possible that you are running out of blocks on the erased list.
Do you have the SSDs set up to do TRIM only, or have you set them to
use the discard option in fstab? If you are doing TRIM only, then
that is likely only happening once a day at the most (unless you have
changed that), so no blocks will be being erased until TRIM is run.
Set the discard option to make the system hand back erasable blocks as
soon as they no longer form part of a file, so that the SSD can erase
them immediately.

And be aware that by recording to your SSDs, you will be seriously
increasing the wear on them. Run smartctl on them and check the wear
values reported by SMART.

When you are playing while recording, is that playing the same
recording that is currently recording, or another different recording?
The former can be a bit glitchy if you get too close to the point
where the recording is happening. It is best to stay a few seconds
back from that point. The code does make an allowance for this
problem and prevents you from getting what it thinks is too close to
the point of recording, but the devs who wrote that code seem to all
be using ATSC, and I think DVB-T is just enough different that you
need to be a bit further back to avoid problems.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythth 32 [ In reply to ]
Since I asked for opinion I'd not be so rude as to refute any, but some comments <smile>

> On 9 Jul 2022, at 8:00 pm, mythtv-dev-request@mythtv.org wrote:
>
> This is not a troll but a genuine question/observation.
>
> I am biased, I do not like ubuntu because of the endless 'You shall not ... " eg dmesg eg mythtv password.
>
> After some fumbling I installed 22.04, built from scratch, using anisible, (32-fixes uptodate). I have issues ie playing while recording ... glitch back to beginning ... later ... pause of 15 secs ...
>
> I installed opensuse leap 15.4: much better but still ocassional (once-a-week) ... pause cica 15 secs ...
>
> Using the same hardware with fixes-31I've never had any glitches, ever.
>
> So it seems ubuntu 22.04 is a tad flakey (there were other minor gotchas) and it seems 32-fixes is less robust than 31-fixes.
>
> This machine runs my development VMs so it is a 4 core i7 NUC with 24G RAM and 8T flash. The only spinning rust is for backups.
>
> I do not think io is an issue
> /dev/nvme0n1p5:
> Timing cached reads: 37824 MB in 2.00 seconds = 18942.04 MB/sec
> Timing buffered disk reads: 7850 MB in 3.00 seconds = 2616.18 MB/sec
> (I know to take hdparn with a pinch)
>
> Can anybody comment on 22.04
> Can anybody comment on 32 vs 31
>

> Ubuntu versions are often a little flakey when they first come out.
> That is one reason why the LTS upgrades are not recommended until the
> .1 version has been released, and you have to manually override the
> system to do an LTS upgrade to a .0 version.
>
> When running any system in a VM, unless you have one that is fully set
> up for real-time operation, there can be times when the main system is
> busy and delays the operation of the VM. It should not be enough to
> cause problems with recordings though. A 15 second pause is way
> bigger than should ever happen.

My frontend is on the main system, not a vm.

> Ubuntu packages can also have customisations of the code for things
> that Ubuntu does a bit differently. So when you compile from the
> original source without the custom patches, you will not be getting
> that code. It is better to just install the Ubuntu packages from the
> Mythbuntu v32-fixes repository. But again, that should not cause 15
> second pauses.

I was unaware - thanks.

> I have been running v32-fixes for quite a while now, and have not
> found it to be any less (or more) reliable than v31-fixes.
>
> The timings you posted for I/O are for reads. NVME disks are
> extremely fast always for reads. Where they can be much slower is for
> writes. That will happen (as on any flash memory) if they run out of
> erased blocks and have to erase one before they can write. Recording
> TV to an SSD causes large numbers of SSD blocks to be written to, so
> it is possible that you are running out of blocks on the erased list.
> Do you have the SSDs set up to do TRIM only, or have you set them to
> use the discard option in fstab? If you are doing TRIM only, then
> that is likely only happening once a day at the most (unless you have
> changed that), so no blocks will be being erased until TRIM is run.
> Set the discard option to make the system hand back erasable blocks as
> soon as they no longer form part of a file, so that the SSD can erase
> them immediately.

If there are Ts available for writing, surely 'available' is moot?

> And be aware that by recording to your SSDs, you will be seriously
> increasing the wear on them. Run smartctl on them and check the wear
> values reported by SMART.

According to SMART, based on 2 years of use (hours : 19221) and TBW (Data Units Written: 37,386,769 [19.1 TB]) them disks will outlast moi.

> When you are playing while recording, is that playing the same
> recording that is currently recording

yup

> , or another different recording?
> The former can be a bit glitchy if you get too close to the point
> where the recording is happening. It is best to stay a few seconds
> back from that point. The code does make an allowance for this
> problem and prevents you from getting what it thinks is too close to
> the point of recording, but the devs who wrote that code seem to all
> be using ATSC, and I think DVB-T is just enough different that you
> need to be a bit further back to avoid problems.

I'll check next time.

Devil-n-Dark Blue Sea: If you have new ie intel-gen 11 hw then you need new os to support it. I must try HSE.
I don't like ubuntu, but they do have a lot of 'nices'.
James
Re: mythth 32 [ In reply to ]
On Sun, 10 Jul 2022 15:01:49 +0800, you wrote:

>Since I asked for opinion I'd not be so rude as to refute any, but some comments <smile>

>> The timings you posted for I/O are for reads. NVME disks are
>> extremely fast always for reads. Where they can be much slower is for
>> writes. That will happen (as on any flash memory) if they run out of
>> erased blocks and have to erase one before they can write. Recording
>> TV to an SSD causes large numbers of SSD blocks to be written to, so
>> it is possible that you are running out of blocks on the erased list.
>> Do you have the SSDs set up to do TRIM only, or have you set them to
>> use the discard option in fstab? If you are doing TRIM only, then
>> that is likely only happening once a day at the most (unless you have
>> changed that), so no blocks will be being erased until TRIM is run.
>> Set the discard option to make the system hand back erasable blocks as
>> soon as they no longer form part of a file, so that the SSD can erase
>> them immediately.
>
>If there are Ts available for writing, surely 'available' is moot?

No, we are talking about the underlying SSD operations here. Once you
have written more data than the size of the SSD, it will have written
to all its data blocks at least once. So after that, before it can
write to a block, it will have to erase a block.

In a flash device like an SSD, writes can only change 1 bits to 0
bits, not the other way around. When a block of flash is erased, it
gets set to all 1 bits and then can be written to by changing all the
1s in that block to 0s wherever the data being written is a 0. Erase
operations can only be done on a block, not at a bit or byte level. A
block might be 4096 bytes - it varies from one SSD to another. Each
flash block can only be erased a certain number of times before it
fails to erase correctly. The SSD operating system keeps track of
which physical flash block is assigned to which logical flash block.
It remaps the physical blocks to logical blocks so that it can control
which blocks have been erased most often. This is how it does wear
leveling.

Whenever the operating system stops using a block of data at the
filesystem level, it needs to tell the SSD that it is no longer using
that address space and it can be erased. That is the function of the
TRIM command that the OS sends to the SSD over its SATA or NVMe
command channel. The SSD receives the TRIM commands and works out
which of its logical flash blocks are now fully unused by the
operating system. Whenever it finds a full flash block that is now
unused, it queues it to be erased. Erase operations are very slow (up
to seconds long) and take a lot of power. Each group of flash blocks
that makes up the entire SSD can only erase one of its blocks at once.
Once a flash block has been erased, it is put on the list of erased
blocks.

When the OS writes to a logical block on the SSD, either that logical
block already contains data and data in the block needs to be changed,
or the block is not in use and a new physical block need to be
assigned to that logical block by the SSD operating system. In the
latter case, the SSD just looks at its list of erased blocks and
assigns one and does the write to that block. In the case of changing
data in an existing logical block, the SSD will check to see if it is
just able to write some more 1 bits down to 0 bits - if so, then it
will do that in the existing physical block. If there are any bits
that need to change from 0 to 1, then it has to use an erased block
where the bits are all 1s. So it gets an erased block from its list
of erased blocks, copies all the data from the old physical block to
RAM, writes the new data over the old data in RAM, then writes the
entire RAM block to the erased physical block it has just assigned for
this job. When the flash write is complete, it moves the old physical
block to the list of blocks to be erased, and assigns the new physical
block to become that logical block.

The problem with TRIM operations is when they are done. If you do not
have the discard option set for a filesystem, then the OS only sends
TRIM commands when it runs its fstrim program, typically once a day as
a cron or systemd job.

The way fstrim works is that it looks at all files on the filesystem
and works out what logical blocks (filesystem logical blocks here) are
not in use and sends TRIM commands to the SSD to say what those unused
areas are. The SSD works out the mappings of those TRIM commands to
its logical flash blocks (which are different from the filesystem
logical blocks) and any logical flash blocks that are now unused by
the operating system are unmapped from physical blocks and their
corresponding physical blocks are queued to be erased.

When discard has been specified for a filesystem, any time the
operating system stops using a filesystem level logical block (when a
file gets deleted, for example), the operating system immediately
queues up a set of TRIM commands for all the filesystem logical blocks
to tell the SSD that those areas are no longer in use. So shortly
after a file is deleted (or shortened), the unused filesystem logical
blocks will have had TRIM commands received by the SSD and the
corresponding physical flash blocks will be on the queue to be erased.
And several seconds after that the erases will be completed and those
physical blocks will be available on the SSD's erased list. Without
discard, no TRIM commands will have been sent and all the
corresponding physical flash blocks on the SSD will still be marked as
in use.

So for the purposes of a worst case example, say you have a 1 Tbyte
SSD you are writing MythTV recordings to, and it has the minimum free
storage that MythTV leaves on a recording partition, around 20
Gibytes, and the rest of the partition is full. MythTV wants to start
a new recording and deletes an old 2 Gibyte recording from the
partition to make room, so there now 22 Gibytes of free space that
MythTV can see, so it starts the new recording. However, fstrim has
not been run for a while, and only the 20 Gibytes of free space that
was there is actually erased on the SSD. MythTV writes its new
recording happily, and keeps on writing as it is cricket game that is
going to go on for hours. Every so often, MythTV sees the free space
on the partition drop below its 20 Gibyte limit and deletes another
old recording. However, fstrim has still not run, so while MythTV is
happily seeing 20 Gibytes or more of free space, the SSD is rapidly
using up its 20 Gibytes of erased flash blocks. After MythTV has
written 20 Gibytes of new recording file, it has deleted 20 Gibytes of
recording space, but the SSD now no longer has any erased blocks
available. The next time MythTV writes to that partition, the write
will fail.

So the rule for SSDs is that if you ever expect to write as much data
to the SSD between fstrim runs as there is free filesystem space when
fstrim last ran, then you need to have discard enabled. Or you need
to run fstrim more often. In my opinion, if you are doing heavy write
traffic to an SSD, you just need to use discard, as that way you will
not get into trouble. In the past, discard was not used as having it
enabled slowed down the filesystem delete operations - the system did
not report the delete complete until the TRIMs had been sent to the
SSD. I think that problem is long fixed, and the TRIMs are now just
queued to be sent in the background and the delete operation will
report complete as soon as the TRIMs have been queued. And in any
case, the slight slowdown was not normally noticeable unless you were
doing a benchmark. But for some reason, most of the advice on the net
still recommends only using fstrim and not discard. And all the
distros still seem to be set up this way by default.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org