Mailing List Archive

Issues with files accessed over NFS
Although I realize that this question is more about NFS than Apache it
still concerns my web server so here goes. This is at work so a bit of
history first, sorry for the novel length email in advance.

The set up I have was already in existence when I started my current
position. The web server, running Apache on top of Debian 8, was directly
connected to a storage server, also on Debian 8, via a dedicated 10Gbps
network link. The storage was accessed via NFS mounts, different mounts for
different areas in order to section them off within the website with
aliases. Not all of the website was stored on the server, just the ones
that used the most disk space.
Of course this was three years ago and Debian 8 went EOL. Before that
occurred it was my job to upgrade the servers to a later version of Debian.
In order to keep the distribution upgrades as far apart as possible I went
with Debian 11, upgrading on a path from 9 to 10 to 11. During the upgrade
all services were checked out for basic functionality and no problems were
seen.
Not long after the upgrade an interesting problem was apparent. Files
accessed over NFS were not downloading properly. The download would start
but then stop anywhere from 30% to 60%. These files were not large with the
smallest ones being just over 1MB. Any file stored on the web server itself
downloaded just fine, no matter the size. I checked just about everything
on that connection from the cable, which was only 1 foot in length so I got
a longer one, to the NFS mount options. Changes were made but they did not
seem to make a difference and all file transfer tests done within the OS
itself worked fine with files up to 100MB tested.
The only thing I did not check until recently was the NFS version. The
mounts were automatically using NFSv4 which I considered, at this point in
its development, to be stable. However when I switched to NFSv3 the problem
went away. So after all that I seemed to have "solved" it by dropping back
to an older version of NFS.

So the questions I have for the community are: Has anyone seen a similar
issue and solved it? If so did you find another solution other than
reverting to NFSv3?

Thanks,

Chris Woelkers
Re: Issues with files accessed over NFS [ In reply to ]
check the mount options for the NFS mount on apache server as well as the underlying file system on the NFS server. unless needed, set to noatime and nodiratime
________________________________
From: Chris Woelkers - NOAA Federal <chris.woelkers@noaa.gov.INVALID>
Sent: Thursday, June 30, 2022 3:07:01 PM
To: users@httpd.apache.org <users@httpd.apache.org>
Subject: [users@httpd] Issues with files accessed over NFS

Although I realize that this question is more about NFS than Apache it still concerns my web server so here goes. This is at work so a bit of history first, sorry for the novel length email in advance.

The set up I have was already in existence when I started my current position. The web server, running Apache on top of Debian 8, was directly connected to a storage server, also on Debian 8, via a dedicated 10Gbps network link. The storage was accessed via NFS mounts, different mounts for different areas in order to section them off within the website with aliases. Not all of the website was stored on the server, just the ones that used the most disk space.
Of course this was three years ago and Debian 8 went EOL. Before that occurred it was my job to upgrade the servers to a later version of Debian. In order to keep the distribution upgrades as far apart as possible I went with Debian 11, upgrading on a path from 9 to 10 to 11. During the upgrade all services were checked out for basic functionality and no problems were seen.
Not long after the upgrade an interesting problem was apparent. Files accessed over NFS were not downloading properly. The download would start but then stop anywhere from 30% to 60%. These files were not large with the smallest ones being just over 1MB. Any file stored on the web server itself downloaded just fine, no matter the size. I checked just about everything on that connection from the cable, which was only 1 foot in length so I got a longer one, to the NFS mount options. Changes were made but they did not seem to make a difference and all file transfer tests done within the OS itself worked fine with files up to 100MB tested.
The only thing I did not check until recently was the NFS version. The mounts were automatically using NFSv4 which I considered, at this point in its development, to be stable. However when I switched to NFSv3 the problem went away. So after all that I seemed to have "solved" it by dropping back to an older version of NFS.

So the questions I have for the community are: Has anyone seen a similar issue and solved it? If so did you find another solution other than reverting to NFSv3?

Thanks,

Chris Woelkers
Re: Issues with files accessed over NFS [ In reply to ]
The underlying filesystem is ZFS and the drives, all 28 of them, are in
good health. I'll check those options out.

Thanks,

Chris Woelkers
IT Specialist
National Oceanic and Atmospheric Administration
Great Lakes Environmental Research Laboratory
4840 S State Rd | Ann Arbor, MI 48108
Office: 734-741-2446
Cell: 734-604-0059
Voice: 734-219-3815?


On Thu, Jun 30, 2022 at 3:22 PM Bender, Charles
<charles@beachcamera.com.invalid> wrote:

> check the mount options for the NFS mount on apache server as well as the
> underlying file system on the NFS server. unless needed, set to noatime and
> nodiratime
> ------------------------------
> *From:* Chris Woelkers - NOAA Federal <chris.woelkers@noaa.gov.INVALID>
> *Sent:* Thursday, June 30, 2022 3:07:01 PM
> *To:* users@httpd.apache.org <users@httpd.apache.org>
> *Subject:* [users@httpd] Issues with files accessed over NFS
>
> Although I realize that this question is more about NFS than Apache it
> still concerns my web server so here goes. This is at work so a bit of
> history first, sorry for the novel length email in advance.
>
> The set up I have was already in existence when I started my current
> position. The web server, running Apache on top of Debian 8, was directly
> connected to a storage server, also on Debian 8, via a dedicated 10Gbps
> network link. The storage was accessed via NFS mounts, different mounts for
> different areas in order to section them off within the website with
> aliases. Not all of the website was stored on the server, just the ones
> that used the most disk space.
> Of course this was three years ago and Debian 8 went EOL. Before that
> occurred it was my job to upgrade the servers to a later version of Debian.
> In order to keep the distribution upgrades as far apart as possible I went
> with Debian 11, upgrading on a path from 9 to 10 to 11. During the upgrade
> all services were checked out for basic functionality and no problems were
> seen.
> Not long after the upgrade an interesting problem was apparent. Files
> accessed over NFS were not downloading properly. The download would start
> but then stop anywhere from 30% to 60%. These files were not large with the
> smallest ones being just over 1MB. Any file stored on the web server itself
> downloaded just fine, no matter the size. I checked just about everything
> on that connection from the cable, which was only 1 foot in length so I got
> a longer one, to the NFS mount options. Changes were made but they did not
> seem to make a difference and all file transfer tests done within the OS
> itself worked fine with files up to 100MB tested.
> The only thing I did not check until recently was the NFS version. The
> mounts were automatically using NFSv4 which I considered, at this point in
> its development, to be stable. However when I switched to NFSv3 the problem
> went away. So after all that I seemed to have "solved" it by dropping back
> to an older version of NFS.
>
> So the questions I have for the community are: Has anyone seen a similar
> issue and solved it? If so did you find another solution other than
> reverting to NFSv3?
>
> Thanks,
>
> Chris Woelkers
>
Re: Issues with files accessed over NFS [ In reply to ]
Am 30.06.2022 um 21:07 schrieb Chris Woelkers - NOAA Federal:
> Although I realize that this question is more about NFS than Apache it
> still concerns my web server so here goes. This is at work so a bit of
> history first, sorry for the novel length email in advance.
>
> The set up I have was already in existence when I started my current
> position. The web server, running Apache on top of Debian 8, was
> directly connected to a storage server, also on Debian 8, via a
> dedicated 10Gbps network link. The storage was accessed via NFS mounts,
> different mounts for different areas in order to section them off within
> the website with aliases. Not all of the website  was stored on the
> server, just the ones that used the most disk space.
> Of course this was three years ago and Debian 8 went EOL. Before that
> occurred it was my job to upgrade the servers to a later version of
> Debian. In order to keep the distribution upgrades as far apart as
> possible I went with Debian 11, upgrading on a path from 9 to 10 to 11.
> During the upgrade all services were checked out for basic functionality
> and no problems were seen.
> Not long after the upgrade an interesting problem was apparent. Files
> accessed over NFS were not downloading properly. The download would
> start but then stop anywhere from 30% to 60%. These files were not large
> with the smallest ones being just over 1MB. Any file stored on the web
> server itself downloaded just fine, no matter the size. I checked just
> about everything on that connection from the cable, which was only 1
> foot in length so I got a longer one, to the NFS mount options. Changes
> were made but they did not seem to make a difference and all file
> transfer tests done within the OS itself worked fine with files up to
> 100MB tested.
> The only thing I did not check until recently was the NFS version. The
> mounts were automatically using NFSv4 which I considered, at this point
> in its development, to be stable. However when I switched to NFSv3 the
> problem went away. So after all that I seemed to have "solved" it by
> dropping back to an older version of NFS.
>
> So the questions I have for the community are: Has anyone seen a similar
> issue and solved it? If so did you find another solution other than
> reverting to NFSv3?

Try disabling sendfile and mmap:

https://httpd.apache.org/docs/2.4/en/mod/core.html#enablesendfile

https://httpd.apache.org/docs/2.4/en/mod/core.html#enablemmap

If that helps, check out, which of the two is the culprit.

Best regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org