From: Erez Zadok Date: Sun, 18 Sep 2005 21:56:55 +0000 (+0000) Subject: * README.attrcache: new file documenting in detail OS bugs X-Git-Tag: before-xstr~16 X-Git-Url: https://git.fsl.cs.stonybrook.edu/?a=commitdiff_plain;h=a3b0b40db6cf1835bbea486d85ff5f5a759fdaa6;p=am-utils-6.1.git * README.attrcache: new file documenting in detail OS bugs relating to attribute caching, which can affect Amd's reliability under heavy load. * doc/Makefile.am: install README.attrcache file as attrache.txt file on am-utils Web site. * Makefile.am (EXTRA_DIST): include README.attrcache in distro. * BUGS, NEWS, doc/am-utils.texi (auto_attrcache Parameter), scripts/amd.conf.5, scripts/amd.conf-sample: document attribute cache bugs on some OSs. --- diff --git a/BUGS b/BUGS index 32f4c04..8c45e33 100644 --- a/BUGS +++ b/BUGS @@ -284,5 +284,20 @@ the resulting configure script by an order of magnitude, and for no real gain. Configure is big enough as it is, we don't need any more useless baggage in it. - -Erez & Ion +(18) NetBSD 2.0.2, FreeBSD 4.6, OpenBSD 3.7, and quite possibly most other + BSDs and other OSs (as of September 2005) + +Some BSD kernels don't have a way to turn off the NFS attribute cache. They +don't have a 'noac' mount flag, and setting various cache timeout fields in +struct nfs_args doesn't turn off the attribute cache; instead, it sets the +attribute cache timeout to some internal hard-coded default (usually +anywhere from 5-30 seconds). If Amd cannot turn off the NFS attribute +cache, under heavy Amd usage, users could get ESTALE errors from automounted +symlinks, or find that those symlinks point to the wrong place. One +workaround which would minimize this effect is to set auto_attrcache=1 in +your amd.conf, but it doesn't eliminate the problem! The best solutions are +(1) to use Amd in Autofs mode, if it's supported in your OS, and (2) talk to +your OS vendor to support a true "noac" flag. See README.attrcache for more +details. + +Erez & the am-utils team. diff --git a/ChangeLog b/ChangeLog index 91e469a..d7e1727 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,18 @@ +2005-09-18 Erez Zadok + + * README.attrcache: new file documenting in detail OS bugs + relating to attribute caching, which can affect Amd's reliability + under heavy load. + + * doc/Makefile.am: install README.attrcache file as attrache.txt + file on am-utils Web site. + + * Makefile.am (EXTRA_DIST): include README.attrcache in distro. + + * BUGS, NEWS, doc/am-utils.texi (auto_attrcache Parameter), + scripts/amd.conf.5, scripts/amd.conf-sample: document attribute + cache bugs on some OSs. + 2005-09-17 Erez Zadok * libamu/xutil.c (clocktime): newly implemented routine. Uses diff --git a/Makefile.am b/Makefile.am index 1e95139..760dce4 100644 --- a/Makefile.am +++ b/Makefile.am @@ -290,6 +290,7 @@ EXTRA_DIST = \ LSM.am-utils \ MIRRORS.txt \ MIRRORS.html \ + README.attrcache \ README.autofs \ README.ldap \ README.osx \ diff --git a/NEWS b/NEWS index b55dfe5..ae499be 100644 --- a/NEWS +++ b/NEWS @@ -23,6 +23,11 @@ time whenever Amd needs to update an mtime/atime/ctime of any node. Finally, we now update the mtime in places that were never updated before (when a whole map is flushed or a single entry times out). +Warning: some OSs, we discovered, are incapable of turning off their +NFS attribute cache entirely. This means that Amd cannot work fully +reliability on these systems, not under heavy load. This is documented in +detail in the README.attrcache file with this distribution. + - minor new ports: i386-pc-linux-deb3.1 powerpc-apple-darwin8.2.0 diff --git a/README.attrcache b/README.attrcache new file mode 100644 index 0000000..04963db --- /dev/null +++ b/README.attrcache @@ -0,0 +1,110 @@ + NFS Attribute Caching OS Problems and Amd + Last updated September 18, 2005 + +* Summary: + +Some OSs don't seem to have a way to turn off the NFS attribute cache, which +breaks the Amd automounter so badly that it is not recommend using Amd on +such OS for heavy use, not until this is fixed. + + +* Details: + +Amd is a user-level NFSv2 server that manages automounts of all other file +systems. The kernel contacts Amd via RPCs, and Amd in turn performs the +actual mounts, and then responds back to the kernel's RPCs. Every kernel +caches attributes of files, in a cache called the Directory Name Lookup +Cache (DNLC), or a Directory Cache (dcache). + +Amd manages its namespace in the user level, but the kernel caches names +itself. So the two must coordinate to ensure that both namespaces are in +sync. If the kernel uses a cached entry from the DNLC, without consulting +Amd, users may see corruption of the automounter namespace (symlinks +pointing to the wrong places, ESTALE errors, and more). For example, +suppose Amd timed out an entry and removed the entry from Amd's namespace. +Amd has to tell the kernel to purge its corresponding DNLC entry too. The +way Amd often does that is by incrementing the last modification time +(mtime) of the parent directory. This is the most common method for kernels +to check if their DNLC entries are stale: if the parent directory mtime is +newer, the kernel will discard all cached entries for that directory, and +will re-issue lookup methods. Those lookups will result in +NFS_GETATTR/NFS_LOOKUP calls sent from the kernel down to Amd, and Amd can +then properly inform the kernel of the new state of automounted entries. + +In order to ensure that Amd is "in charge" of its namespace without +interference from the kernel, Amd will try to turn off the NFS attribute +cache. It does so by using the NFSMNT_NOAC flag, if it exists, or by +setting various "cache timeout" fields in struct nfs_args to 0 (acregmin, +acregmax, acdirmin, or acdirmax). + +We have released a major new version of am-utils, version 6.1, in June 2005. +Since then, a lot of people have experimented with Amd, in anticipation of +migrating from the very old am-utils 6.0 to the new 6.1. For a couple of +months since the release of 6.1, we have received reports of problems with +Amd, especially under heavy use. Users reported getting ESTALE errors from +time to time, or seeing automounted entries whose symlinks don't point to +where it should be. After much debugging, we traced it to a few places in +Amd where it wasn't updating the parent directory mtime as it should have; +in some places where Amd was indeed updating the mtime, it was using a +resolution of only 1 second, which was not fine enough under heavy load. We +fixed this problem and switched to using a microsecond resolution mtime. + +After fixing this in Amd, we went on to verify that things work for other +OSs. When we got to FreeBSD 4.6 and NetBSD 2.0.2, we found out that they +always cache directory entries, and there is no way to turn it off +completely. Specifically, if we set the ac{reg,dir}{min,max} fields in +struct nfs_args all to zero, the kernel seems to cache the entries for a +default number of seconds (something like 5-30 seconds). On some OSs, +setting these four fields to 0 turns off the attribute cache, but not on +these two BSDs. We were able to verify this using Amd and a script that +exercises the interaction of the kernel's attrcache and Amd. (If you're +interested, the script can be made available.) + +We then experimented by setting the ac{reg,dir}{min,max} fields in struct +nfs_args all to 1, the smallest non-zero value we could. When we ran the +Amd exercising script, we found that the value of 1 reduced the race between +the DNLC and Amd, and the script took a little longer to run before it +detected an incoherency. That makes sense: the smaller the DNLC cache +interval is, the shorter the window of vulnerability is. (BTW, the man +pages on some OSs say that the ac{reg,dir}{min,max} fields use a 1 second +resolution, but experimentation indicated it was in 0.1 second units.) + +Clearly, setting the ac{reg,dir}{min,max} fields to 0 is worse than setting +it to 1 on those OSs that don't have a way to turn off the attribute cache. +So the current workaround I've implemented in am-utils is to create a +configuration parameter called "broken_attrcache" which, if turned on, will +set these nfs_args fields to 1 instead of 0. I wish I didn't have to create +such ugly workaround features in Amd, but I've got no choice. + +The near term solution is for FreeBSD to support a true 'noac' flag, which +can be added fairly easily. This'd make Amd work reliably. + +The long term solution is to implement Autofs support for all OSs and to +support it in Amd. Currently, Amd supports autofs on Solaris and Linux; +FreeBSD is next. Still, we found that even with autofs support, many +sysadmins still prefer to use the good 'ol non-autofs mode. + + +* Confirmed Status + +This is the confirmed status of various OSs' vulnerability to this attribute +cache bug. We are slowly checking the status of other OSs. The status of +any OS not listed is unknown as of the date at the top of this file. + +** Not Vulnerable (support a proper "noac" flag): + +Sun Solaris 8 and 9 (10 probably works fine) +Linux: 2.6.11 kernel (2.4.latest probably works fine) + +** Vulnerable (don't support a proper "noac" flag): + +NetBSD 2.0.2 (older versions also probably affected) +FreeBSD 4.6 (older versions also probably affected) +OpenBSD 3.7 (older versions also probably affected) + +Note: NetBSD has promised to support a noac flag hopefully after 2.1.0 is +released (maybe in 3.0 or 2.2). + + +Sincerely, +Erez. diff --git a/doc/Makefile.am b/doc/Makefile.am index 4b06bca..03c62bf 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -78,6 +78,10 @@ install-y2k: $(AMU_HTML_TOP)/y2k.txt $(AMU_HTML_TOP)/y2k.txt: $(top_srcdir)/README.y2k install -c -m 644 $< $@ +install-attrcache: $(AMU_HTML_TOP)/attrcache.txt +$(AMU_HTML_TOP)/attrcache.txt: $(top_srcdir)/README.attrcache + install -c -m 644 $< $@ + install-ldap: $(AMU_HTML_TOP)/ldap.txt $(AMU_HTML_TOP)/ldap.txt: $(top_srcdir)/README.ldap install -c -m 644 $< $@ diff --git a/doc/am-utils.texi b/doc/am-utils.texi index 2eb32aa..59758fd 100644 --- a/doc/am-utils.texi +++ b/doc/am-utils.texi @@ -4374,14 +4374,34 @@ option to @i{Amd}. Allows you to override the value of the @i{arch} @subsection @t{auto_attrcache} Parameter @cindex auto_attrcache Parameter -(type=numeric, default=0). Specify in seconds, the (kernel-side) NFS -attribute cache timeout for @i{Amd}'s own automount points. A value -of 0 turns off attribute caching, meaning that @i{Amd} will be -consulted via a kernel-RPC each time someone stat's the mount point +(type=numeric, default=0). Specify in seconds (or units of 0.1 +seconds, depending on the OS), what is the (kernel-side) NFS attribute +cache timeout for @i{Amd}'s own automount points. A value of 0 is +supposed to turn off attribute caching, meaning that @i{Amd} will be +consulted via a kernel-RPC each time someone stat()'s the mount point (which could be abused as a denial-of-service attack). -@emph{WARNING}: if you set this option to any non-zero value, -especially a large value, and you get ESTALE errors on your particular -OS, then set this value back to 0 seconds. + +@emph{WARNING}: @i{Amd} depends on being able to turn off the NFS +attribute cache of the client OS. If it cannot be turned off, then +users may get ESTALE errors or symlinks that point to the wrong +places; this is more likely under heavy use of @i{Amd}. Therefore, +under normal circumstances, this parameter should remain set to 0, to +ensure that the attribute cache is indeed off. + +Unfortunately, some kernels (e.g., certain BSDs) don't have a way to +turn off the NFS attribute cache. Setting this parameter to 0 is +supposed to turn off attribute caching entirely, but unfortunately it +does not; instead, the attribute cache is set to some internal +hard-coded default (usually anywhere from 5-30 seconds). If you +suspect that your OS doesn't have a reliable way of turning off the +attribute cache, then it is better to set this parameter to the +smallest possible non-zero value (set auto_attrcache=1 in your +@code{amd.conf}). This will not eliminate the problem, but reduce the +risk window somewhat. The best solutions are (1) to use Amd in Autofs +mode, if it's supported in your OS, and (2) talk to your OS vendor to +support a true @samp{noac} flag. See the +@uref{http://www.am-utils.org/docs/am-utils/attrcache.txt,README.attrcache} +document for more details. @c ---------------------------------------------------------------- @node auto_dir Parameter, cache_duration Parameter, auto_attrcache Parameter, Global Parameters diff --git a/scripts/amd.conf-sample b/scripts/amd.conf-sample index ae491c1..f753069 100644 --- a/scripts/amd.conf-sample +++ b/scripts/amd.conf-sample @@ -49,7 +49,9 @@ nfs_retry_interval_udp = 16 nfs_retransmit_counter_udp = 22 nfs_retry_interval_tcp = 4 nfs_retransmit_counter_tcp = 5 -# NFS attribute cache value for Amd's automount points (seconds) +# NFS attribute cache value for Amd's automount points +# (some OSs use this parameter in units of seconds, and others in 0.1 seconds) +# See README.attrcache for important information about this parameter. auto_attrcache = 0 # (amd -w) dismount_interval = 120 diff --git a/scripts/amd.conf.5 b/scripts/amd.conf.5 index e5d517f..5daceca 100644 --- a/scripts/amd.conf.5 +++ b/scripts/amd.conf.5 @@ -208,13 +208,14 @@ Amd variable. .TP .BR auto_attrcache " (numeric, default=0)" -Specify in seconds, the (kernel-side) NFS attribute cache timeout for Amd's -own automount points. A value of 0 turns off attribute caching, meaning -that Amd will be consulted via a kernel-RPC each time someone stat's the -mount point (which could be abused as a denial-of-service attack). Warning: -if you set this option to any non-zero value, especially a large value, and -you get ESTALE errors on your particular OS , then set this value back to 0 -seconds. +Specify in seconds (or units of 0.1 seconds, depending on the OS), what is +the (kernel-side) NFS attribute cache timeout for @i{Amd}'s own automount +points. A value of 0 is supposed to turn off attribute caching, meaning +that @i{Amd} will be consulted via a kernel-RPC each time someone stat()'s +the mount point (which could be abused as a denial-of-service attack). +Warning: some OSs are incapable of turning off the NFS attribute cache +reliably. On such systems, Amd may not work reliably under heavy load. See +the README.attrcache document in the Am-utils distribution for more details. .TP .BR auto_dir " (string, default=/a)"