diff --git a/Kernel-Issues-&-Bugs.md b/Kernel-Issues-&-Bugs.md index 10ab77b..0e718f0 100644 --- a/Kernel-Issues-&-Bugs.md +++ b/Kernel-Issues-&-Bugs.md @@ -2,14 +2,17 @@ There have been a number of kernel issues / bugs over the years which mergerfs h ## NFS and EIO errors -Over the years some users have reported that while exporting mergerfs via NFS, after significant filesystem activity, not only will the NFS client start returning ESTALE and EIO errors but mergerfs itself would start returning EIO errors. The problem was that no one could reliability reproduce the issue. After a string of reports in late 2023 and early 2024 more investigation was done. While it is unclear why numerous users started reporting the problem the proximate cause was discovered after gaining enough information from users to create a reliable reproduction of the problem. +Over the years some users have reported that while exporting mergerfs via NFS, after significant filesystem activity, not only will the NFS client start returning ESTALE and EIO errors but mergerfs itself would start returning EIO errors. The problem was that no one could reliability reproduce the issue. After a string of reports in late 2023 and early 2024 more investigation was done. -As of 2024-02-19 the specifics of the issue are yet known but what happens is that the kernel flushes some cached dentries and when queried later FUSE sends a `lookup(nodeid=FUSE_ROOT_ID, name="..")` request to the server (mergerfs). This probably shouldn't be happening as there is no parent to root. No matter what mergerfs responds with it results in the same situation where the kernel then issues a `forget(nodeid=1)` and all future interaction with the mount results in EIO errors. +In Linux 5.14 new validation was put into FUSE which caught a few invalid situations and would tag a FUSE node as invalid if a check failed. Such checks include invalid file type, changing of type from one request to another, a size greater than 63bit, and the generation of a inode changing while in use. -A workaround, ironically, is disabling FUSE's EXPORT_SUPPORT feature. This flag tells FUSE that the server can understand requests for "." and ".." which (as I understand) is used by FUSE to work around the fact that NFS deals in file handles but FUSE does not. It seems that at least some NFS server/client combinations will work even when this is disabled and the kernel returns ESTALE for certain requests as a reult. +What happened was that mergerfs was using a different fixed, non-zero value for the generation of all nodes as it was suggested that unique inode + generation pairs are needed for proper integration with NFS. That non-zero value was being sent back to the kernel when a lookup request was made for root. The reason this was hard to track down was because NFS almost uniquely uses an API which can lead to a lookup of the root node that simply won't happen under normal workloads and usage. And that lookup will only happen if child nodes of the root were forgotten but NFS still had a handle to that node and later asked for details about it. It would trigger a set of requests to lookup info on those nodes. -This section will be updated as new information is found. +This wasn't a bug in FUSE but mergerfs. However, the incorrect behavior of mergerfs lead to FUSE behave in an unexpected and incorrect manner. It would issue a lookup of the "parent of a child of the root" and mergerfs would send the invalid generation value. As a result the kernel would mark the root node as "bad" which would then trigger the kernel to issue a "forget root" message. In between those it would issue a request for the parent of the root... which doesn't exist. +So the kernel was doing two invalid things. Requesting the parent of the root and then when that failed issuing a forget for the root. These led to chasing after the wrong possible causes. + +How best to handle these edge cases are still being discussed as of 2024-02-25 but mergerfs has been fixed to ensure generation=0 when root lookups are requested. Nodes generally shouldn't be marked as bad so it might be good to have the kernel log when it happens for better debugging. ## Truncated files