kern/58901: nfs server should allow specifying fsid in exports(5)

[campbell+netbsd%mumble.net@localhost]

>Number:         58901
>Category:       kern
>Synopsis:       nfs server should allow specifying fsid in exports(5)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Dec 13 21:55:00 +0000 2024
>Originator:     Taylor R Campbell
>Release:        current, 10, 9, ...
>Organization:
The Nfsid Foundinode
>Environment:
>Description:
When exporting a local file system, the NFS server uses the fsid of the local file system.  This fsid is often assembled by boot-dependent quantities like the ordering of bus scanning or file system mounting:

   1523 	mp->mnt_stat.f_fsidx.__fsid_val[0] = (long)dev;
   1524 	mp->mnt_stat.f_fsidx.__fsid_val[1] = makefstype(MOUNT_FFS);
   1525 	mp->mnt_stat.f_fsid = mp->mnt_stat.f_fsidx.__fsid_val[0];

https://nxr.netbsd.org/xref/src/sys/ufs/ffs/ffs_vfsops.c?r=1.382#1523

    232 	mutex_enter(&mntid_lock);
    233 	if (xxxfs_mntid == 0)
    234 		++xxxfs_mntid;
    235 	mtype = makefstype(mp->mnt_op->vfs_name);
    236 	tfsid.__fsid_val[0] = makedev(mtype & 0xff, xxxfs_mntid);
    237 	tfsid.__fsid_val[1] = mtype;
    238 	/* Always increment to not return the same fsid to parallel mounts. */
    239 	xxxfs_mntid++;
    240 
    241 	/*
    242 	 * Directly walk mountlist to prevent deadlock through
    243 	 * mountlist_iterator_next() -> vfs_busy().
    244 	 */
    245 	mutex_enter(&mountlist_lock);
    246 	for (me = TAILQ_FIRST(&mountlist); me != TAILQ_END(&mountlist); ) {
    247 		if (me->me_type == ME_MOUNT &&
    248 		    me->me_mount->mnt_stat.f_fsidx.__fsid_val[0] ==
    249 		    tfsid.__fsid_val[0] &&
    250 		    me->me_mount->mnt_stat.f_fsidx.__fsid_val[1] ==
    251 		    tfsid.__fsid_val[1]) {
    252 			tfsid.__fsid_val[0]++;
    253 			xxxfs_mntid++;
    254 			me = TAILQ_FIRST(&mountlist);
    255 		} else {
    256 			me = TAILQ_NEXT(me, me_list);
    257 		}
    258 	}
    259 	mutex_exit(&mountlist_lock);
    260 
    261 	mp->mnt_stat.f_fsidx.__fsid_val[0] = tfsid.__fsid_val[0];
    262 	mp->mnt_stat.f_fsidx.__fsid_val[1] = tfsid.__fsid_val[1];
    263 	mp->mnt_stat.f_fsid = mp->mnt_stat.f_fsidx.__fsid_val[0];

https://nxr.netbsd.org/xref/src/sys/kern/vfs_mount.c#221

If anything changes this -- you move disks around or change partitions so the dkN numbers change, you edit /etc/fstab to mount file systems in a different order, or whatever -- all NFS file handles for what should be the `same' local file system will change.

Worse, if /export/A is mounted from /dev/dk3 and /export/B from /dev/dk4 at first, and then they are exchanged so /export/A is mounted from /dev/dk4 and /export/B is mounted from /dev/dk3, file handles could collide, causing the wrong data to be read or written by clients with cached file handles.

zfs avoids this by randomly generating its fsids and storing them in the file system, so they're independent of the device enumeration at boot and the number of times other file systems have been mounted.  But file systems that haven't reserved space in their metadata for this can't take advantage of the same approach.
>How-To-Repeat:
1. Mount a file system on /dev/dkN via named wedge and export it.
2. Mount a file system on /dev/dkM via named wedge and export it (M =/= N).
3. Start some clients caching some file handles.
4. Reboot and rejigger the hardware or partitioning so dkN and dkM are swapped but the same file systems are mounted at the same paths as before.
5. ???
6. Profit!
>Fix:
1. Create new syntax (or search other NFS server software for syntax) in /etc/exports for explicitly specifying the fsid of each exported file system.
2. Decide how to detect and address the entirely new problem of accidentally exporting an _actually different_ file system at the same path with the same fsid recorded in /etc/exports.