Re: fs-independent quotas

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Wed, Oct 19, 2011 at 06:09:27PM +0000, David Holland wrote:
> So, a few months back we got a new improved quota format for FFS.
> Unfortunately, one of the side effects of this was to sprinkle
> specific knowledge of the new format through all the userlevel quota
> tools and quota support logic. To be fair, this was alongside the
> existing specific knowledge of the old quota format; nonetheless, it's
> messy and unscalable.

of course there's been changes to the tools, as there's a new format.

> 
> We may want to add more quota formats (e.g. the different and
> incompatible new quota format FreeBSD added last year) or add quota
> support to other filesystems (tempfs, perhaps v7fs) or even add other
> filesystems that have or may have their own native quota handling
> (zfs, Hammer, you name it). Also, my planned lfs-renovation is
> currently hung up on the VFS-level quota interface, because I don't
> want to rip out the existing maybe-partial support for quotas but
> can't plug new code into the existing framework.

You'll have to explain this. lfs is some variant of ffs, I see no reasons
why it coudln't use the new format.

in fact the new format is fs-independant. 

> 
> For these and other reasons I've been intending to rework the quota
> system. As some people know I've also been avoiding doing it, because
> it looks like work; however, it really ought to go in before netbsd-6
> is branched and it's getting to the point where I feel like I'm
> starting to hold that up, and that's just not on.
> 
> So, here's what I'm planning to do.
> 
> It seems to me that quotas are fundamentally a special-purpose
> key/value store; that is, you look up quota information for a
> particular thing (the key) and get back the quota settings and current
> usage information (the value). This means, to me at least, that the
> quota system can and should be accessed like a key/value store; this
> means that a dump of the quota information can be tabular rather than
> hierarchically structured and there's therefore no need to involve
> proplib.

But this is just what the current propib format is ! a set of tables
with key/values pair !

> 
> I am going to change the exposed schema slightly from the traditional
> quota system. In the traditional quota system, there are two quota
> tables (which manifest physically as files), one for user quotas and
> one for group quotas. Within each table the id (uid or gid) is the
> key, and the value contains quota and usage information for both block
> and file quotas. This is simple and effective, but not very flexible;
> also, it's no longer the 80s and it's not necessary to link the
> physical representation closely to the logical representation.
> 
> Therefore, I'll do a bit of relational normalization and provide the
> following logical representation:
> 
>    - the quota key is:
>         the quota *class*
>         the id

Don't forget we now have a new id: "default"

>         the quota *type*
> 
>    - the quota value is:
>         the configured hard limit
>         the configured soft limit
>         the configured grace period
>         the current usage
>         the current grace expiry time (if any)

This is exactly the format described in quotactl(2).

> 
> The quota *class* is the thing the quota is imposed on; this is
> currently either "user" or "group". There is no likely prospect of
> additional quota classes appearing.

I don't think we should limit ourselve to these class. I could see
per-host or per-hostgroup quotas for networked filesystems for example.

> 
> The quota *type* is the thing the quota is about; this is currently
> either "blocks" or "files". There is, however, prior art out there
> (not in NetBSD though) that provides quotas for additional types.
> 
> Ideally as much of the code as possible should be written to be able
> to transparently handle additional quota classes and types; this may
> end up being more of a long-term goal than a direct consequence of the
> current work, though.
> 
> In this schema the quotas pertaining to a user "joe", with uid 101 and
> gid 100, might appear as:
> 
>    class  id    type    hard    soft    usage   grace   expire
>    ------------------------------------------------------------
>    user   101   block   11000   10000   5072    7d      -
>    user   101   file    1100    1000    280     7d      -
>    gid    100   block   22000   20000   11543   14d     -
>    gid    100   file    2200    2000    5072    14d     -
> 
> In the traditional quota implementation these four rows in the
> (filesystem-independent) logical schema fit into two struct dqblks,
> one holding the group data and one holding the user data. I believe
> the quota2 physical representation is similar.

the on-disk format has these fields, yes. But it's not a table, it's
a linked list.


> 
> (I have no plans to change any of the physical representations or the
> code that manages it.)
> 
> The current structural plan is for this logical schema to be exposed
> by each file system at the VFS layer. That is, each filesystem will be
> responsible for translating between its internal, on-disk format and
> the filesystem-independent logical schema. The VFS- and syscall-level
> kernel code should not need to do much of anything but hand off to the
> filesystem; this is more or less how things currently are and always
> have been.

This is what we have now: the logical schema is a proplib-based table;
and each filesystem translate it to its own format.
We can provide some helper functions to assist with the transforms,
this is what I started to do in quota2_subr.c. It looks ffs-specific but
is really close to what you're proposing here.


> 
> The userlevel quota library is going to be completely rewritten to
> provide a key/value access API to the logical schema described above.
> This will be converted to quotactl calls to the kernel... and also
> some other actions, such as contacting rquotad on NFS servers. There
> are also some cases with the old-style quotas where the tools access
> the quota files directly; some of these cases may go away, but I'm not
> sure they all can.

They can't if you want to keep some level of backward-compat.

> This logic and the FS-specific knowledge it
> requires can and should be contained inside libquota.

No, I don't think it has its place in libquota. libquota should only use
the fs-independant interface.
Right now, the places where the quota files are directly accessed are:
- repquota, mostly as a way to convert from quota1 to quota2 (it exports
  the content of the quota file to a plist that can be feed to quotactl).
- quotacheck (quota1-specific tool anyway).

this code has really no place in libquota.

> I'm also going to crib from FreeBSD's quota library and add libquota
> calls for things like turning quotas on and off. This should make the
> userlevel tools simpler, and should make life easier for any
> third-party tools that want to manipulate quotas. (There aren't many,
> but a few do exist.) Unfortunately, direct compat with FreeBSD's quota
> library isn't feasible as theirs is not FS-independent.

I don't think there should be userland calls to turn quota on or off.
We have it for quota1, but really nothing else should use it.
When you turn quota on from userland, you have to also provide the current
usage (like quotacheck does), and this is a FS-specific tool.

> 
> I expect the following tools to become FS-independent:
> 
>    quota(1)
>    quot(8)
>    edquota(8)

they already are.

>    quotaon(8)

This one is there for quota1, but really there should be no such tool any more.
quota management should be handled internally by the filesystem at mount
time.

>    repquota(8)
>    rpc.rquotad(8)

they already are fs-independant.

> 
> I'm also intending to add quotadump(8) and quotarestore(8) tools to
> allow backing up quota settings easily. With the traditional quota
> system you can just back up the quota files (and since they're exposed
> in the filesystem, this happens by default unless you explicitly
> exclude them) but with in-FS quotas that no longer works and a
> dump/restore method is needed. I think quotadump and quotarestore will
> probably end up as hard links to edquota, but that's not entirely
> clear yet.

We already have this: quotactl(8).

> 
> I'm going to remove the current quotactl(8) as it seems to be entirely
> specific to the current proplib-based interface.

One thing I had in mind with the proplib-based interface is to have an easy
way to deal with quota from scripts. What do you propose to remplace the
proplib-based interface ?

> 
> Note that quotacheck(8) is specific to the old-style FFS quotas and is
> not FS-independent; this will not (and cannot) change.
> 
> One remaining thing: I'm intending to systematize the current mess of
> quotas enabled/disabled/on/off/vanilla/chocolate/strawberry as
> follows:
> 
> 1. A file system type can have or not have support for quotas. If
> there is no support for quotas, nothing else works.
> 
> 2. Any given filesystem volume may have or not have quota data on it.
> This is the filesystem code's problem and irrelevant to the
> FS-independent logic.
> 
> 3. Any given filesystem volume may be mounted with or without quotas
> enabled. If quotas are not enabled, quota information is not available
> and the quota utilities will not be able to do anything.
> 
> 4. Once mounted, quotas can be either on or off. As far as the
> FS-independent code is concerned, quotas being off means only that
> they aren't enforced; that is, with quotas off operations that
> increase usage do not fail with EDQUOT. When quotas are off, quota
> information can still be inspected or updated.

What is the purpose of this ?

> 
> I am not intending to change the specific semantics that turning
> quotas on has the traditional quota system. Those semantics are
> required for quotacheck to be able to do its thing properly. However,
> knowledge of this behavior should be limited to the code in FFS (and
> probably some in libquota) that needs to know the gory details.
> 
> Currently there are, as far as I can tell, multiple ways to enable
> quotas for a filesystem in /etc/fstab, and the quota utilities check
> fstab in various (and I think not always consistent) ways to try to
> figure out what's going on. My intent is to nuke all that: only mount
> should care what's in /etc/fstab, because otherwise the tools won't
> work properly on temporary mounts. The quota library (and thus the
> tools) should detect whether a mounted filesystem has quotas enabled
> by calling quotactl; if quotactl fails, quotas are not enabled. (In
> the long run there should be a FS-independent mount flag to indicate
> this; however, I'm not sure we're ready for that just yet.)

No, in the new world only quotacheck and quotaon checks the fstab
to know where quotas should be checked/enabled and where the quota
file is. These are quota1-specific. I think they should be left as-is
until quota1 support is removed.

> 
> It is not specified whether a filesystem mounted with quotas enabled
> comes up with quotas turned on or off. The traditional system requires
> "off", of course, but I think the default for new code that doesn't
> require quotacheck should be "on".

It is "on" right now for quota2.

> 
> The following is a sketch of the intended libquota API:
> 
>    #include <quota.h>
> 
>    struct quotavolume; /* Opaque. */
>    struct quotacursor; /* Opaque. */
> 
>    struct quotakey {
>            unsigned qk_class;
>            id_t qk_id;
>            unsigned qk_type;
>    };
>    struct quotaval {
>            uint64_t qv_hardlimit;
>            uint64_t qv_softlimit;
>            uint64_t qv_usage;
>          time_t qv_grace;
>          time_t qv_expire;
>    };
> 
>    #define QUOTA_CLASS_USER   0
>    #define QUOTA_CLASS_GROUP  1
> 
>    #define QUOTA_TYPE_BLOCKS  0
>    #define QUOTA_TYPE_FILES   1
> 
>    #define QUOTA_DEFAULTID    ((id_t)-1)

-1 can also be a uid or gid, isn't it ?

> 
>    #define QUOTA_NOLIMIT      ((uint64_t)0xffffffffffffffff)
> 
>    #define QUOTA_INFINITEGRACE        ((time_t)-1)
>    #define QUOTA_NOGRACE      ((time_t)0)
> 
>    struct quotavolume *quota_open(const char *volume_path);
>    void quota_close(struct quotavolume *);
> 
>    const char *quota_getschemaname(struct quotavolume *);
> 
>    unsigned quota_getnumclasses(struct quotavolume *);
>    const char *quota_getclassname(struct quotavolume *, unsigned class);
> 
>    unsigned quota_getnumtypes(struct quotavolume *);
>    const char *quota_gettypename(struct quotavolume *, unsigned type);
> 
>    int quota_on(struct quotavolume *);
>    int quota_off(struct quotavolume *);
> 
>    int quota_get(struct quotavolume *, const struct quotakey *key,
>                struct quotaval *val_ret);
> 
>    int quota_put(struct quotavolume *, const struct quotakey *key,
>                const struct quotaval *val);
> 
>    int quota_delete(struct quotavolume *, const struct quotakey *key);
> 
>    struct quotacursor *quota_opencursor(struct quotavolume *);
>    void quotacursor_close(struct quotacursor *);
> 
>    int quotacursor_get(struct quotacursor *qc, struct quotakey *key_ret,
>                      struct quotaval *val_ret);
> 
>    int quotacursor_getn(struct quotacursor *qc, struct quotakey *keys_ret,
>                       struct quotaval *vals_ret, int maxnum);
> 
>    bool quotacursor_atend(struct quotacursor *);
>    int quotacursor_rewind(struct quotacursor *);


What interface do you plan between kernel and userland ? keep the
proplib-based interface ?

All of what you propose can be fully implemented with the
current proplib interface and its schema, so it looks like you're proposing
to rework libquota.

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--