GSoC project: Userspace file system and device driver code sharing

[Vyacheslav Matyushin <v.matyushin%gmail.com@localhost>]

Hi! My name is Vyacheslav Matyushin, and here is my proposal
for the project "Userspace file system and device driver code sharing".

http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/vmatyushin/1

About the project
-----------------
NetBSD has a puffs - Pass-to-Userspace Framework File System
development interface,
which provides the support for implementing file servers in userspace.
The example of the implemented file server which uses puffs is psshfs.

Puffs has it's own userspace library - libpuffs,
upon which the file servers are supposed to be built.
It provides:
    - file system routines;
    - callback interface;
    - buffering and event handling suitable for network file systems;
    - kernel request handling;
    - cooperative multitasking.
See the "Appendix" section for rough analysis of the libpuffs source code.

NetBSD also has a pud - Pass-to-Userspace Device, which provides support
for implementing character and block device servers in userspace.
However the pud doesn't have it's userspace library, so in order to
implement a device server, one must interact directly with the kernel
by reading and writing raw request/response structures.

The goal of the project is to move generic parts from libpuffs to
the separate library and provide libpuffs and libpud
based on this library.

Deliverables of the project
---------------------------
The deliverables include:
    - The library, which will be a common code base for libpuffs, libpud
      and other possible pass-to-userspace frameworks
      (it's name is still to be figured out).
      It will consist of:
        - cooperative multitasking mechanism, allowing to switch
          the execution context between the file or device server and
          the library;
        - kernel request and response handling mechanism;
        - buffering and event handling mechanism
          (still a subject of investigation though).

    - libpud - a userspace library for pud, which will allow to implement
      device servers in as convenient way as libpuffs does.
      It will take care of pud initialization and, if the device
driver writer wants,
      automatically handling the requests by using callbacks;

    - Documentation for code base library;

    - Updated documentation for libpuffs;

    - Documentation and examples for libpud.

Timeline
--------
April 11 - May 22
Continue figuring out contents of future libpuffs and libpud common code base.
Discussing the code base library and libpud API.
Possible early coding start.

May 23 - June 12
Implement libpuffs and libpud code base library.
Document it.

June 13 - July 3
Modify libpuffs to be based on that library.
Update libpuffs documentation.
Update libpuffs examples and test them.

July 4 - July 15 - Midterm evaluation deadline.
Buffering zone for the previous two stages.

July 16 - July 31
Write libpud using the code base library created previously.
Rewrite the pud's "intro" example using it.

August 1 - August 22
Write the examples and documentation for libpud.
Final testing.

About me
--------
I'm a 4th year student of Bryansk State Technical University (BSTU).
My BSD programming experience is:
    - http://code.google.com/p/osx-bcm5906m/
      Writing BCM5906M Fast Ethernet Adapter driver for Mac OS X
      using the FreeBSD and NetBSD's bge(4) (though NetBSD version
      is a slightly modified FreeBSD's one).

    - reading through NetBSD's ext2fs driver.
      My diploma's project theme is "Implementing Ext3 driver for NetBSD"
      and there is still much to learn,  for example get familiar with
      the WAPBL implementation, but I hope that working with
      puffs/pud code will raise my BSD programming skills
      and prepare for writing the diploma.

Appendix
--------
Here is the rough analysis of libpuffs source code:
    callcontext.c - continuation routines

    dispatcher.c - analyses the request fetched from /dev/puffs,
                   calls the appropriate file server operation and
                   writes back the result.

    framebuf.c - buffering and event handling.

    requests.c - functions used by framectrl to read and write data
        from and to "upstream provider", /dev/puffs.
        Each puffs vfs and vnode operation deliveres the request to
        /dev/puffs using putter.
        Then it is taken out by libpuffs's framectrl using
        puffs__fsframe_read, processed in the dispatcher.c functions
        and the response is written back by puffs__fsframe_write.

    flush.c, null.c, opdump.c, paths.c, pnode.c, subr.c, suspend.c,
    all headers - puffs stuff, file system related routines.

    puffs.c - among puffs routines contains a couple of notable things:
              1) initialization of fs operation vector passed from file server;
              2) event loop.

    The puffs workloop is:
        - return to each operation which we escaped previously
          using puffs_cc_yield
        - write out pending data
        - check file descriptors for the read and write possibility using kevent
          and call puffs__framev_input and puffs__framev_output respectively.

    For now pud is used like this:
        - either read the pud request, process and write the response
in one loop iteration
          (like in the "intro" pud example or "umserv" rump example)
        - or read the pud request and launch the thread function
          which will act like the dispatcher, processing the request
          and writing back the responce ("ttyserv" rump example).

    But in libpuffs the puffs requests are read and written
    using the framebuf mechanism, and the appropriate file server
    operation is called by the dispatching mechanism.

    So, the possible work scheme of libpud is:
        - define functions to process each kind of pud request,
          like PUD_CDEV_OPEN, PUD_CDEV_IOCTL and others.
        - fill the operations vector with them and pass to pud_init().
          with the other init info like desirable device name,
major/minor and flags.
        - than do some other yet unknown initialization stuff and call
pud_mainloop().

        - the pud_mainloop will act like libpuffs' one,
          processing read and write events.
        - the framebuf can be used to control reading and writing
          of pud requests from and to /dev/pud
          and calling the dispatching function, like it is used in libpuffs.

    So, the common code base for libpuffs and libpud is:
        callcontext.c, framebuf.c, requests.c, part of the dispatcher.c
        (everything except the big "dispatch" function
         which is puffs-specific)
        and part of the puffs.c (theloop() seems to be useful).
    The framebuf mechanism however is targeted to network interaction
    so it's relevance to pud is an open question.

Thanks,
Vyacheslav.