Lua in NetBSD

[Marc Balmer <marc%msys.ch@localhost>]

About one week ago, I suggested to the NetBSD community to add the Lua  
language to NetBSD as a language for scripting applications and as a  
general scripting language.

My original post is here: 
http://marc.info/?l=netbsd-tech-userlevel&m=125577239909997&w=2

I am overwehelmed by the number of positive reactions I got to my  
proposal, it looks like I have been suggesting something that has been  
waited for by many.  While I suggested to import Lua into base, much  
to my surprise it was even suggested to use it in the kernel, an idea  
I like the longer I think of it.

There was some criticism, too, especially since I did not state what I  
am going use Lua for, if it was in base;  I wanted to discuss the  
general case first, so I was vague in this regard on purpose.

I am currently working on a scheme to interface all kind of external  
reference clocks to the system; not to get a timing base for  
timecounters, but to get absolute time information. This is something  
different.  For timecounters you need a regular pulse, but a radio  
clock might only give the correct time once a day.  Maybe some people  
know about my work in OpenBSD in this area.  My current work builds on  
top of that previous work, but provides better abstractions and better  
interface (and thus more usuable information).

One family of external reference clocks provides timing information  
using bytestream protocols, the NMEA 0183 protocol used by most GPS  
receivers is one of the best known protocols that falls into this  
category. The decoding of the NMEA protocol has been done using a tty  
line discipline.

Using line disciplines for this kind of decoding comes at a price:   
Line disciplines are a vehicle not to many programmers are familar  
with (I could be wrong here, this is just an assumption), they are  
somewhat hard to get right and to maintain.  It means to do a lot of  
string processing in the kernel, god beware if we have a serious,  
undetected bug in that code...  But the main problem is that there are  
dozens of such protocols, most GPS vendors have their own proprietary  
binary protocol, Meinberg has it's own serial protocol, endrun comes  
to my mind and so on.  So for every protocol we need a tty line  
discipline, need to define a name for the line discipline, have a  
program that can attach the line discipline to an open filedescriptor  
and so on (for the latter I wrote ldattach(8) in OpenBSD).  One hast  
to touch many parts of the system to support a new protocol with a  
line discipline.

Time sources are kernel entities (drivers) that provide time  
information in a uniform way.  They need to be drivers, since many  
clocks are attached to a PCI bus or use USB etc.  To avoid the line  
discipline mess, I have a timesource that can be fed the time  
externally.  So there is a daemon that decodes the time information  
and feeds it to the system.  And this is were Lua comes into play.   
The decoders for the protocols are written in Lua, and new decoders  
can thus be added with ease and without requiring a kernel/userland  
rebuild.  Besides of that I use Lua to write software with graphical  
user interfaces using X11/Xt and OpenMotif and to provide access to  
system functionality like sysctls, gpio etc. from Lua.

Lua, being a small language that interfaces exceptionally well with C  
programs, could be of good use in various other places, during the  
discussion, the following uses have been proposed or mentioned

- extending DHCP clients for options processing
- extending the bootloader
- extending sysinst
- uses in pkgsrc
- scripting nvi
- providing access to NetBSD from Lua (though for this, the Lua in  
pkgsrc is just fine)
- general scriptiing
- maybe stuff that I have forgotten to list...

Thor proposed to add it to the kernel as well.  He would be willing to  
devote time and manpower to this project, so would I, and Lourival  
Vieira Neto from PUC Rio (where Lua is being maintained) offered to  
help, since he does a similar project in Linux ("Lunatik").  Having  
Lua in the kernel would actually allow to use NetBSD in total novel  
ways or for new fields of application:

- rapid prototyping of device drivers, or device drivers in Lua
- experimentation with new network protocols
- kauth(9) security models in Lua
- controlling the hardware environment (cpu freq etc) based on  
evironmental measurements (the Lunatik project at PUC Rio goes  
somewhat in this direction it seems)
- accept filters
- filter rules

Of course some implementation issues with Lua in the kernel remain to  
be solved:  The number format.  How to integrate it in the kernel.   
Should the compiler be included or only the bytecode interpreter  
(which means Lua scripts would be compiled to bytecode in userland and  
the bytecode then be loaded into the kernel), etc.

A quick recap of why Lua is a language well suited for this kind of  
applications:

- small memory footprint
- fast execution speed
- superb integration with C
- code can be prcompiled to bytecode for faster loading
- easy to learn with a "natural" syntax, not statically typed
- extensible in useful ways using the metatables mechanism
- very good data handling capabilities
- no bloated libraries

Quite some developers told me in private that they would use Lua for  
their work, if it was in the system and I think this all speaks for  
the addition of Lua to NetBSD.  If there are not to many objections to  
this, I will ask core to give me a thumbs-up to initiate two  
subprojects:

- Adding Lua to NetBSD userland
- Adding Lua to the NetBSD kernel

Thanks for your time && let's do it!

- Marc Balmer