Re: kern/55663 (Support for EVFILT_USER in kqueue(2))

[Ruslan Nikolaev <nruslan_devel%yahoo.com@localhost>]

The following reply was made to PR kern/55663; it has been noted by GNATS.

From: Ruslan Nikolaev <nruslan_devel%yahoo.com@localhost>
To: gnats-bugs%netbsd.org@localhost, jnemeth%netbsd.org@localhost, kern-bug-people%netbsd.org@localhost,
 netbsd-bugs%netbsd.org@localhost, gnats-admin%netbsd.org@localhost
Cc: 
Subject: Re: kern/55663 (Support for EVFILT_USER in kqueue(2))
Date: Tue, 20 Oct 2020 17:21:29 -0400

 [I am re-sending the email, the previous message was corrupted by the 
 email client ]
 
 I greatly apologize for the delay. Recently, I had so many things to do...
 Please keep in mind that my understanding may be incomplete since I have 
 not really introduced this feature, I just ported it from 
 FreeBSD/OpenBSD. In fact, I have far more experience with epoll() rather 
 than kqueue(). So, let me start the discussion with the motivation 
 first, and then we can go to specific technical details (in a separate 
 email which I will try to prepare next as needed).
 
 1. Why is it needed?
 
 Quite a few applications use it. To be specific, please consider Nginx. 
 Although, it *does* compile under NetBSD, several critical features are 
 missing without EVFILT_USER.
 
 Take a look at 
 https://nginx.googlesource.com/nginx/+/refs/heads/master/src/core/ngx_thread_pool.c:
 
 static ngx_int_t
 ngx_thread_pool_init_worker(ngx_cycle_t *cycle)
 {
 ...
  Â Â Â  if (ngx_notify_init == NULL || ngx_notify == NULL) {
  Â Â Â Â Â Â Â  ngx_log_error(NGX_LOG_ALERT, cycle->log, 0,
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â  "the configured event method cannot be used with thread 
 pools");
  Â Â Â Â Â Â Â  return NGX_ERROR;
  Â Â Â  }
 
 ...
 
 
 Note that if ngx_notify is *NOT* initialized to some function, Nginx 
 will not support the thread pool mode, which is arguably a very 
 important Nginx feature that boosts performance greatly!
 
 So, where is this function initialized? For kqueue, Nginx provides 
 https://nginx.googlesource.com/nginx/+/refs/heads/master/src/event/modules/ngx_kqueue_module.c
 
 Note these lines:
 static ngx_event_module_t  ngx_kqueue_module_ctx = {
 ...
          NULL,                              /* delete an connection */
 #ifdef EVFILT_USER
          ngx_kqueue_notify_init,            /* init a notify */
          ngx_kqueue_notify,                 /* trigger a notify */
          NULL,                              /* close a notify */
 #else
          NULL,                              /* init a notify */
          NULL,                              /* trigger a notify */
          NULL,                              /* close a notify */
 #endif
 ...
 
 So, in other words, Nginx compiles and runs but lacks the thread pool 
 support, which is an important feature. I am not saying that they could 
 not have implemented it differently, but quite a few applications just 
 assume you have EVFILT_USER and do not provide any good fallback.
 If we can just implement EVFILT_USER as in FreeBSD (regardless of its 
 merits), that would take care of those cases completely.
 
 As another example, see the bug report for mySQL: 
 http://gnats.netbsd.org/54627
 Without EVFILT_USER, it will even have uninitialized data
 
 2. Regarding the EVFILT_USER itself, the FreeBSD man page says the 
 following:
 
 EVFILT_USERÂ Â Â Â Â  Establishes a user event identified by ident which is
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  not associated with any kernel mechanism but is trig-
                   gered by user level code.  The lower 24 bits of the
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  fflags may be used for user defined flags and manipu-
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  lated using the following:
 
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  NOTE_FFNOPÂ Â Â Â Â Â Â Â Â  Ignore the input fflags.
 
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  NOTE_FFANDÂ Â Â Â Â Â Â Â Â  Bitwise AND fflags.
 
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  NOTE_FFORÂ Â Â Â Â Â Â Â Â Â  Bitwise OR fflags.
 
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  NOTE_FFCOPYÂ Â Â Â Â Â Â Â  Copy fflags.
 
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  NOTE_FFCTRLMASKÂ Â Â Â  Control mask for fflags.
 
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  NOTE_FFLAGSMASKÂ Â Â Â  User defined flag mask for
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  fflags.
 
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  A user event is triggered for output with the follow-
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  ing:
 
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  NOTE_TRIGGERÂ Â Â Â Â Â Â  Cause the event to be triggered.
 
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  On return, fflags contains the users defined flags in
  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  the lower 24 bits.
 
 Please let me know if EVFILT_USER may sound interesting for NetBSD. If 
 so, I can try to recollect technical details (it has been really a 
 while). But for the most part it was based on OpenBSD's patch (with some 
 adaptations for NetBSD): 
 https://www.mail-archive.com/tech%openbsd.org@localhost/msg42433.html
 
 
 Thanks,
 Ruslan