writev() to /dev/bpf from different sockets block

[Jason Curl <jcurl%arcor.de@localhost>]

Hello,

I've found an odd behaviour I can't explain on NetBSD 10.1 on the RPi4 
(image downloaded from NetBSD site). If I have two threads that write a 
Layer 2 ETH packet to "/dev/bpf" with writev(), the two threads 
eventually block and never return. The SSH connection eventually drops 
too. Only about 10-20 packets went over the wire and captured on another 
machine.

The pseudo code, which is implementing a stress test and trying to 
measure the network performance based on different techniques to send data:

fd = open("/dev/bpf", O_RDWR);
if (fd == -1) {
  perror("open");
  return;
}

strlcpy(&ifr.ifr_name[0], "genet0", IFNAMSIZ);
if (ioctl(fd, BIOCSETIF, &ifr) == -1) {
  perror("ioctl(BIOCSETIF)");
  return;
}

unsigned int hdr_complete = 1;
if (ioctl(fd, BIOCSHDRCMPLT, &hdr_complete) == -1) {
  perror("ioctl(BIOCSHDRCMPLT");
  return;
}

for(;;) {
  struct iovec *msg = get_packet(); // Creates a UDPv4 frame
  if (writev(fd, msg, 2) == -1) {
    if (errno == ENOBUFS) {
      delay(750000);
      continue;
    }
    perror("writev");
    return;
  }
}

This seems to be NetBSD specific, not observed on FreeBSD, QNX 7.1 
(based on old NetBSD with their fixes).

Secondary problem: I couldn't find an effective way to handle ENOBUFS, 
other than with a 750us delay (couldn't find how to handle this properly 
to know when I should resend data).

Any hints on what I'm doing wrong is appreciated.

Regards,
Jason.