kern/53962: npf: weird 'stateful' behavior

[fstd.lkml%gmail.com@localhost]

>Number:         53962
>Category:       kern
>Synopsis:       npf: weird 'stateful' behavior
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Feb 09 00:35:00 +0000 2019
>Originator:     Timo Buhrmester
>Release:        8.0_STABLE as of 2019-01-26
>Organization:
>Environment:
NetBSD lemon.pr0.tips 8.0_STABLE NetBSD 8.0_STABLE (LEMONKERND) #0: Sat Jan 26 15:22:56 CET 2019  build%kiwi.pr0.tips@localhost:/stor/netbsd/foreign/lemon-apu/obj/sys/arch/amd64/compile/LEMONKERND amd64
>Description:
I've been attempting to migrate from ipf to npf and I'm having trouble understanding how 'stateful' is supposed to work.


In a test-case sort of scenario, I'm having a NetBSD router connected to two networks:
wm1: 192.168.1.1/24
wm2: 192.168.2.1/24

I want to allow the network behind wm1 to SSH to the network behind wm2.


This is what I initially tried, it's how things worked in ipf:

npf.conf:
| group "net1" on wm1 {
|         pass stateful in final proto tcp from 192.168.1.0/24 to 192.168.2.0/24 port 22 apply "log"
|         block all apply "log"
| }
| 
| group "net2" on wm2 {
|         block all apply "log"
| }
| 
| group default {
|         pass final on lo0 all
|         block all apply "log"
| }

$ nc 192.168.2.13 22  #run on 192.168.1.14

npflog0:
| 00:36:03.085493 rule 5.rules.0/0(match): pass in on wm1: 192.168.1.14.42714 > 192.168.2.13.22: Flags [S], seq 453874606, win 29200, options [mss 1460,sackOK,TS val 561118807 ecr 0,nop,wscale 7], length 0
| 00:36:03.085524 rule 8.rules.0/0(match): block out on wm2: 192.168.1.14.42714 > 192.168.2.13.22: Flags [S], seq 453874606, win 29200, options [mss 1460,sackOK,TS val 561118807 ecr 0,nop,wscale 7], length 0

My SYN doesn't make it out of wm2, so apparently I need a rule for that as well.

Therefore my next attempt:

npf.conf:
| procedure "log" {
|         log: npflog0
| }
| 
| group "net1" on wm1 {
|         pass stateful in final proto tcp from 192.168.1.0/24 to 192.168.2.0/24 port 22 apply "log"
|         block all apply "log"
| }
| 
| group "net2" on wm2 {
|         pass stateful out final proto tcp from 192.168.1.0/24 to 192.168.2.13 port 22 apply "log"
|         block all apply "log"
| }
| 
| group default {
|         pass final on lo0 all
|         block all apply "log"
| }

$ nc 192.168.2.13 22  #run on 192.168.1.14

npflog0:
| 00:36:52.006564 rule 5.rules.0/0(match): pass in on wm1: 192.168.1.14.42718 > 192.168.2.13.22: Flags [S], seq 2274723816, win 29200, options [mss 1460,sackOK,TS val 561131037 ecr 0,nop,wscale 7], length 0
| 00:36:52.006639 rule 8.rules.0/0(match): pass out on wm2: 192.168.1.14.42718 > 192.168.2.13.22: Flags [S], seq 2274723816, win 29200, options [mss 1460,sackOK,TS val 561131037 ecr 0,nop,wscale 7], length 0
| 00:36:52.007299 rule 9.rules.0/0(match): block in on wm2: 192.168.2.13.22 > 192.168.1.14.42718: Flags [S.], seq 283128975, ack 2274723817, win 28960, options [mss 1460,sackOK,TS val 4155176782 ecr 561131037,nop,wscale 6], length 0

Now the router is blocking the SYN/ACK -- why?  The rule was 'stateful' and therefore state should've been kept, no?

Anyway, let's see what happens when the wm1 rule is no longer stateful:

npf.conf:
| procedure "log" {
|         log: npflog0
| }
| 
| group "net1" on wm1 {
|         pass in final proto tcp from 192.168.1.0/24 to 192.168.2.0/24 port 22 apply "log"
|         block all apply "log"
| }
| 
| group "net2" on wm2 {
|         pass stateful out final proto tcp from 192.168.1.0/24 to 192.168.2.13 port 22 apply "log"
|         block all apply "log"
| }
| 
| group default {
|         pass final on lo0 all
|         block all apply "log"
| }

$ nc 192.168.2.13 22  #run on 192.168.1.14

npflog0:
| 00:39:00.969658 rule 5.rules.0/0(match): pass in on wm1: 192.168.1.14.42732 > 192.168.2.13.22: Flags [S], seq 1423577834, win 29200, options [mss 1460,sackOK,TS val 561163278 ecr 0,nop,wscale 7], length 0
| 00:39:00.969758 rule 8.rules.0/0(match): pass out on wm2: 192.168.1.14.42732 > 192.168.2.13.22: Flags [S], seq 1423577834, win 29200, options [mss 1460,sackOK,TS val 561163278 ecr 0,nop,wscale 7], length 0
| 00:39:00.970346 rule 8.rules.0/0(match): pass out on wm2: 192.168.2.13.22 > 192.168.1.14.42732: Flags [S.], seq 1701442932, ack 1423577835, win 28960, options [mss 1460,sackOK,TS val 4155305749 ecr 561163278,nop,wscale 6], length 0
| 00:39:00.970379 rule 6.rules.0/0(match): block out on wm1: 192.168.2.13.22 > 192.168.1.14.42732: Flags [S.], seq 1701442932, ack 1423577835, win 28960, options [mss 1460,sackOK,TS val 4155305749 ecr 561163278,nop,wscale 6], length 0

Now the SYN/ACK is passed OUT on wm2?  This seems fishy, that's the same direction we sent the SYN.  It totally should've been "in" there.  Is this a bug in npf's logging?

(mlelstv explained on IRC: "I am pretty sure that the direction in the log is not about the packet but the rule so "pass out on wm2" is actually the stateful handling of the "pass out" rule for the incoming packet."  so I guess this concern can be disregarded)

Anyway assuming 'out' was actually 'in' on the 3rd line, apparently now state is kept for the (sole) 'stateful' rule.  But I have no state on the wm1 rule anymore, so this still doesn't get through.


Out of curiosity, the last case: wm1 rule is 'stateful', wm2 rule isn't:

npf.conf:
| procedure "log" {
|         log: npflog0
| }
| 
| group "net1" on wm1 {
|         pass stateful in final proto tcp from 192.168.1.0/24 to 192.168.2.0/24 port 22 apply "log"
|         block all apply "log"
| }
| 
| group "net2" on wm2 {
|         pass out final proto tcp from 192.168.1.0/24 to 192.168.2.13 port 22 apply "log"
|         block all apply "log"
| }
| 
| group default {
|         pass final on lo0 all
|         block all apply "log"
| }

$ nc 192.168.2.13 22  #run on 192.168.1.14

npflog0:
| 00:40:43.323638 rule 5.rules.0/0(match): pass in on wm1: 192.168.1.14.42738 > 192.168.2.13.22: Flags [S], seq 1102658105, win 29200, options [mss 1460,sackOK,TS val 561188866 ecr 0,nop,wscale 7], length 0
| 00:40:43.323669 rule 8.rules.0/0(match): pass out on wm2: 192.168.1.14.42738 > 192.168.2.13.22: Flags [S], seq 1102658105, win 29200, options [mss 1460,sackOK,TS val 561188866 ecr 0,nop,wscale 7], length 0
| 00:40:43.324247 rule 9.rules.0/0(match): block in on wm2: 192.168.2.13.22 > 192.168.1.14.42738: Flags [S.], seq 1712581557, ack 1102658106, win 28960, options [mss 1460,sackOK,TS val 4155408105 ecr 561188866,nop,wscale 6], length 0

I never learn whether state was kept on the wm1 rule since it's blocked on wm2 ingress (as it should as far as my understanding of what 'stateful' is supposed to do goes).


So what's the wizardry here?

>How-To-Repeat:

>Fix: