[ABANDONED] Having a weird issue with primarily UDP traffic

[ledoktre]

Greetings all -

My first post on the forum.  Wanted to start out by saying that aside from the issues I've had, things are working perfectly  ;)

I have OPNsense loaded on a virtual machine on my file server (KVM).  I have 3 nic's in the server, and so I have two of the nics in bridge mode (separate bridges) with the adapters in the router.  Internet is working great.  Surfing, streaming music / videos etc all working great.

I do also use the captive portal (what got me going on this project in the first place away from DDWRT on a Linksys router).  enabled on my LAN adapter only.  That has had a few hiccups but by far and far its working GREAT.

So now thats the background.  Here's the issue.

I have the servers and several of my main workstations (password protected) whitelisted in the captive portal, either by mac or by ip.  I have an ATA (for VoIP) also whitelisted - I cannot get it to connect to my hosted Asterisk box (UDP port 5060).  It tries and tries but never logs in.  Asterisk doesn't even register an attempt.

Pinging through the router does not work either - timeout.  I can do DNS lookups with an external DNS from a client machine - but just cannot ping them. 

Cannot make Steam games work either.  Ive been told the login is TCP but all the game play is UDP.  I can login to Steam but cannot bring up an online game.

One I just discovered tonight is that I cannot do an NTP lookup from a client machine to 0.us.pool.ntp.org.  If I plug that into the OPN box, and tell my client machine to use the router for NTP - works fine.

I am not sure whats going on - surfing, port forwarding etc all work fine.  But am just having some weirdnesses happening.

Need some help!!

Thanks,

[ledoktre]

Bump.

[franco]

Possibly the quietest time of the year being vacation time for most people. In any case welcome and thanks. If it's not too much to ask any hiccup report can help make OPNsense better. Don't just accept them, at least getting a discussion going may make things clearer or end up in a bug fix. :)

Do you see blocked UDP traffic when you filter for it in the firewall log? Need to see if the issue is pf or IPFW in your case.

[ledoktre]

Hello,

Thanks for replying.  I did poke around in the firewall logs but either somethings not working quite right or I am just not up to speed how to control the stuff because when I just look at the 60 second snapshot, I can't find anything.  If I try to filter it by source IP, etc, then it just sits there and spins - can't get it to bring any records up.

?

Thanks,

[ledoktre]

It may not be just UDP traffic - I can use an outisde DNS server and do a lookup (UDP port 53 as I recall?) but I cannot ping anything outside to save my life.

Today I was using a BitDefender live cd to scan a computer, and I authorized through the CP, could surf, but could not get it to update the definition files.  That ought to be TCP Im sure.  So were they just coincidentally down or too busy?  Perhaps. 

[franco]

Will this also stall (see screenshot)? It may be a problem with the source filter in particular, although not too likely.

[AdSchellevis]

@ledoktre could you try to login to the firewall and manually disable all ipfw rules? (used by captive portal).

you can do that by running this in the shell:

Code Select Expand

ipfw flush

(and answer y )

If the problem doesn't exist then anymore, please reboot and dump  the output from ipfw using:

Code Select Expand

ipfw -t show

Maybe the generated rules show something odd.

[ledoktre]

I had a chance to try the suggestions that you fellas posted.  This one in particular - when I flushed the ipfw, everything appeared fine.  In fact, the phone was already registered by the time I logged into its interface.  I thought I may have to reboot it but no - it took right off.

I am in the process of trying to get it to reboot (it is not rebooting for some reason) and will post back an ipfw dump.

Thanks - starting to feel better :)

[ledoktre]

Here is the dump - does anything stand out?

00100     0        0                         allow pfsync from any to any
00110     0        0                         allow carp from any to any
00120     0        0                         allow ip from any to any layer2 mac-type 0x0806,0x8035
00130     0        0                         allow ip from any to any layer2 mac-type 0x888e,0x88c7
00140     0        0                         allow ip from any to any layer2 mac-type 0x8863,0x8864
00150     0        0                         deny ip from any to any layer2 not mac-type 0x0800,0x86dd
00200     0        0                         skipto 60000 ip6 from ::1 to any
00201     0        0                         skipto 60000 ip4 from 127.0.0.0/8 to any
00202     0        0                         skipto 60000 ip6 from any to ::1
00203     0        0                         skipto 60000 ip4 from any to 127.0.0.0/8
01001   281    28044 Fri Aug  7 00:06:12 2015 skipto 60000 udp from any to 10.24.72.254 dst-port 53 keep-state
01001   200    87490 Fri Aug  7 00:06:14 2015 skipto 60000 ip from any to { 255.255.255.255 or 10.24.72.254 } in
01001    23    11668 Fri Aug  7 00:06:13 2015 skipto 60000 ip from { 255.255.255.255 or 10.24.72.254 } to any out
01001     0        0                         skipto 60000 icmp from { 255.255.255.255 or 10.24.72.254 } to any out icmptypes 0
01001     0        0                         skipto 60000 icmp from any to { 255.255.255.255 or 10.24.72.254 } in icmptypes 8
03021     0        0                         skipto 12001 ip from table(7) to any via vtnet1
03022     0        0                         skipto 12001 ip from table(7) to any via vtnet1
03023  1586   526243 Fri Aug  7 00:06:15 2015 skipto 12001 ip from table(9) to any via vtnet1
03024     0        0                         skipto 12001 ip from table(9) to any via vtnet1
03025 16376 18009415 Fri Aug  7 00:06:15 2015 skipto 12001 ip from table(11) to any via vtnet1
03026     0        0                         skipto 12001 ip from table(11) to any via vtnet1
05002    19     1700 Fri Aug  7 00:06:13 2015 fwd 127.0.0.1,8002 tcp from any to any dst-port 80 in via vtnet1
05002     0        0                         allow ip from any to any dst-port 80 via vtnet1
06002 29877 21075833 Fri Aug  7 00:06:15 2015 skipto 60000 ip from any to any via vtnet0
06200 10049  1947902 Fri Aug  7 00:06:15 2015 allow tcp from any to any out
06201   849   117811 Fri Aug  7 00:06:14 2015 skipto 65534 ip from any to any
12001 17960 18535546 Fri Aug  7 00:06:15 2015 count ip from any to any via vtnet1
12998 17960 18535546 Fri Aug  7 00:06:15 2015 skipto 30000 ip from any to any via vtnet1
12999     0        0                         deny ip from any to any not via vtnet1
30000 17960 18535546 Fri Aug  7 00:06:15 2015 count ip from any to any
60000     0        0                         return ip from any to any
65533 48341 39738581 Fri Aug  7 00:06:15 2015 allow ip from any to any
65534   849   117811 Fri Aug  7 00:06:14 2015 deny ip from any to any
65535    79    80726 Fri Aug  7 00:01:50 2015 allow ip from any to any

[ledoktre]

I ran it again to be sure I did it right and here's my second (same?) results :

00100      0         0                         allow pfsync from any to any
00110      0         0                         allow carp from any to any
00120      0         0                         allow ip from any to any layer2 mac-type 0x0806,0x8035
00130      0         0                         allow ip from any to any layer2 mac-type 0x888e,0x88c7
00140      0         0                         allow ip from any to any layer2 mac-type 0x8863,0x8864
00150      0         0                         deny ip from any to any layer2 not mac-type 0x0800,0x86dd
00200      0         0                         skipto 60000 ip6 from ::1 to any
00201      0         0                         skipto 60000 ip4 from 127.0.0.0/8 to any
00202      0         0                         skipto 60000 ip6 from any to ::1
00203      0         0                         skipto 60000 ip4 from any to 127.0.0.0/8
01001    383     36672 Fri Aug  7 00:12:26 2015 skipto 60000 udp from any to 10.24.72.254 dst-port 53 keep-state
01001    523    224331 Fri Aug  7 00:12:58 2015 skipto 60000 ip from any to { 255.255.255.255 or 10.24.72.254 } in
01001     72     27269 Fri Aug  7 00:11:17 2015 skipto 60000 ip from { 255.255.255.255 or 10.24.72.254 } to any out
01001      0         0                         skipto 60000 icmp from { 255.255.255.255 or 10.24.72.254 } to any out icmptypes 0
01001      0         0                         skipto 60000 icmp from any to { 255.255.255.255 or 10.24.72.254 } in icmptypes 8
03021      0         0                         skipto 12001 ip from table(7) to any via vtnet1
03022      0         0                         skipto 12001 ip from table(7) to any via vtnet1
03023   5852   2887570 Fri Aug  7 00:12:59 2015 skipto 12001 ip from table(9) to any via vtnet1
03024      0         0                         skipto 12001 ip from table(9) to any via vtnet1
03025  90246 112437060 Fri Aug  7 00:12:59 2015 skipto 12001 ip from table(11) to any via vtnet1
03026      0         0                         skipto 12001 ip from table(11) to any via vtnet1
05002     29      2516 Fri Aug  7 00:08:37 2015 fwd 127.0.0.1,8002 tcp from any to any dst-port 80 in via vtnet1
05002      0         0                         allow ip from any to any dst-port 80 via vtnet1
06002 148318 120239801 Fri Aug  7 00:12:59 2015 skipto 60000 ip from any to any via vtnet0
06200  48865   3949703 Fri Aug  7 00:12:59 2015 allow tcp from any to any out
06201   1622    231261 Fri Aug  7 00:12:59 2015 skipto 65534 ip from any to any
12001  96096 115324518 Fri Aug  7 00:12:59 2015 count ip from any to any via vtnet1
12998  96096 115324518 Fri Aug  7 00:12:59 2015 skipto 30000 ip from any to any via vtnet1
12999      0         0                         deny ip from any to any not via vtnet1
30000  96096 115324518 Fri Aug  7 00:12:59 2015 count ip from any to any
60000      0         0                         return ip from any to any
65533 245392 235852591 Fri Aug  7 00:12:59 2015 allow ip from any to any
65534   1622    231261 Fri Aug  7 00:12:59 2015 deny ip from any to any
65535     79     80726 Fri Aug  7 00:01:50 2015 allow ip from any to any

[ledoktre]

vnet1 = lan (10.24.72.254)
vnet0 = wan (static IP)

BTW :)

Main issues that Ive ran across :

-Bitdefender live cd wont update its definition files
-can't ping outside the network (to test connectivity, latency or packet loss, for example)
-can't use ntp unless its using the ntp built into OPNsense
-can't connect to Steam (someone told me that the gameplay is UDP) - well the logon doesn't work most of the time either.  98% probably.
-Can't get VoIP phone to register wit hosted Asterisk - UDP 5060

Hopefully someone can look at the ruleset above and help decipher what might be going on.  I'd appreciate it.  To the best of my knowledge this is a stock ipfw config save a few things punched through the firewall for services to access from outside.

Thank you!

[AdSchellevis]

The rules look quite normal, when you are logged in via CP, the flow looks like this :

[#3023/3025] -> [#12001] -> [#12998] -> [#30000] -> allow al ip [#65533]

Strange, can you supply some additional information:

* the ip address of your client
* the output of both:

ipfw table 9 list
ipfw table 11 list

* the virtual network driver user? ( some seem to have issues )

* and another simple test,

ipfw add 06200 allow ip from any to any

(and check again)

[ledoktre]

When I add the one that you gave me, phone etc works fine.  I am not very familiar with ipfw, bsd, or OPNsense, so can you tell me what we just did (I know its obvious but humor me), is it safe, and if so, how to make it permanent?

Wonder how is best to proceed.

Thanks,

Edit : just saw you edited - i will do what you ask and repost - thanks

[AdSchellevis]

You shouldn't add it permanent, it's a rule to allow all that isn't matched on the CP rules..
The big question is, why doesn't it allow your traffic if you should be logged-in?

(I pressed <enter> way to soon  :) )

[ledoktre]

The two commands you wanted me to run :

root@router:/home/ledoktre # ipfw table 9 list
10.24.72.2/32 999
10.24.72.12/32 999
10.24.72.201/32 999
10.24.72.202/32 999
10.24.72.203/32 999
10.24.72.207/32 999
10.24.72.208/32 999
10.24.72.209/32 999
10.24.72.210/32 999

root@router:/home/ledoktre # ipfw table 11 list
10.24.72.23/32 999
10.24.72.151/32 999
10.24.72.172/32 999
10.24.72.198/32 999

The router is 10.24.72.254, the phone was 10.24.72.12, the desktop that I tested from was 10.24.72.23.

I setup this VM using virtualbox - I would have used KVM or something like that, but went with virtualbox because this particular machine does not have any hardware extensions for virtualization.  Virtualbox is running under root at the moment (I know, I know.. I am still setting it up..)

To clarify, that last one you requested I add everything worked fine after that.

I have quite a few IP addresses and MAC addresses (a combination) entered into the whitelist on the captive portal.  Also, before I jumped on here, I did try disabling the captive portal both by unchecking it and by also clikcing the stop button.  I didnt reboot or reload any ruleset, but I still had connectivity issues then.

Thanks - hope we can chat back and forth a bit and get this.  Otherwise its late here and Im gonna crash. :)

Latest ipfw show (might have changed a bit - enabled ssh access from outside as I am not at the location of this firewall atm):

00100     0        0                         allow pfsync from any to any
00110     0        0                         allow carp from any to any
00120     0        0                         allow ip from any to any layer2 mac-type 0x0806,0x8035
00130     0        0                         allow ip from any to any layer2 mac-type 0x888e,0x88c7
00140     0        0                         allow ip from any to any layer2 mac-type 0x8863,0x8864
00150     0        0                         deny ip from any to any layer2 not mac-type 0x0800,0x86dd
00200     0        0                         skipto 60000 ip6 from ::1 to any
00201     0        0                         skipto 60000 ip4 from 127.0.0.0/8 to any
00202     0        0                         skipto 60000 ip6 from any to ::1
00203     0        0                         skipto 60000 ip4 from any to 127.0.0.0/8
01001   816    79026 Fri Aug  7 02:25:50 2015 skipto 60000 udp from any to 10.24.72.254 dst-port 53 keep-state
01001  1481   635776 Fri Aug  7 02:26:10 2015 skipto 60000 ip from any to { 255.255.255.255 or 10.24.72.254 } in
01001   189    94095 Fri Aug  7 02:25:48 2015 skipto 60000 ip from { 255.255.255.255 or 10.24.72.254 } to any out
01001     0        0                         skipto 60000 icmp from { 255.255.255.255 or 10.24.72.254 } to any out icmptypes 0
01001     0        0                         skipto 60000 icmp from any to { 255.255.255.255 or 10.24.72.254 } in icmptypes 8
03021     0        0                         skipto 12001 ip from table(7) to any via vtnet1
03022     0        0                         skipto 12001 ip from table(7) to any via vtnet1
03023  7343  1119770 Fri Aug  7 02:26:12 2015 skipto 12001 ip from table(9) to any via vtnet1
03024     0        0                         skipto 12001 ip from table(9) to any via vtnet1
03025  4840  1163763 Fri Aug  7 02:26:12 2015 skipto 12001 ip from table(11) to any via vtnet1
03026     0        0                         skipto 12001 ip from table(11) to any via vtnet1
05002   544    41757 Fri Aug  7 02:25:48 2015 fwd 127.0.0.1,8002 tcp from any to any dst-port 80 in via vtnet1
05002     0        0                         allow ip from any to any dst-port 80 via vtnet1
06002 33780  8195933 Fri Aug  7 02:26:12 2015 skipto 60000 ip from any to any via vtnet0
06200  8894  1888332 Fri Aug  7 02:26:12 2015 allow tcp from any to any out
06201  6871  1213275 Fri Aug  7 02:26:12 2015 skipto 65534 ip from any to any
12001 12183  2283533 Fri Aug  7 02:26:12 2015 count ip from any to any via vtnet1
12998 12183  2283533 Fri Aug  7 02:26:12 2015 skipto 30000 ip from any to any via vtnet1
12999     0        0                         deny ip from any to any not via vtnet1
30000 12183  2283533 Fri Aug  7 02:26:12 2015 count ip from any to any
60000     0        0                         return ip from any to any
65533 48449 11288363 Fri Aug  7 02:26:12 2015 allow ip from any to any
65534  6871  1213275 Fri Aug  7 02:26:12 2015 deny ip from any to any
65535    52     7985 Fri Aug  7 01:54:28 2015 allow ip from any to any