OPNsense Forum

Archive => 15.7 Legacy Series => Topic started by: ledoktre on August 02, 2015, 02:29:02 am

Title: [ABANDONED] Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 02, 2015, 02:29:02 am: Greetings all -

My first post on the forum. Wanted to start out by saying that aside from the issues I've had, things are working perfectly ;)

I have OPNsense loaded on a virtual machine on my file server (KVM). I have 3 nic's in the server, and so I have two of the nics in bridge mode (separate bridges) with the adapters in the router. Internet is working great. Surfing, streaming music / videos etc all working great.

I do also use the captive portal (what got me going on this project in the first place away from DDWRT on a Linksys router). enabled on my LAN adapter only. That has had a few hiccups but by far and far its working GREAT.

So now thats the background. Here's the issue.

I have the servers and several of my main workstations (password protected) whitelisted in the captive portal, either by mac or by ip. I have an ATA (for VoIP) also whitelisted - I cannot get it to connect to my hosted Asterisk box (UDP port 5060). It tries and tries but never logs in. Asterisk doesn't even register an attempt.

Pinging through the router does not work either - timeout. I can do DNS lookups with an external DNS from a client machine - but just cannot ping them.

Cannot make Steam games work either. Ive been told the login is TCP but all the game play is UDP. I can login to Steam but cannot bring up an online game.

One I just discovered tonight is that I cannot do an NTP lookup from a client machine to 0.us.pool.ntp.org. If I plug that into the OPN box, and tell my client machine to use the router for NTP - works fine.

I am not sure whats going on - surfing, port forwarding etc all work fine. But am just having some weirdnesses happening.

Need some help!!

Thanks,
Title: Re: Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 04, 2015, 11:09:37 pm: Bump.
Title: Re: Having a weird issue with primarily UDP traffic
Post by: franco on August 05, 2015, 05:43:15 am: Possibly the quietest time of the year being vacation time for most people. In any case welcome and thanks. If it's not too much to ask any hiccup report can help make OPNsense better. Don't just accept them, at least getting a discussion going may make things clearer or end up in a bug fix. :)

Do you see blocked UDP traffic when you filter for it in the firewall log? Need to see if the issue is pf or IPFW in your case.
Title: Re: Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 05, 2015, 06:19:42 am: Hello,

Thanks for replying. I did poke around in the firewall logs but either somethings not working quite right or I am just not up to speed how to control the stuff because when I just look at the 60 second snapshot, I can't find anything. If I try to filter it by source IP, etc, then it just sits there and spins - can't get it to bring any records up.

?

Thanks,
Title: Re: Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 05, 2015, 06:21:34 am: It may not be just UDP traffic - I can use an outisde DNS server and do a lookup (UDP port 53 as I recall?) but I cannot ping anything outside to save my life.

Today I was using a BitDefender live cd to scan a computer, and I authorized through the CP, could surf, but could not get it to update the definition files. That ought to be TCP Im sure. So were they just coincidentally down or too busy? Perhaps.
Title: Re: Having a weird issue with primarily UDP traffic
Post by: franco on August 05, 2015, 09:32:08 am: Will this also stall (see screenshot)? It may be a problem with the source filter in particular, although not too likely.
Title: Re: Having a weird issue with primarily UDP traffic
Post by: AdSchellevis on August 05, 2015, 11:07:15 am: @ledoktre could you try to login to the firewall and manually disable all ipfw rules? (used by captive portal).

you can do that by running this in the shell:

Code: [Select]
ipfw flush
(and answer y )

If the problem doesn't exist then anymore, please reboot and dump the output from ipfw using:

Code: [Select]
ipfw -t show
Maybe the generated rules show something odd.
Title: Re: Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 07, 2015, 06:02:56 am: I had a chance to try the suggestions that you fellas posted. This one in particular - when I flushed the ipfw, everything appeared fine. In fact, the phone was already registered by the time I logged into its interface. I thought I may have to reboot it but no - it took right off.

I am in the process of trying to get it to reboot (it is not rebooting for some reason) and will post back an ipfw dump.

Thanks - starting to feel better :)
Title: Re: Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 07, 2015, 07:12:09 am: Here is the dump - does anything stand out?

00100 0 0 allow pfsync from any to any
00110 0 0 allow carp from any to any
00120 0 0 allow ip from any to any layer2 mac-type 0x0806,0x8035
00130 0 0 allow ip from any to any layer2 mac-type 0x888e,0x88c7
00140 0 0 allow ip from any to any layer2 mac-type 0x8863,0x8864
00150 0 0 deny ip from any to any layer2 not mac-type 0x0800,0x86dd
00200 0 0 skipto 60000 ip6 from ::1 to any
00201 0 0 skipto 60000 ip4 from 127.0.0.0/8 to any
00202 0 0 skipto 60000 ip6 from any to ::1
00203 0 0 skipto 60000 ip4 from any to 127.0.0.0/8
01001 281 28044 Fri Aug 7 00:06:12 2015 skipto 60000 udp from any to 10.24.72.254 dst-port 53 keep-state
01001 200 87490 Fri Aug 7 00:06:14 2015 skipto 60000 ip from any to { 255.255.255.255 or 10.24.72.254 } in
01001 23 11668 Fri Aug 7 00:06:13 2015 skipto 60000 ip from { 255.255.255.255 or 10.24.72.254 } to any out
01001 0 0 skipto 60000 icmp from { 255.255.255.255 or 10.24.72.254 } to any out icmptypes 0
01001 0 0 skipto 60000 icmp from any to { 255.255.255.255 or 10.24.72.254 } in icmptypes 8
03021 0 0 skipto 12001 ip from table(7) to any via vtnet1
03022 0 0 skipto 12001 ip from table(7) to any via vtnet1
03023 1586 526243 Fri Aug 7 00:06:15 2015 skipto 12001 ip from table(9) to any via vtnet1
03024 0 0 skipto 12001 ip from table(9) to any via vtnet1
03025 16376 18009415 Fri Aug 7 00:06:15 2015 skipto 12001 ip from table(11) to any via vtnet1
03026 0 0 skipto 12001 ip from table(11) to any via vtnet1
05002 19 1700 Fri Aug 7 00:06:13 2015 fwd 127.0.0.1,8002 tcp from any to any dst-port 80 in via vtnet1
05002 0 0 allow ip from any to any dst-port 80 via vtnet1
06002 29877 21075833 Fri Aug 7 00:06:15 2015 skipto 60000 ip from any to any via vtnet0
06200 10049 1947902 Fri Aug 7 00:06:15 2015 allow tcp from any to any out
06201 849 117811 Fri Aug 7 00:06:14 2015 skipto 65534 ip from any to any
12001 17960 18535546 Fri Aug 7 00:06:15 2015 count ip from any to any via vtnet1
12998 17960 18535546 Fri Aug 7 00:06:15 2015 skipto 30000 ip from any to any via vtnet1
12999 0 0 deny ip from any to any not via vtnet1
30000 17960 18535546 Fri Aug 7 00:06:15 2015 count ip from any to any
60000 0 0 return ip from any to any
65533 48341 39738581 Fri Aug 7 00:06:15 2015 allow ip from any to any
65534 849 117811 Fri Aug 7 00:06:14 2015 deny ip from any to any
65535 79 80726 Fri Aug 7 00:01:50 2015 allow ip from any to any
Title: Re: Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 07, 2015, 07:14:07 am: I ran it again to be sure I did it right and here's my second (same?) results :

00100 0 0 allow pfsync from any to any
00110 0 0 allow carp from any to any
00120 0 0 allow ip from any to any layer2 mac-type 0x0806,0x8035
00130 0 0 allow ip from any to any layer2 mac-type 0x888e,0x88c7
00140 0 0 allow ip from any to any layer2 mac-type 0x8863,0x8864
00150 0 0 deny ip from any to any layer2 not mac-type 0x0800,0x86dd
00200 0 0 skipto 60000 ip6 from ::1 to any
00201 0 0 skipto 60000 ip4 from 127.0.0.0/8 to any
00202 0 0 skipto 60000 ip6 from any to ::1
00203 0 0 skipto 60000 ip4 from any to 127.0.0.0/8
01001 383 36672 Fri Aug 7 00:12:26 2015 skipto 60000 udp from any to 10.24.72.254 dst-port 53 keep-state
01001 523 224331 Fri Aug 7 00:12:58 2015 skipto 60000 ip from any to { 255.255.255.255 or 10.24.72.254 } in
01001 72 27269 Fri Aug 7 00:11:17 2015 skipto 60000 ip from { 255.255.255.255 or 10.24.72.254 } to any out
01001 0 0 skipto 60000 icmp from { 255.255.255.255 or 10.24.72.254 } to any out icmptypes 0
01001 0 0 skipto 60000 icmp from any to { 255.255.255.255 or 10.24.72.254 } in icmptypes 8
03021 0 0 skipto 12001 ip from table(7) to any via vtnet1
03022 0 0 skipto 12001 ip from table(7) to any via vtnet1
03023 5852 2887570 Fri Aug 7 00:12:59 2015 skipto 12001 ip from table(9) to any via vtnet1
03024 0 0 skipto 12001 ip from table(9) to any via vtnet1
03025 90246 112437060 Fri Aug 7 00:12:59 2015 skipto 12001 ip from table(11) to any via vtnet1
03026 0 0 skipto 12001 ip from table(11) to any via vtnet1
05002 29 2516 Fri Aug 7 00:08:37 2015 fwd 127.0.0.1,8002 tcp from any to any dst-port 80 in via vtnet1
05002 0 0 allow ip from any to any dst-port 80 via vtnet1
06002 148318 120239801 Fri Aug 7 00:12:59 2015 skipto 60000 ip from any to any via vtnet0
06200 48865 3949703 Fri Aug 7 00:12:59 2015 allow tcp from any to any out
06201 1622 231261 Fri Aug 7 00:12:59 2015 skipto 65534 ip from any to any
12001 96096 115324518 Fri Aug 7 00:12:59 2015 count ip from any to any via vtnet1
12998 96096 115324518 Fri Aug 7 00:12:59 2015 skipto 30000 ip from any to any via vtnet1
12999 0 0 deny ip from any to any not via vtnet1
30000 96096 115324518 Fri Aug 7 00:12:59 2015 count ip from any to any
60000 0 0 return ip from any to any
65533 245392 235852591 Fri Aug 7 00:12:59 2015 allow ip from any to any
65534 1622 231261 Fri Aug 7 00:12:59 2015 deny ip from any to any
65535 79 80726 Fri Aug 7 00:01:50 2015 allow ip from any to any
Title: Re: Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 07, 2015, 07:15:09 am: vnet1 = lan (10.24.72.254)
vnet0 = wan (static IP)

BTW :)

Main issues that Ive ran across :

-Bitdefender live cd wont update its definition files
-can't ping outside the network (to test connectivity, latency or packet loss, for example)
-can't use ntp unless its using the ntp built into OPNsense
-can't connect to Steam (someone told me that the gameplay is UDP) - well the logon doesn't work most of the time either. 98% probably.
-Can't get VoIP phone to register wit hosted Asterisk - UDP 5060

Hopefully someone can look at the ruleset above and help decipher what might be going on. I'd appreciate it. To the best of my knowledge this is a stock ipfw config save a few things punched through the firewall for services to access from outside.

Thank you!
Title: Re: Having a weird issue with primarily UDP traffic
Post by: AdSchellevis on August 07, 2015, 08:40:43 am: The rules look quite normal, when you are logged in via CP, the flow looks like this :

[#3023/3025] -> [#12001] -> [#12998] -> [#30000] -> allow al ip [#65533]

Strange, can you supply some additional information:

* the ip address of your client
* the output of both:

ipfw table 9 list
ipfw table 11 list

* the virtual network driver user? ( some seem to have issues )

* and another simple test,

ipfw add 06200 allow ip from any to any

(and check again)
Title: Re: Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 07, 2015, 08:45:25 am: When I add the one that you gave me, phone etc works fine. I am not very familiar with ipfw, bsd, or OPNsense, so can you tell me what we just did (I know its obvious but humor me), is it safe, and if so, how to make it permanent?

Wonder how is best to proceed.

Thanks,

Edit : just saw you edited - i will do what you ask and repost - thanks
Title: Re: Having a weird issue with primarily UDP traffic
Post by: AdSchellevis on August 07, 2015, 08:56:45 am: You shouldn't add it permanent, it's a rule to allow all that isn't matched on the CP rules..
The big question is, why doesn't it allow your traffic if you should be logged-in?

(I pressed <enter> way to soon :) )
Title: Re: Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 07, 2015, 09:24:44 am: The two commands you wanted me to run :

root@router:/home/ledoktre # ipfw table 9 list
10.24.72.2/32 999
10.24.72.12/32 999
10.24.72.201/32 999
10.24.72.202/32 999
10.24.72.203/32 999
10.24.72.207/32 999
10.24.72.208/32 999
10.24.72.209/32 999
10.24.72.210/32 999

root@router:/home/ledoktre # ipfw table 11 list
10.24.72.23/32 999
10.24.72.151/32 999
10.24.72.172/32 999
10.24.72.198/32 999

The router is 10.24.72.254, the phone was 10.24.72.12, the desktop that I tested from was 10.24.72.23.

I setup this VM using virtualbox - I would have used KVM or something like that, but went with virtualbox because this particular machine does not have any hardware extensions for virtualization. Virtualbox is running under root at the moment (I know, I know.. I am still setting it up..)

To clarify, that last one you requested I add everything worked fine after that.

I have quite a few IP addresses and MAC addresses (a combination) entered into the whitelist on the captive portal. Also, before I jumped on here, I did try disabling the captive portal both by unchecking it and by also clikcing the stop button. I didnt reboot or reload any ruleset, but I still had connectivity issues then.

Thanks - hope we can chat back and forth a bit and get this. Otherwise its late here and Im gonna crash. :)

Latest ipfw show (might have changed a bit - enabled ssh access from outside as I am not at the location of this firewall atm):

00100 0 0 allow pfsync from any to any
00110 0 0 allow carp from any to any
00120 0 0 allow ip from any to any layer2 mac-type 0x0806,0x8035
00130 0 0 allow ip from any to any layer2 mac-type 0x888e,0x88c7
00140 0 0 allow ip from any to any layer2 mac-type 0x8863,0x8864
00150 0 0 deny ip from any to any layer2 not mac-type 0x0800,0x86dd
00200 0 0 skipto 60000 ip6 from ::1 to any
00201 0 0 skipto 60000 ip4 from 127.0.0.0/8 to any
00202 0 0 skipto 60000 ip6 from any to ::1
00203 0 0 skipto 60000 ip4 from any to 127.0.0.0/8
01001 816 79026 Fri Aug 7 02:25:50 2015 skipto 60000 udp from any to 10.24.72.254 dst-port 53 keep-state
01001 1481 635776 Fri Aug 7 02:26:10 2015 skipto 60000 ip from any to { 255.255.255.255 or 10.24.72.254 } in
01001 189 94095 Fri Aug 7 02:25:48 2015 skipto 60000 ip from { 255.255.255.255 or 10.24.72.254 } to any out
01001 0 0 skipto 60000 icmp from { 255.255.255.255 or 10.24.72.254 } to any out icmptypes 0
01001 0 0 skipto 60000 icmp from any to { 255.255.255.255 or 10.24.72.254 } in icmptypes 8
03021 0 0 skipto 12001 ip from table(7) to any via vtnet1
03022 0 0 skipto 12001 ip from table(7) to any via vtnet1
03023 7343 1119770 Fri Aug 7 02:26:12 2015 skipto 12001 ip from table(9) to any via vtnet1
03024 0 0 skipto 12001 ip from table(9) to any via vtnet1
03025 4840 1163763 Fri Aug 7 02:26:12 2015 skipto 12001 ip from table(11) to any via vtnet1
03026 0 0 skipto 12001 ip from table(11) to any via vtnet1
05002 544 41757 Fri Aug 7 02:25:48 2015 fwd 127.0.0.1,8002 tcp from any to any dst-port 80 in via vtnet1
05002 0 0 allow ip from any to any dst-port 80 via vtnet1
06002 33780 8195933 Fri Aug 7 02:26:12 2015 skipto 60000 ip from any to any via vtnet0
06200 8894 1888332 Fri Aug 7 02:26:12 2015 allow tcp from any to any out
06201 6871 1213275 Fri Aug 7 02:26:12 2015 skipto 65534 ip from any to any
12001 12183 2283533 Fri Aug 7 02:26:12 2015 count ip from any to any via vtnet1
12998 12183 2283533 Fri Aug 7 02:26:12 2015 skipto 30000 ip from any to any via vtnet1
12999 0 0 deny ip from any to any not via vtnet1
30000 12183 2283533 Fri Aug 7 02:26:12 2015 count ip from any to any
60000 0 0 return ip from any to any
65533 48449 11288363 Fri Aug 7 02:26:12 2015 allow ip from any to any
65534 6871 1213275 Fri Aug 7 02:26:12 2015 deny ip from any to any
65535 52 7985 Fri Aug 7 01:54:28 2015 allow ip from any to any
Title: Re: Having a weird issue with primarily UDP traffic
Post by: AdSchellevis on August 07, 2015, 10:55:26 am: I hope I'm still on time, time difference always is a challenge.

Any chance the ip addresses changed in the mean time? maybe something goes wrong with the arp sync...

Could you try to reload the captive portal and add the full range in one of the tables to see what happens then?

you can do that with :

Code: [Select]
ipfw table 9 add 10.24.72.2/24

and the try again, make sure the manual rule is gone (the reload should do that), because that opens up all.
Title: Re: Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 08, 2015, 05:42:13 am: Greetings,

I wonder what the chance is I'll find you online right now. I'd really like to get this figured out. Yeah, I did miss you last night. I waited around for a while and just couldn't hold my eyes open any longer, sorry.

So in answer to your questions-

1. IP address changed in the mean time? I am not sure which address you are referring to, but all the important stuff is setup either as sticky dynamic or static (phone, server, desktop I tested from, etc).

2. Could you try to reload the captive portal and add the full range in one of the tables to see what happens then? Yeah, I'll give that a try now. Actually funny that you are mentioning the captive portal. I found that this morning the captive portal was not obeying the whitelisted mac and ip addresses. My users had to login with their usernames- very odd.

3. Manual rule? Ive rebooted a few times since then and my phone quit so I assumed that rule should be cone? UPDATE- that rule was still there after reboots. I ended up deleting it manually.. ???

I am going to go ahead and probably reboot the router, try to output its config, try to figure out some of the rules for myself. I'll look for that one rule you referred to (3) above too and try that code you wanted me to try.

Thanks- will post back soon.
Title: Re: Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 08, 2015, 08:44:17 pm: Ok, so a couple things.

1. I just figured out that you are one of the top dawgs. Cool to be conversing with someone so close. Thanks for helping me.

2. I read somewhere that pfSense had some issue with using mac addresses (which I would prefer for security purposes) in their captive portal, so I moved all mine over to static dhcp leases and put them on the captive portal whitelist. All good so far. Only problem is, NOTHING can surf.

So I look at each table (I thought I had previously figured out table 7 was for CP logged in users, table 9 was those on a IP whitelist and table 11 was those on a mac whitelist). 7 was there as I had 2 users logged in by username. But 9 and 11 were both empty (and no net connectivity on anything on the inside except for those logged in with username). How could that be? The IP list had a lot of them in there.

3. I added the line for 10.24.72.0/24 like you'd mentioned and I started seeing connectivity again. I haven't tried testing the original core problem yet, but what would cause the GUI to be correct but ipfw to NOT be correct (as in completely empty table 9)?

Thanks,
Title: Re: Having a weird issue with primarily UDP traffic
Post by: AdSchellevis on August 09, 2015, 01:32:49 pm: Your welcome :)

The captive portal feature is scheduled for a full replacement in the next major version 16.1, the version pfSense had relied very heavily on all kinds of non standard kernel patches we didn't want in there.
We had to do something for this version to keep CP working without breaking too much, so our implementation is already very different then the one in pfSense.

You where right with your assessment on the tables, but for future reference I will try to clarify all a bit more.

Let me first try to explain where the tables are used for in addition to what you already found out, hopefully we could eventually find your issue or find ourselves a way around them.

*   The first group: authenticated users
In your case table 7, if there are more zone's the next will be 13 ( a gap of 6 for every zone).
When a user isn't authenticated yet and tries to connect, it will be forwarded to the login page via the forward rule.
( fwd 127.0.0.1,8002 tcp from any to any dst-port 80 in via em0 )
Which is serviced by an instance of "lighttpd".

*   The second group: authenticated hosts
In your case table 9, the next zone will use the same offset of six.
A listing of this group should be an exact copy of the "Allowed IP addresses"
It's odd that this is empty in your last enquiry, we will have to look at this. (I will give some hints later on).

*   The third group: authenticated mac addresses
In your case table 11, next zone same story about the offsets.
This group is a bit trickier, one of the kernel patches of pfSense added an option to combine both ip and mac addresses in ipfw when an ip packet is matched, this is something that's not in ipfw and probably will never be in there. (different layers of the OSI model are combined here)
In our case this table also contains ip addresses, just like the second group. The only difference is that it uses the firewall's arp table to determine what ip address should be valid based on the mac address.
So we need a network packet before we can do the allow, there might be a little lag.
A process in userspace should do the administration for this.

The periodic administration of both changing mac addresses and expired users should be handled by /usr/local/etc/rc.prunecaptiveportal, but when looking at the code I'm not sure this handles all correct now.
This could be one of the issues you have.
To confirm this you could go to the "ip addresses" page, edit an item without changing anything press save and look at the contents of table 9 again.
When this is populated, we might have caught your issue.

Because not all information is available in the ipfw table, there's an additional administration in a sqlite database.
In that database, all extra information is stored. This one is stored under /var/db/captiveportal*.db.
If the first apply didn't work, please remove all captiveportal*.db files and try that same apply again.
In case this fixes the issue, we know we have a cleaning issue after boot.

I hope this clarifies some of it and help fix your issue, although I'm aware that there's still a lot of work to do before this component is really as solid as we would like it to be.
(That's why it's on the roadmap for 16.1)

Cheers,

Ad
Title: Re: Having a weird issue with primarily UDP traffic
Post by: ledoktre on August 10, 2015, 06:56:29 pm: Thanks for the information. I was able to confirm what you said was correct, and since I can't wait around for 16.1, I opted to *try* pfSense out and see if it was the same or if it worked. It turns out, it worked just fine. I'm gonna keep OPNsense on the back burner until I see the 16.1 come out. It really has a lot nicer interface than pf, but at the end of the day, its working with captive portal as expected.

TY much for all your help and I'll def keep an eye on development.

Thanks,