Messages

Is there any way to capture packets on OPNsense for multiple interfaces simultaneously, rather than resorting to command line?

TCPdump can select multiple interfaces, but why can we not select multiple interfaces when taking a trace? I'm troubleshooting an issue where I apparently have asynchronous routing on one subnet and it would be far more helpful to capture the two interfaces I suspect rather than one which misses half the data I'm looking for.

EDIT: Also, how do I know if the traffic I'm seeing is the traffic ingress or egress for the VLAN captured?

NetScaler has a cool packet capture format which lets you see the VLAN it came in/went out on, if the packet was Rx or Tx, and more. Sure makes reading their traces easier compared to other network devices.

EDIT 2: What? I attempted to capture VLAN 1 and VLAN 99, simultaneously, using 2 tabs. The traces are identical. Did it lose the 1st capture when I tried to start the 2nd? If so, how do I capture these two interfaces at the same time? :( Do I have to resort to CLI? If so, please consider this my feature request.

LOL nevermind. I figured out my problem as soon as I posted this.

Web Proxying occurs before outbound NAT, and the test subnet was set to use the proxy.

This happened on 17.7 and now also on 18.7.6. I am using the OPNsense as an internal firewall, with 6 interfaces, where one interface is a transit Subnet from the OPNsense firewall to the external firewall. OPNsense uses the external Firewall's Interface IP as the default route for OPNsense. I do not want any outbound NAT to occur. The external router should see the source IP as the real IP of the server that sent the packet.

Example:
192.168.1.1/24 is the OPNsense Interface 1 and is set to use 192.168.1.254/24 as it's default gateway. This is an internal subnet used as a transit VLAN for access to the external WAN router.
192.168.1.254/24 is the external firewall's interface IP.
192.168.100.1/24 is OPNsense Interface 2 and is another subnet for servers.
192.168.100.232/24 is the real server's IP in this example, which the external firewall should be able to see as the source IP of any packets

Routing works fine, but for some reason all traffic the OPNsense sends to it's default gateway is NATed and the external firewall sees the source IP as the OPNsense Interface IP (192.168.1.1) instead of the real server's IP of 192.168.100.232.

I have tried setting Outbound NAT to use Manual rules and set the 192.168.100.0/24 source subnet to NONAT and have also tried Disabling Outbound NAT rules. In both cases the IP seen on the external firewall is the OPNsense NATed IP of 192.168.1.1.

Please assist. Am I doing this wrong in OPNsense perhaps?

Thanks!

This has nothing to do with the direction of the packets you want to filter accordingly. Normally, enforcing policies is on (1.) and rarely on (2.), because why would you forward something through a firewall if you are going to discard it when it is ready to exit?

I hope that explains it a bit better.


Cheers,
Franco

Yes, it does, and it negates most of my concerns. Thank you!

Travis

Hi there,

I would say the concepts have not changed here.

All rules tabs other than floating are inbound-only, so that's one's first distinction if one needs outbound filtering, but most of the time one doesn't.

2) This is confusing in terminology, you said outbound but hopefully meant incoming VLAN traffic from the perspective of the firewall.

4) Last match wins is an idiosyncrasy of the pf packet filter. "first match / quick" is the normal mode of operation for most firewalls but it was added later to the filter in historical terms. The original rule evaluation was based on "last match" where rules could be written ordered as unspecific to super specific. If you disable "quick" on floating rules, you will gain this behaviour for that particular rule.

The most useful way to use last match is to have a floating rule (which is evaluated before the other rules tabs) in last-match mode that acts as a placeholder for more specific rules in the individual tabs and yields authority to a later match there.


Cheers,
Franco

Franco,

This discussion has very much confused me...

a) Why would you want to separate Inbound and Outbound rules by forcing us to put Inbound rules in the Floating section and Outbound rules in the Interface section? It makes it harder to manage and keep rules straight!

b) I have been under the impression all along (because it made sense given that Floating rules apply before Interface rules) that Floating rules would apply to all interfaces first, and then the specific Interface rules would apply, and that both Floating and Interface specific rules could be for Inbound & outbound rules. Could you explain the intent / reasoning for this design?

c) You said "if one needs outbound filtering, but most of the time one doesn't", and I'm unclear as to why you say this because best practice for any corporate network is to have a DMZ where you NEED Explicitly defined inbound and outbound rules with specific allowed sources, destinations, and ports. Could you explain the thought behind your statement?

Thanks!
Travis

Didn't get it was that error causing it to fail... Why not just ignore bad entries?

Anyway, I removed the site from the ACL and re-ran squid -z.

That totally locked up OPNsense. No console no gui.

A subsequent hard reset and the Proxy came up.

It was working fine, and after an upgrade it no longer starts. Nothing in the proxy logs.

However in shell, I found this:

Code Select Expand

root@OPNsense:/var/log # cat squid.syslog.log
Feb 28 11:28:57 OPNsense (squid-1): Bungled (null) line 3: sslproxy_cert_sign signTrusted all
Feb 28 11:29:04 OPNsense (squid-1):     Failed to verify one of the swap directories, Check cache.log   for details.  Run 'squid -z' to create swap directories         if needed, or if running Squid for the first time.
Feb 28 11:29:12 OPNsense (squid-1):     Failed to verify one of the swap directories, Check cache.log   for details.  Run 'squid -z' to create swap directories         if needed, or if running Squid for the first time.
Feb 28 11:29:19 OPNsense (squid-1):     Failed to verify one of the swap directories, Check cache.log   for details.  Run 'squid -z' to create swap directories         if needed, or if running Squid for the first time.
Feb 28 11:29:27 OPNsense (squid-1):     Failed to verify one of the swap directories, Check cache.log   for details.  Run 'squid -z' to create swap directories         if needed, or if running Squid for the first time.
Feb 28 11:29:38 OPNsense (squid-1):     Failed to verify one of the swap directories, Check cache.log   for details.  Run 'squid -z' to create swap directories         if needed, or if running Squid for the first time.
Mar  8 09:48:43 OPNsense (squid-1): Bungled (null) line 3: sslproxy_cert_sign signTrusted all
CLOG▒▒▒root@OPNsense:/var/log # cat cache.log
cat: cache.log: No such file or directory


And upon running squid -z I got this:

Code Select Expand


root@OPNsense:/var/log # squid -z
2017/11/06 18:50:40| ERROR: '.tg.local' is a subdomain of 'tg.local'
2017/11/06 18:50:40| ERROR: You need to remove '.tg.local' from the ACL named 'bump_nobumpsites'
FATAL: Bungled /usr/local/etc/squid/squid.conf line 27: acl bump_nobumpsites ssl::server_name "/usr/local/etc/squid/nobumpsites.acl"
Squid Cache (Version 3.5.27): Terminated abnormally.
CPU Usage: 0.015 seconds = 0.007 user + 0.007 sys
Maximum Resident Size: 46000 KB
Page faults with physical i/o: 0
root@OPNsense:/var/log #


Thanks!

@Franco, i never thought that i'll see the day when someone suggest using any>any rules on an opensource/firewall product forum!?

I agree, but perhaps he was meaning as a troubleshooting step to verify it's not a issue with a rule not working as intended?

Which brings me to: Why do the rule logs not show Which Rule was hit? :(

Franco,

Can you answer why VIPs only route out the Default Gateway? Is it by design? If not, can you test and verify if you can make it route out a different gateway? I'm stuck...

So PFsense does the same thing. Virtual IPs apparently are not designed to work like a true VIP.
I even created a new gateway, using the VIP, and added a new route to use the new gateway. Still doesn't follow the rules. The VIPs ONLY want to route out the default gateway :(

So I think the reason it's failing is because it's not honoring the routes. Routes and VIP Attached.

I assume you applied the rule after creating it?

Other than that possibility, the rule looks correct. Edit the rule to allow logging and apply it. Then check the logs.

If you do not see logs of it being blocked, then use the Interfaces->Diagnostics->Packet Capture option to get a trace of icmp packets only and check the trace to ensure you see that icmp traffic hitting the firewall.

Resources per VM are too limited for that, but it might work for just the firewall; worth a try.

On a side note, I found out why the IP doesn't respond. OPNsense is routing the reply wrong. It's sending the reply ICMP packet to the default gateway mac instead of the source MAC. See attached. Note since original posting I've upgraded to 17.1.

So I know about Virtual IPs.

I tried configuring one but it does not seem to work. I can't ping the VIP from another PC on the same subnet. I don't know if I'm missing something in the configuration. I added Firewall allow rules for that Subnet, and the firewall logs show an allowed packet for ICMP, but the VIP does not respond.

I found out why it doesn't respond... It's sending the reply to the default route instead of back to the souce MAC. WHY? See attached.

Second virtual adapter - not possible. This is a cloud system managed by a 3rd party provided for free for labbing up a Virtual environment and I don't have the ability to add another NIC.

Virtual IP on the LAN: I tried configuring one but it does not seem to work. I can't ping the VIP from another PC on the same subnet. I don't know if I'm missing something in the configuration. I added Firewall allow rules for that Subnet, and the firewall logs show an allowed packet for ICMP, but the VIP does not respond.

Thanks!