Messages

Looks like I've narrowed it down to at least one, if not two python processes which spike and then a hiccup.   The first one I found was flowd_aggregate.py.  I have stopped the netflow service.

Issue still happens and when it does, /usr/local/bin/python3 /usr/local/opnsense/scripts/filter/update_tables.py spikes.

Going to remove my aliases which resolve various youtube names (moved this to pi-hole anyway) and see if the issue goes away.  I did notice there was a few names that were not resolving anymore, maybe related?

What modem model do you have?

OPNsense can do PPPoE (that's what I am using for my fiber).  If you can set your modem to full bridge mode you should be able to setup the PPPoE connection in OPNsense.  Without that, you're going to be doing double-NAT, which is exactly what you're describing being able to access the ADSL router from a machine behind OPNsense.  OPNsense is NATing the PCs traffic to it's WAN address and sending it out to the modem.  Double NAT is messy.

Hi

Seeing some hiccups/blips in connections every 20-30 seconds, more noticeable with UDP traffic such as video conference calls, and some games running on my server in another VLAN.  All games from LAN to server in DMZ experience the blip at the same time.

Used iperf as client on OPNsense via SSH in a few different scenarios and the firewall is the only common piece.  Tests out LAN interface, and WAN (over ipsec to Azure) show same exact behavior.  They are sharing a dual port GbE Intel card, gonna try and swap it when I can find time.

PktCaps from the firewall only show one packet out of order, so it's like things hum a long fine, then queue up and burst out (see attachment.  No re-transmissions or ZeroWindows that I can see.

Any diagnostics that would show a hardware or software issue?  I checked Interfaces > Overview and don't see any errors on the interfaces in question, however enc0 for the ipsec stuff doesn't show?

Thanks!

Curious what you mean by tagging.  Would you mind sharing your config?  TIA!

This doesn't make any sense.  From the shell, arp doesn't even show the ASA entry during active pings from it.

root@OPNsense:~ # tcpdump -i hn1 -ne icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on hn1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:52:32.920731 44:d3:ca:12:15:c0 > 00:15:5d:01:02:bd, ethertype IPv4 (0x0800), length 74: 192.168.10.11 > 192.168.10.22: ICMP echo request, id 1, seq 15999, length 40
16:52:32.920891 00:15:5d:01:02:bd > a0:36:9f:28:75:1c, ethertype IPv4 (0x0800), length 74: 192.168.10.22 > 192.168.10.11: ICMP echo reply, id 1, seq 15999, length 40

root@OPNsense:~ # arp -na
? (192.168.10.1) at a0:36:9f:28:75:1c on hn1 expires in 943 seconds [ethernet]
? (192.168.10.22) at 00:15:5d:01:02:bd on hn1 permanent [ethernet]
? (192.168.100.10) at 00:15:5d:01:02:be on hn0 expires in 1075 seconds [ethernet]
? (192.168.100.1) at 00:15:5d:01:02:bc on hn0 permanent [ethernet]

As soon as I ping from the OPNsense VM, it's there.  However, the ping response still goes to the wrong MAC.

root@OPNsense:~ # ping 192.168.10.11
PING 192.168.10.11 (192.168.10.11): 56 data bytes
64 bytes from 192.168.10.11: icmp_seq=0 ttl=255 time=1.870 ms
64 bytes from 192.168.10.11: icmp_seq=1 ttl=255 time=1.305 ms

root@OPNsense:~ # arp -na
? (192.168.10.1) at a0:36:9f:28:75:1c on hn1 expires in 911 seconds [ethernet]
? (192.168.10.11) at 44:d3:ca:12:15:c0 on hn1 expires in 1197 seconds [ethernet]
? (192.168.10.22) at 00:15:5d:01:02:bd on hn1 permanent [ethernet]
? (192.168.100.10) at 00:15:5d:01:02:be on hn0 expires in 1171 seconds [ethernet]
? (192.168.100.1) at 00:15:5d:01:02:bc on hn0 permanent [ethernet]

I've been using OPNsense for a little over a year as my home firewall (after switching from Sophos).  Been amazing.  I have my LAN interface setup with several VLANs, one of those VLANs is part of my lab.  I have a Cisco ASA plugged into that VLAN that's been working fine with a few VMs behind it.  I've been wanting to play around (troubleshoot for work) with some IPsec stuff on the ASA and so I put a OPNsense VM in the same lab VLAN, with a VM behind it just like the ASA.

ASA outside and OPNsense WAN are in the same /24 subnet.

ASA 192.168.10.11/24
OPNsense 192.168.10.22/24

Here's the bug, traffic leaves the Cisco ASA (or any other VM in the lab subnet), hits the WAN of OPNsense VM, but the response traffic is borked.  The dest IP is the Cisco ASA (great), but the MAC address of the Ethernet header has the gateway of my home OPNsense (physical one).

Route table on the OPNsense VM shows:

ipv4   default   192.168.10.11   UGS   56135   1500   hn1   wan       
ipv4   127.0.0.1   link#2   UH   74   16384   lo0   Loopback       
ipv4   192.168.10.0/24   link#6   U   172   1500   hn1   wan       
ipv4   192.168.10.1   00:15:5d:01:02:bd   UHS   6414   1500   hn1   wan       
ipv4   192.168.10.11/32   192.168.10.11   UGS   54731   1500   hn1   wan       
ipv4   192.168.10.22   link#6   UHS   0   16384   lo0   Loopback       
ipv4   192.168.100.0/24   link#5   U   79009   1500   hn0   lan       
ipv4   192.168.100.1   link#5   UHS   0   16384   lo0   Loopback

Why with the direct /24 route (as well as a /32 with a gateway IP of the Cisco) does the traffic use the wrong MAC address?  This makes OPNsense hard to use in a lab.  Sure I could put each in their own VLAN and route between them on my main OPNsense box, but I shouldn't have to.  Sophos (UTM and XG), pfsense, Cisco, Juniper and all my Linux and Windows VM work just fine in this scenario with no special config.  Is this a bug, or am I missing something?

Oh, I see what you mean.  However I see a pre and a post include in the main server block, but nothing inside the individual Location blocks, or am I missing something?  It would be nice if the location did so I could add a directive to a single [Existing] location which has various WAF Polcies and other settings already applied from the GUI.  I guess I could hack something together by creating a unused Location so it creates the upstream (unless all upstreams are added regardless), etc. and then duplicate it into a new hook.  Thoughts?

Hi

Using the WAF/NGINX for a few different apps and it's been awesome!  One thing I am looking to figure out is how can we add specific headers for a single Location via the GUI?  I am working on getting the notifications working for Bitwarden_rs which uses websockets for a specific location/path to route to another port on the backend server.

I need to add the following for /notifications/hub:

  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $http_connection;

I was able to create the location as needed and then edit /usr/local/etc/nginx/nginx.conf manually and reload the service, but I would rather not have to do this each time I make a change.  I noticed the notifications are working without, but would be nice to include them.

TIA!

Did you turn on IPS?  IPS can have a huge hit on throughput in my experience.

Hi
Just moved one of my VMs into my DMZ vlan, which is on the same physical interfaces as the LAN vlan.  This is a minecraft server which has worked flawlessly before attaching to the other vlan.  Having an issue where after 15-20 minutes the clients will just drop, and the server reports as 'xxxxxx left the game'.

This got me interested in knowing the life of the connection and why the connection/state was closed by the server.  I started poking around in logs, and stuff.  At one point I had netflow data going to my Graylog instance, but I must have turned that off as all I have now is filterlog and nginx/waf logs.

So my question is, obviously we have the filterlog which shows the src, dst, interface, action, and so on.  Is there any log like conntrack or similar which will report the states end reason (age-out, FIN, RST, etc), bytes sent/recv, and other metrics?  Would be nice to know for instances where something happened and I can't run a tcpdump while reproducing.

Thanks in advance!

Thank you for this plug-in!!  Now I can put my Minecraft server in my DMZ and still have my kids find the server without having to fuss in the server lists!

Yeah, it appears that my Zimbra server is rejecting it.  I think it's because I have postfix configured with the same name as my mail server as I was trying to avoid MX record changes.  I've backed out all my changes and will work on this another time.  Need to update my reverse DNS on another public IP and configure postfix with a different name.  Thanks for the help!

[DMARC:Quarantine]

I actually just figured this issue out, I had changed the interfaces that postfix was listening on, to the public IP that is my MX record.  I guess since it was only bound to the WAN, it wasn't able to route out the internal IP.

Now I have a new issue.  Test emails from Gmail are coming through, but I'm seeing "250 2.0.0 OK DMARC:Quarantine" in the postfix logs, and the email is not in my inbox (even though I am seeing the connections in tcpdump on my mail server).  Trying to figure out where messages are being quarantined to in Zimbra, but I'm wondering if postfix is modifying the message or something that is causing this?  Never had delivery issues.

Hi

20.7.7_1 and running postfix to relay external email to my internal Zimbra server (in case it's unavailable, etc.).  I have my only domain configured with the internal IP address (192.168.1.X).  I can see email getting deferred in the log:

Code Select Expand

status=deferred (connect to 192.168.1.X[192.168.1.X]:25: Operation timed out)

I am able to Port-Probe and ping the host just fine from both the GUI and SSH.  So I start a packet capture and the traffic is heading out the pppoe0 interface, when it should be going out the igb0 interface which is the LAN!

Code Select Expand

# route get 192.168.1.X
   route to: mail
destination: 192.168.1.0
       mask: 255.255.255.0
        fib: 0
  interface: [color=red]igb0[/color]
      flags: <UP,DONE,PINNED>

I'm scratching my head.

[[55753]]

Hi

Noticed today that my Graylog instance wasn't parsing the filterlog events, after looking at my regex I noticed that there appears to be a number in square brackets after filterlog.

used to be like this:
<134>Oct 14 18:28:48 gw.domain.com filterlog: 80,,,0,igb0,match,pass,out....

now it's:
<134>Oct 14 18:28:48 gw.domain.com filterlog[55753]: 80,,,0,igb0,match,pass,out....

This must have changed after 20.1?  Any indication what the number is, my guess is the PID of the pf daemon/service.

Thanks