Topics

I am running Opnsense under Proxmox. I had an i350 with 4 ports. I passed one port to the Opnsense VM and set up a 3 member vlan. I upgraded my NIC to a x710 10Gbe and had to reconfigure the NIC. Now when Opnsense starts I get the error below:

ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_ERROR, error I40E_AQ_RC_ENOENT

What is this telling me and how do I fix it?

IIRC back in 22.x due to a bug we had to create a Parent interface for vlan WAN connections or turn on Promiscuous mode. I have read this was fixed and no longer necessary. I tried removing the the Parent on one a system and when I do the WAN interface no longer works. It receives a DHCP address from the WAN but I can't ping anything. I tried deleting the Parent then run the interface wizard to redefine the interface to no avail.

What are the proper steps to removed the Parent to get things working? 

I upgraded to 23.7.9 and now my Wireguard PIA tunnel is broken. This also happened on the last upgrade but I rolled back to 23.7.7.3 which works fine. I see this generic error in the WG diag logs

Code Select Expand

/usr/local/opnsense/scripts/Wireguard/wg-service-control.php: Skipping gateway WG_PIA_GW due to empty 'gateway' property.

Looking at Wiregurad Diagnostics I see an active connection:

Code Select Expand

Name Port/Endpoint Handshake SendReceived
PIA-Server xxx.xxx.xxx.xxx:1337 2023-11-24 11:28:20        1.23 KB 368.00 Bytes

Looking at Interfaces -> Overview I don't see any packets being transmitted

Code Select Expand


Status up
MAC address 00:00:00:00:00:00 - XEROX CORPORATION
MTU                       1420
IPv4 address                 xxx.xxx.xxx.xxx/32
In/out packets         0 / 0 (0 bytes / 0 bytes)
In/out packets (pass) 0 / 0 (0 bytes / 0 bytes)
In/out packets (block) 0 / 0 (0 bytes / 0 bytes)
In/out errors                 0 / 0
Collisions                 0




What changed and how do I fix this?

I have two gateways, one for regular internet and one for PIA VPN using wireguard. This has been running fine for a few years and every upgrade up to 23.7.7. Now the lan IP I have routed through the PIA Wireguard gateway fail. Any idea why these IPs are not getting routed correctly now? Let me know what configuration files etc you would want to review.

Sometime after I updated to OPNsense 23.1.5-amd64 the system loses access to the WAN. I have OPNSense running under Proxmox. In the Proxmox server I have a quad port Intel I350 GigE adapter. One port is passed through to Opensense. I have the interface vlan'd, one for the wan, one for home lan, and one for home guest. I still have a parent set up for the wan. This setup has been rock steady for many years.

My question is how to best troubleshoot this issue. Proxmox logs show nothing. What logs in Opnsense should I look at?

I have a back up system, SMC Atom C2758 running Proxmox with Opnsense, and it is rock solid (using it now to post this). This system is setup with a vm vlan aware bridge interface. It does not use a parent for the WAN, just three vlans setup on the bridged network.

Let me know anything else I need to post to help troubleshoot this.

Resolution RTFM, IE this was documented in the Upgrade notes but somewhat cryptic, which has since been clarified..  Read on

22.1 changed how vlan MAC address spoofing worked. In 22.1 you either turn on promiscuous mode on a vlan'd DHCP enabled WAN interface, or add the sppof MAC address to the Parent.

To spoof the Parent vlan interface: in the GUI add the Parent, enable it and add the spoof MAC. The spoofed MAC will propagate to all vlan'd interfaces. If you want you can change <Parent>_vlan<id> interface MACs.

UPDATE TO BElOW. Got a new DHCP lease request on the Partaker and that knocked out the WAN connection so the Virtual and Bare Metal routers have the same behavior.

UPDATE 2 I suspect I am having the same issue as this post https://forum.opnsense.org/index.php?topic=26554.0 Will do a TCP dump to verify when I have a chance.

I have an Opnsense VM running as a Proxmox guest. I have two Proxmox servers a 6.4 and 7.0. I have setup a MicroTik mid router on my network so I can do upgrades to test and make sure things work post upgrade.

I have Verizon FIOS as my provider and use MAC spoofing. Both Proxmox systems have a quad port Intel i350 GigE card which I PCI passthrough one port to Opnsense. This setup has worked well for some time. I also have a 2 port fanless Partaker PC that I have Opnsense running as a backup. Opnsense is running baremetal on the Partaker.

The setups use a single vlan nic, wan vlan10, lan vlan100 and guest vlan200. I upgraded the Partaker PC first and it worked without issue on the MicroTik and Verizon networks.

I then upgraded the backup Proxmox system (PM 7) behind the MicroTik and that went well. I tested throughput on the MicroTik network and everything was great. Now comes the odd part. When I move the Proxmox Opnsense port from behind the MicroTik to my LAN, the WAN port will not acquire an IP address from Verizon. The Partaker PC which uses the same vlan setup and connects to Verizon fine. FWIW the Partaker uses Intel 82574L nics.

The Proxmox Lan side works and I am assuming the guest side does also but I haven't tested it. So there is something that either Verizon or Opnsense Proxmox Guest doesn't like on the WAN side of things with 22.1. Any suggestions on what to look for?

I have a mini PC (Partaker i5) that I use as a backup router. My goal is to fire this up via WOL if the main router fails. The other issue I have with this system is it won't power on after a power outage thus the need for WOL.

The problem I have. If I do a normal Opnsense shutdown WOL fails. If I remove power then add power, WOL works. If I boot a live Linux version a shutdown then WOL works. It seems like Opnsense is disabling WOL.

My ifconifg -m which shows wol_magic. Any ideas??

Code Select Expand

ifconfig -m em1
em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
        capabilities=953d9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP>
        ether 00:e8:4c:68:48:57
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        supported media:
                media autoselect
                media 1000baseT
                media 1000baseT mediaopt full-duplex
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

I have a Supermicro system with quad Intel I354 adapters. I upgraded to 21.1.6 last night and now the adapters are dead. I use a single vlan'd adapter our of the four (igb0). It is seen in pciconf but shows no carrier in ifconfig. Any suggestions?

root@crawford:~ # pciconf -lbcevV pci0:0:20:0
igb0@pci0:0:20:0:       class=0x020000 card=0x1f4115d9 chip=0x1f418086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection I354'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xdf260000, size 131072, enabled
    bar   [18] = type I/O Port, range 32, base 0xe0c0, size 32, enabled
    bar   [20] = type Memory, range 64, base 0xdf30c000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 10 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 2 root endpoint max data 512(512) FLR NS
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 0cc47affffdbed74
    ecap 0017[1a0] = TPH Requester 1
    ecap 000d[1d0] = ACS 1
root@crawford:~ #

root@crawford:~ # ifconfig igb0
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC>
        ether d4:a9:28:14:e8:98
        hwaddr 0c:c4:7a:db:ed:74
        inet6 fe80::ec4:7aff:fedb:ed74%igb0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@crawford:~ #

Cable and switch ports are good.

I am running Opnsense 21.1.4 as a guest of a Proxmox host. I have a dual port Intel 82576 NIC which one port is set up with 7 VFs. I am trying to get Opnsense to use one of the VFs. If I do a pciconf -lv I see the interface, but when I do a ifconfig it does not come up. I am guessing there is no driver for this. Is there a driver for this I  can install? BTW NIC passthrough works on the same NIC. Its just an issue with Opnsense and the VF NIC.

I am trying to get hosts on my Guest vlan to use my Pihole server on my Home vlan. I have setup a rule on Guest

Protocol - IPV4 TCP/UDP
Source - Any
Port - Any
Destination - <PiHole Server>
Port - 53(DNS)
Gateway - Any

and it doesn't work

I turned on Rule logging and can see what I think packets going to Pihole but its not working. Any suggestions? Images of rule an log below

I have a nut server running and trying to attach my opnsense system to it. I installed the NUT service but it won't start. Not sure what I am missing. BTW if I drop into the console and run uspc <myupsname@nutserveraddress:port> I get the UPS status as expected. So I must be doing something wrong in the setup panel.

What I have:

General Setting:

Enable Nut - checked
Service Mode - netclient
Name - deltec-PRA1500 (UPS Name on the NUT Server)
Listen Address - 127.0.0.1 (I also tried <NUT server ip>

UPS Type:

Netclient
Enabled Checked
IP Address <NUT Server IP address>
Username <blank>
Password <blank>

Thought I would try Wireguard client connection to PIA. I used Jonny's Wireguard PIA setup script which did its job. I then moved my VPN IPs to PIA Wireguard from PIA OpenVPN. 

Over time I run into connection issues. I am using Pihole for DNS forwarding. The longer WG is up I get more and more browser connection issues. I can ping the FQDN without issues during the timeouts. I am wondering if this is a routing issue? As I write this I figure a trace route would be in order when the connection issues start.

Anyone seen these type of issues with WG. I have no issues with OVPN.

I am trying to get Opnsense to route certain IPs on my network to PIA VPN. I have the OpenVPN client setup and connected and assigned as a gateway. I can't get traffic from my LAN to PIA Gateway. I looked a many guides and nothing. To verify I check one of the Whats my IP websites on the client I am trying to route and it shows my ISPs assigned address instead of the PIA address. Any suggestions to what to look for and what I might be missing.

Pictures below:

Gateway info and state



Alias for the a system I want to route. Once this starts working I will add more IPs



Firewall -> NAT -> Outbound



Firewall -> Rules -> Lan



VPN -> OpenVPN -> Clients -> Don't pull routes




Followup 01-01-21

I see a few come across this post and have questions. First if things don't work post your config so people can help.

Since I posted this I noticed things I missed mentioning in the original post which I think will help.

First: Gateway Priority. Check it, and make sure your ISPs priority is higher than your VPNs priority. Higher means lower value. I now set my ISP Gateway priority to 250, and add 2 to any VPN added. Adjust as necessary.



Just an observation. when you setup PIA Wireguard (use Johnny's excellent script https://github.com/FingerlessGlov3s/OPNsensePIAWireguard) a Nat Outbound Automatic rule is created. No need to do anything. However if you do use OpenVPN then create a Hybrid as described in all the HowTo's. I have both setup to test, and don't see any difference between the two. I do have to say Wireguard VPN performance on a standalone Linux system is better than Opnsense Wireguard. Probably the difference between Userspace and Kernel implementations.



Speaking of Wireguard, make sure when the interface is added you set the MSS (Maximum Segment Size) to 1380.Not sure why this isn't negotiated at connect time, but its not and your performance will suffer. Johnny does mention this, but I wanted to reinforce the point.



Finally I want to mention the Kill Switch. Somehow I kept missing adding to the tag to the Lan rule and matching the tag to the Wan Floating rule. You will probably have to hit the Advanced Show/Hide to see the field. Again I just wanted to reinforce this.

Create Local Tag in Lan rule for your Aliases



Finally the Wan Floating Match Local Tag set to block VPN destined traffic if the VPN goes down.

Match Local Tag. Remember to click on Advanced Show/Hide



The Block Rule

For the last few years I have run Opnsense under KVM. For the past few months randomly Opnsense stops working. It loses the wan interface, and I can't log in fully. I can bring up an SSH session type in the user name and password but then it hangs. I have waited for up to 15 minutes but I never get to the console GUI. Trying the web admin is similar. The system is not completely unresponsive since I can reboot cleanly via virsh reboot.

Any idea how I might be able to shoot this issue? I am running 18.7.9.

I run Opnsense as a Fedora 26 sever KVM Guest. So far no issues. I decided to take the plunge and upgrade to 18.1 today. The upgrade was ultimately successful but I had to manually power cycle the VM 3 times before all completed. The auto reboot mechanisms didn't seem to work.

I'm just posting this as an FYI. If there are logs you want me to collect I am happy to do so. I have a backup setup running as a SmartOS KVM guest which I need to upgrade. Not sure if I will run into the same issue but if you want me to monitor something just in case I get the same hang let me know. I'll do the upgrade sometime later this week.

Is there a way to show this from the command line?

I am trying to upgrade from 17.1.3 to 17.1.4 but it gets stuck. I tried from the GUI and the console without success. From the GUI I see this

There are 38 updates available, total download size is none. This update requires a reboot.

Then I hit the Upgrade Now Button

and the log shows

Code Select Expand


***GOT REQUEST TO UPGRADE: all***
Updating OPNsense repository catalogue...
OPNsense repository is up-to-date.
All repositories are up-to-date.
Updating OPNsense repository catalogue...
OPNsense repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (46 candidates): .......... done
Processing candidates (46 candidates): ........ done
Checking integrity... done (1 conflicting)
  - py27-setuptools-32.1.0_1 conflicts with py27-setuptools27-32.1.0 on /usr/local/bin/easy_install
Cannot solve problem using SAT solver, trying another plan
Checking integrity... done (0 conflicting)
The following 38 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
py27-sqlite3-2.7.13_7
opnsense-17.1.3
py27-setuptools27-32.1.0

New packages to be INSTALLED:
py27-setuptools: 32.1.0_1

Installed packages to be UPGRADED:
squid: 3.5.24 -> 3.5.24_2
png: 1.6.28 -> 1.6.29
pkgconf: 1.3.0_3 -> 1.3.0,1
php70-zlib: 7.0.16 -> 7.0.17
php70-xml: 7.0.16 -> 7.0.17
php70-sqlite3: 7.0.16 -> 7.0.17
php70-sockets: 7.0.16 -> 7.0.17
php70-simplexml: 7.0.16 -> 7.0.17
php70-session: 7.0.16 -> 7.0.17
php70-pdo: 7.0.16 -> 7.0.17
php70-openssl: 7.0.16 -> 7.0.17
php70-mcrypt: 7.0.16 -> 7.0.17
php70-ldap: 7.0.16 -> 7.0.17
php70-json: 7.0.16 -> 7.0.17
php70-hash: 7.0.16 -> 7.0.17
php70-gettext: 7.0.16 -> 7.0.17
php70-filter: 7.0.16 -> 7.0.17
php70-dom: 7.0.16 -> 7.0.17
php70-curl: 7.0.16 -> 7.0.17
php70-ctype: 7.0.16 -> 7.0.17
php70: 7.0.16 -> 7.0.17
opnsense-update: 17.1.3 -> 17.1.4
opnsense-lang: 17.1.3 -> 17.1.4
ntp: 4.2.8p9_4 -> 4.2.8p10_2
lzo2: 2.09 -> 2.10_1

Installed packages to be REINSTALLED:
py27-ujson-1.35 (direct dependency changed: py27-setuptools)
py27-requests-2.11.1 (direct dependency changed: py27-setuptools)
py27-pytz-2016.10,1 (direct dependency changed: py27-setuptools)
py27-netaddr-0.7.18 (direct dependency changed: py27-setuptools)
py27-MarkupSafe-1.0 (direct dependency changed: py27-setuptools)
py27-Jinja2-2.8 (direct dependency changed: py27-Babel)
py27-Babel-2.3.4 (direct dependency changed: py27-setuptools)
openvpn23-2.3.14_1 (options changed)
dnsmasq-2.76,1 (options changed)

Number of packages to be removed: 3
Number of packages to be installed: 1
Number of packages to be upgraded: 25
Number of packages to be reinstalled: 9

The operation will free 20 MiB.
pkg-static: Cannot delete vital package: opnsense!
pkg-static: If you are sure you want to remove opnsense,
pkg-static: unset the 'vital' flag with: pkg set -v 0 opnsense
Starting web GUI...done.
Generating RRD graphs...done.
***DONE***

is it safe to run pkg set -v 0 opnsense??

I finally got a chance to test Opnsense under Fedora 25 KVM. In my case the VirtIO driver under Fedora 25 Server caused some real problems. I could ping but couldn't get any traffic through the VM. I switched to e1000 and everything worked... From reading issues with PFSense the VirtIO drivers can be an issue.. So if you deploy on KVM keep this in mind if performance is in the toilet

I decided to test Opnsense running under a hypervisor and am running into performance issues. I tested SmarOS and ESXi 6.5 with the same results. If I run iperf from Opnsense/BSD as a client pointing to an iperf server on the wan or lan side I get good performance IE approx 550Mb/s. If I setup iperf as a server on Opnsense/BSD and point a client from either the wan or lan I get horrible performance IE 70Mb/s.

I tested a server on the wan and a client on the lan and vise-verse with similar results. It seems like when running under a hypervisor receiving packets are an issue. Anyone seen this and have a tweak? Its not a resource issue, CPU is low 15% and plenty of memory available.

BTW I get very similar results if I run vyos. So there is something specific about routers in a virtual setting.

Is it possible and if so is there a guide to setup and openvpn bridge instead of a tunnel? This would be a client to server not site to site.