Messages

I seemed to have solved my own issue.  Apparently my first Nic port is defective.  Finally noticed with trying different things and plugging and unplugging that the ethernet data lights were not lighting up when I plugged into Nic #1.  Moved everything over to another Nic and BAM, it started negotiating and working as expected on a quick test.

So I have put everything back to previous still using the RG IP passthrough for now until I get a chance to properly adjust everything on a permanent basis.

Sometimes it is the simplest of things....  :-[

[wpa_supplicant.conf]

I cannot get this to work on my end.  I have followed the latest post additions and still am unable to bypass my BGW210.

I am suspecting my issue is I am running Opnsense as a VM in Proxmox and my setup for bridging my NIC to my VM is not setup properly?

Based on information I have seen online, my WAN interface on my Proxmox is setup as follows:

Code Select Expand


iface enp1s0 inet manual

iface enp1s0.0 inet manual

auto vmbr1
iface vmbr1 inet manual
bridge-ports enp1s0.0
bridge-stp off
bridge-fd 0
post-up echo 8 > /sys/class/net/vmbr1/bridge/group_fwd_mask


My Proxmox OPNsense VM WAN network is setup as follows:

Code Select Expand


net0: virtio=<BGW210 MAC address>,bridge=vmbr1,queues=8


And my various config files based on the above posts are as follows:
wpa_supplicant.conf

Code Select Expand


# Change file names to absolute paths
ctrl_interface=DIR=/var/run/wpa_supplicant
openssl_ciphers=DEFAULT@SECLEVEL=0
eapol_version=2
ap_scan=0
fast_reauth=1
network={
        ca_cert="/conf/wpa/CA.pem"
        client_cert="/conf/wpa/Client.pem"
        eap=TLS
        eapol_flags=0
        identity="<BGW210 MAC address>" # Internet (ONT) interface MAC address must match this value
        key_mgmt=IEEE8021X
        phase1="allow_canned_success=1"
        private_key="/conf/wpa/PrivateKey.pem"
}


openssl.conf

Code Select Expand


openssl_conf = openssl_init

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Options = UnsafeLegacyRenegotiation
MinProtocol = TLSv1
CipherString = DEFAULT@SECLEVEL=0


/usr/local/etc/rc.syshook.d/early/04-wpa

Code Select Expand


#!/bin/sh

/sbin/ifconfig vtnet0 link <BGW210 MAC address>

env OPENSSL_CONF=/conf/wpa/openssl.conf /usr/local/sbin/wpa_supplicant -Dwired -i vtnet0 -B -C /var/run/wpa_supplicant -c /conf/wpa/wpa_supplicant.conf -t -dd


And finally, I set my WAN MAC to <BGW210 MAC address> in Opnsense GUI.

When I boot up, it hangs on WAN initialization and when I try to manually start 04-wpa, it fails.

Any idea what I need to adjust to get the working on my end?  I am running Opnsense version 24.7.7-amd64.

Thank you!

Happy to report that the patch solved my issue, as well.  Thank you!

I have just tested the patch, it fixed the VIP issue and no more error related to VIP in log.
Thanks for your help.

Bunch, just curious, did you also turn off "Allow service binding" in your VIP settings with the applied patch as suggested in this thread? 

I am hopeful I can apply and test the patch in the next couple of days.  But it looks like it does the trick!

Can you point me to a tutorial on how to apply patches to my install?  I have seen these posted from time to time, but have never learned how to apply them manually (which I assume is something I need to do).

Sorry to spam the thread.  I just dug a bit deeper in your post and realized you already provided the way to apply the patch via https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-patch: opnsense-patch 9a618ba6

I will try to apply this week and report back.

I just checked my Logs and have the same error related to Virtual IP as Bunch reported.

/firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '10.17.0.1' -alias' failed to execute

So, if I can get a gentle nudge on how to apply the patch, I can verify if it resolves the issue.  Do I manually edit the parameters in the files or is there a script that executes the patch?

Hmm, no feedback at all?

Sorry, have not had time to come back and follow up on this issue.  I appreciate Bunch being more thorough with the troubleshooting than I have been.

Can you point me to a tutorial on how to apply patches to my install?  I have seen these posted from time to time, but have never learned how to apply them manually (which I assume is something I need to do).

So who reverts the option? Can you check the configuration history?

I checked config history and the only "change" to VIP settings is me re-saving the settings.  But maybe I didn't explain myself clearly. Nothing is changed and I don't actually change the settings, I just navigate to the settings and click Save.  After that HAproxy starts up with no issues.  It will not start until I do this step manually.

In general, however, your issue is that something listens to the IP you want HAProxy to listen to. Disabling the service bind doesn't do anything for HAProxy plugin so it's another service, likely the web GUI configured to listen to said interface manually, which we don't recommend for exactly this reason.

I moved from pfsense over a year ago to OPNsense.  I started using HAProxy in pfsense to redirect to various servers on my local LAN.  I ran into an issue where when I was on the local LAN, I could not reach the internal servers using the domains I setup for them that worked from outside my LAN.  I found a guide that instructed me to setup a combination of Virtual IP and firewall rules to get it working internally as it did externally.

When I moved to OPNsense, it was not a direct 1 to 1 transfer and it took me some time to get it all working but I eventually did.  I have been running OPNsense with this configuration for over a year with no issues until I did the recent update.

Are you saying that I should NOT have "Allow service binding" active on my Virtual IP in order for HAProxy to properly bind my listening frontends to?

Just updated to OPNsense 22.1.3-amd64 this morning and finally resolved an issue with HAProxy not starting.

HAProxy was working fine before the update, but after HAProxy would not start.  Config file reported no errors, but I could not find anything in log files as to why HAProxy was not starting.  So I finally connected via terminal and tried to start HAPRoxy manually and got an error that it could not bind to my internal Virtual IP.

SO I went to the VirtualIP settings in the GUI and confirmed that I had Allow service binding turned on.  As a test I re-saved and applied the settings.  After that HAProxy started up just fine.

I rebooted to test and the same thing happened.  I had to manually re-save the Virtual IP and apply the settings and the HAPRoxy could be started.

Is there some new setting somewhere that I need to activate to get this to work automatically like before?

I Agree with above comment. I use AWS R53 and Tunnel Broker(HE.net) which are built in options in the old dyndns client. They don't exist in the ddclient settings. It doesn't  have to be an all in one in my opinion. I've seen projects for aws53 updates that could maybe be worked into a new module. For example: https://github.com/crazy-max/ddns-route53

Adding another voice for AWS Route53 support with the new preferred dynamic DNS plugin.  I just upgraded to 22.1 and got knee-jerk nervous on the warning message about DYNDNS deprecation.  Tried the new plugin and saw Route53 is not an option, so reverted back to DYNDNS. 

I also echo others in this thread about choosing a replacement that has at least if not more of the same options of the deprecated plugin.  Though understand about the difficulty also expressed.

Hopefully by the time 22.7 rolls around, a more robust solution is available.

Thanks!

Just set this up myself, there are a few Unifi-isms, I'm running:

3 x Unifi AC AP Pros
1 x Unifi Pro 48 Port Gen2 switch
1 x Unifi Pro 24 Port POE Gen2 switch
A number of Flex and Flex Mini switches

- Firstly, firmware 4.3.20 is key for me.  On APs and switches.  .21 and .22 caused all sorts of havoc.  I shall be staying on this firmware version

- Firewalls rules as you have above, destination 224.0.0.251, UDP, port 5353, inbound all VLANs you want to repeat

- Enable 'Multicast Enhancement' on each wifi network that you have mDNS repeater setup for on the Unifi controller (Settings -> Wifi - > Edit -> Advanced

- Enable IGMP Snooping on the Unifi for each VLAN/profile setup with mDNS (Settings -> Advanced Features ->Network Isolation -> Edit)

- I had to allow all traffic BACK from my AppleTVs, to the streaming devices (iphones, computers).  I think the port range is huge, so I decided to create 2 groups and allow all traffic between them. Airport express seemed to work ok without this, but I believe AppleTV needs to be able to initiate connections back to the iPhone, computer, etc.

.. think that's it

Sorry to revive an old thread but am trying to get this working for myself.  I have recently made the switch from pfSense to OPNSense and am trying to get my AirPrint to work across VLANS.  I also am running UniFi AP's and switches, so have turned on the features you mentioned on those items.

My printers are on a IOT VLAN (103) with IPs 10.103.0.0/24
I have a LAN network with IPS 10.1.0.0/23
I have a Guest VLAN (102) with IPs 10.102.0.0/24

I have activated os-mdns-repeater and have it listening on the LAN, IOT and Guest interfaces.  I can see and print to the printers from my LAN, which has access to all the other VLANs.  The IOT and Guest VLAN has rules blocking anything originating on those nets to the LAN net.

I am trying to get my Guest net to also see and print to the IOT printers, but AirPrint fails to discover them. I am sure it is a Firewall rule, but am having a hard time understanding the discussions I come across that discuss the rules.  In particular from your post:

Firewalls rules as you have above, destination 224.0.0.251, UDP, port 5353, inbound all VLANs you want to repeat

I am not following what this means.  Can you please show me a shot from your rules table with these rules so I can decipher what I need to set.  Where does this rule get placed? 

Thank you!

Not sure what I mucked up, but I decided to reset OPNSense back to defaults to help another issue and now it is working.  So I had some rule or other configuration messed up somewhere that apparently was causing the problem. 

I just setup one of my other ports on my Proxmox+OPNsense test as a separate interface OPT5.  I then plugged my laptop into it.  I get DHCP assignment and can resolve DNS, but nothing beyond that.  Just like my VLAN virtual port test above.

So it seems like it is something in my OPNSesne setup that is preventing connection form my downstream ports/VLANS to the WAN port so it can resolve internet connections.

I am not sure what it is as I never had issues like this with pfsense.  I would just setup the interface/VLAN and connections worked.

Any insight on how to get this working, would be greatly appreciated.

[Option 1]

I have been testing a setup of OPNSense running on Proxmox as a potential replacement for my old pfsense system that never seemed to be able to achieve 1 GB throughput (or near enough) for my fiber internet.  I believe it is hardware choking it as I can not seem to get it over the hump.

I am currently testing the new Proxmox system behind my pfsense system as I have some complicated setups with VLANS and HAPRoxy that I want to get setup on OPNsense before I transition to ensure limited downtime.

In my existing setup, I have a separate Ubuntu server running some tasks on separate hardware.  My hope is to roll that into Proxmox as a VM to reduce the amount of boxes.  So I have generated the Ubuntu VM on Proxmox.  This is where I am running into issues.  I cannot seem to get my Ubuntu VM (UVM) to connect to the internet.

My test setup is this:

WAN -> PFSense -> LAN (17216.1.1/16) -> HomeLab VLAN (10.0.1.1/24) -> UniFi switch w/ VLAN Tagged -> New Proxmox box w/ 6 ports

Proxmox box setup
Port 1 bridge setup with 10.0.1.254 for managing Proxmox from my LAN network
Port 6 bridged (vmbr6) to act as WAN port for OPNSense (vtnet0)
Port 5 bridged (vmbr5) to act as  LAN port for OPNSense (vtnet1)

I have tried the three following ways with varied results:

Option 1
- UVM w/ virtual port based on vmbr5 (port 5 above), tagged with VLAN 105
- Setup VLAN 105 on OPNSense
- Create OPNSense interface for VLAN w/ LAN Parent interface with IP 10.105.0.253
- DHCP 10.105.0.1/29 for VLAN Interface

In this setup, I get DHCP address assigned to UVM as 10.105.0.249, DNS seems to function as it revolves google.com to IP with ping.  However, I do not have internet connection, as ping fails.  I cannot curl or anything else.  I can ping OPNSense gateway at both 10.105.0.253 (setup as interface IP in OPNSense for VLAN 105) and 10.0.1.253 (how I access OPNSense from my main LAN).  But beyond that, no route is established.

Option 2
- Create new vmbr7 bridge in Proxmox, not tied to port with CDIR 10.105.0.1/29
- UVM w/ virtual port based on vmbr7, no VLAN tag as vtnet03
- Create OPNSense interface for  vtnet03 with IP 10.105.0.253
- DHCP 10.105.0.1/29 for vtnet03 Interface

In this setup, I get DHCP to assign IP to UVM, but nothing else works.  No DNS, no internet, cannot ping anything.

Option 3
- UVM w/ virtual port based on LAN (vmbr5 above), no VLAN tag
- Gets IP assigned from pfsense VLAN for HomeLab

In this setup, I get IP assigned from my pfsense box and I can DNS, reach internet, etc.  But of course, can ping OPNSense on 10.0.1.253.  In this scenario, I am essentially bypassing OPNSense.  So , this will not ultimately work in my scenario when I remove pfsense.

So, why can I not reach internet.  It seems like either 1 or 2 above would work.  I am not sure why the internet route is breaking down.  Based on my searches on this, I see posts saying to ensure there is a route from the VLAN interface to the gateway (WAN in my case?).  However, auto outbound NAT sows a connection between Ubuntu interfaces in both option 1 or 2 above.  Does that not establish that route? 

Note, in these scenarios, OPNSense can reach the internet, resolve DNS, etc.  So the breakdown apparently seems to be within OPNSense itself, but I cannot figure out where.  I do not think I am doing anything exotic that I haven't already done in my original pfsense setup.  Th only difference is that the Ubuntu setup is now virtual inside Proxmox instead of physical hardware.  I am just duplicating that setup (option 1) in this test setup.

I have turned off Firewall in Proxmox on the virtual ports to ensure it wasn't causing the issues.  I have also made sure I have Pass all traffic on the the interfaces in OPNSense in both option 1 and 2 above. 

Any help on getting this working would be greatly appreciated. 

Following up and issues seem to have self resolved.  I let the box sit pending any insight to my issues.  When I returned to it, connection speed seemed to be back to normal.  Though I would have to reboot the unit after every few package installs, as it would appear to stall on subsequent installs.

I have also replaced the ethernet cables this morning to see if it was "hardware" issues. 

But, all in all, seem to be back to testing out.