Topics

1

20.1 Production Series / Help with HA setup

« on: May 28, 2020, 01:46:02 pm »

Hope someone can help me with this.
I had OPNSense running in a VM under esxi for some time. I didn't like loosing the network on server maintenance so I bought  an i3 NUC. While the NUC was coming I modified the existing installation to adapt to the new one.
I had three vlans on a virtual adapter each and the wan in a dedicated passthrough one. I changed it to a single trunk adapter with a router-on-stick configuration, with four vlans.

When I received the NUC I installed OPNS and restored a backup from the VM. Then I thought I could leave the VM and configure a CARP HA.
I followed this: https://www.thomas-krenn.com/en/wiki/OPNsense_HA_Cluster_configuration
and this: https://docs.opnsense.org/manual/how-tos/carp.html

I basically followed those instructions, but created a new vlan interface (also configured on switch) for the PFSync interface.
[As the NUC only has one net adapter (for now) I could not make the "mysterious and undocumented" LAGG overcome to allow syncs, so I left states synchronization deactivated.
I configured XMLRPC Sync.]

I fully configured the LAGG interface and HA settings.

Also, as I have several vlans, when documentation says to create a firewall rule to allow CARP, I created a vlan group with all the intranet+wan vlans and made the rule here*

As I have more than one interface, I created subsequent virtual IPs increasing the VHID group (WAN 1 / LAN 3 / TLN 4 / IOT 5)

The network was operative but I have the following problems:

• When the documentation says to create a fw rule for PFSync interface, "as it is a direct cable"... Mine is not a direct cable, and I didn't know how to create that rule, as documentation also doesn't show it. I created an All to all regular one.
  
• My firewall is full of block impossible hits. Like saying on interface TLN HOST1:80 contacts to HOST2:34234, when host1 is not at TLN but IOT and traffic seems inverted looking at ports. This doesn't affect normal usage.
  
• Failover doesn't work. I shutdown the NUC and network goes over. Not only, but whenever before I could manage switch by setting a fixed ip and attaching my pc on management vlan, now it fails. (could any ARP config in switch mess this?)
  
• Trying to XMLRPC Sync, fails saying that backup is not present. I can ping it and ping the master from the VM. I can see the allowed port 80 connection in the FW2 logs.
  
• The HA->Status menu point also says that there's no communication with the backup node. FW1 seems to hang for a while but FW2 spits the error immediately.
  
• After a day, all the appliances in TLN vlan (trusted) stopped working. The DHCP for this interface was deactivated at configuration (only this one). So I turned it on and started, but it doesn't work and keeps pulling this at log:
  (It does the same in both FWs)
  

Code: [Select]

2020-05-28T12:58:16 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:59 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:51 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:46 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:41 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:36 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:19 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:11 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:07 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:02 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:56:57 dhcpd: failover peer dhcp_lan: I move from startup to communications-interrupted
2020-05-28T12:56:57 dhcpd: failover peer dhcp_opt1: I move from startup to communications-interrupted
2020-05-28T12:56:57 dhcpd: failover peer dhcp_opt2: I move from startup to recover
2020-05-28T12:56:42 dhcpd: Server starting service.
2020-05-28T12:56:42 dhcpd: failover peer dhcp_lan: I move from communications-interrupted to startup
2020-05-28T12:56:42 dhcpd: failover peer dhcp_opt1: I move from communications-interrupted to startup
2020-05-28T12:56:42 dhcpd: failover peer dhcp_opt2: I move from recover to startup
2020-05-28T12:56:42 dhcpd: Sending on   Socket/fallback/fallback-net
2020-05-28T12:56:42 dhcpd: Sending on   BPF/em0_vlan1/f4:4d:30:6a:fb:9c/192.168.0.0/24
2020-05-28T12:56:42 dhcpd: Listening on BPF/em0_vlan1/f4:4d:30:6a:fb:9c/192.168.0.0/24
2020-05-28T12:56:42 dhcpd: Sending on   BPF/em0_vlan50/f4:4d:30:6a:fb:9c/192.168.50.0/24
2020-05-28T12:56:42 dhcpd: Listening on BPF/em0_vlan50/f4:4d:30:6a:fb:9c/192.168.50.0/24
2020-05-28T12:56:42 dhcpd: Sending on   BPF/em0_vlan10/f4:4d:30:6a:fb:9c/192.168.10.0/24
2020-05-28T12:56:42 dhcpd: Listening on BPF/em0_vlan10/f4:4d:30:6a:fb:9c/192.168.10.0/24
2020-05-28T12:56:42 dhcpd: Wrote 150 leases to leases file.
2020-05-28T12:56:42 dhcpd: Wrote 0 new dynamic host decls to leases file.
2020-05-28T12:56:42 dhcpd: Wrote 0 deleted host decls to leases file.
2020-05-28T12:56:42 dhcpd: For info, please visit https://www.isc.org/software/dhcp/
2020-05-28T12:56:42 dhcpd: All rights reserved.
2020-05-28T12:56:42 dhcpd: Copyright 2004-2020 Internet Systems Consortium.
2020-05-28T12:56:42 dhcpd: Internet Systems Consortium DHCP Server 4.4.2
2020-05-28T12:56:42 dhcpd: PID file: /var/run/dhcpd.pid
2020-05-28T12:56:42 dhcpd: Database file: /var/db/dhcpd.leases
2020-05-28T12:56:42 dhcpd: Config file: /etc/dhcpd.conf
2020-05-28T12:56:42 dhcpd: For info, please visit https://www.isc.org/software/dhcp/
2020-05-28T12:56:42 dhcpd: All rights reserved.
2020-05-28T12:56:42 dhcpd: Copyright 2004-2020 Internet Systems Consortium.
2020-05-28T12:56:42 dhcpd: Internet Systems Consortium DHCP Server 4.4.2
Note: Output from before LAGG implementation. Now lagg0_vlan**.

The DHCP in the other two interfaces work as expected. I can see successful DHCP negotiation for other interfaces in the log.

Thank you very much.

EDIT: I can see this block hit on the FW2 when I restart pfsync in the master:
BLOCK   wan      May 29 00:43:12   192.168.8.11   224.0.0.240   pfsync   Block private networks from WAN

This is an automatic rule. I removed it from the interface and added a PFSYNC rule with no effect to HA.

2

19.7 Legacy Series / FW Fails to apply rules in order and mixes descriptions

« on: September 08, 2019, 05:53:28 pm »

Today my opnsense started to act strangely. I have three local interfaces in my system (LAN,TLAN,IOT), with their fw rules and two groups, one that includes all the networks (ALL_LOCAL) and other that only includes user networks (INT).

I have observed the following:

• The rules in my system are not executing in the correct order. And depending on the alias names or group names, they stop working altogether.
• The fw log mixes information of two rules in one line. This is the more apparent and easy to see.
• The fw is blocking (is not processing all the rules until the default block) now and then for internet requests, that are valid.

In the capture you can see that the "allow" hits, that come from my "Allow multicast" rules, show in the description the text for the default WAN block rule. ??
Then in the second capture, by the time I wrote the message, descriptions have changed, but still from other rule. This time a NAT forward one.
Then sometimes, blocks some traffic to internet to some devices, It's normal HTTP/HTTPS traffic. I'll add a screenshot as it happens.
I've rebooted and still does this. What I can do? It is a totally mess.

3

Spanish - Español / [SOLUCIONADO] Ayuda con migración WAN de PPPOE a 4G

« on: July 31, 2019, 09:56:06 pm »

Hola. He abierto un hilo sobre este mismo tema en el foro inglés, pero como parece más muerto que los dinosaurios, pido ayuda en Español.

Alguien me puede ayudar a configurar una wan en paralelo, para una nueva línea 4G donde ya existe una ADSL?
Advierto que aunque sea 4G, va con el router de la compañía, que ya he comprobado que no se puede poner en modo bridge. Simplemente le he anulado el firewall, wifi y he definido una DMZ a la ip (fija) del opnsense en el nuevo interfaz. Pero no consigo que me resuelva nombres, a pesar de que aparentemente dice que está "online"

Agradeceré toda ayuda que podáis prestarme.

4

General Discussion / [SOLVED] Migrate PPPOE gateway to LTE (router) how?

« on: July 29, 2019, 07:58:13 pm »

The wan connection for my homelab is a plain DSL connection. Now I've accepted the offer of another operator that proposes a LTE internet connection. For that purpose they gave me the LTE router they offer for testing purposes.

As for now, I have a DSL router, in bridge, connected to the main switch and reaches the opnsense machine in my vmware server through a dedicated VLAN.
So I thought, Just bring up another VLAN and port in the switch for the new LTE router. As the router does not accept bridging, deactivate wifi, fw, dhcp, set static ip to 10.0.0.1 and set a DMZ to 10.0.0.2.

In the vmw server I gave another adapter to the opnsense machine tagged for this VLAN, and in opns, I created a new interface to it (call it WAN4G) with static ip 10.0.0.2 and a gateway with ipv4 for 10.0.0.1. I also activated monitoring, that in fact says me that the connection is alive.
But I could not make my router to get internet from this new GW.

Then I went to the failover/balancing documentation and done anything it said about gw groups and fw. So I disconnected my dsl cable and it seemed to switch to the LTE but there was no internet.
PPPOE gw dissapeared anywhere so I can't restablish connection without pulling cables.

If I leave both connected, I can see traffic on the graph for LTE interface but I dont have inet from it (only pppoe) as it "jumps" to lte I loose connectivity. I have to pull the cable of the new router.
Also some strange things occur, like I can't get into the mgmnt web if I'm not in a trunk port. In fact it seems that routing is not working as I can't get into any machine by its dns name or get into the ones that are on different VLAN (I have rules to allow some traffic)
And I can't even ping the own VLAN gateway address.

Can someone tell me how to do this? (to create a new wan connection)

Edit: Solved. Thx for nothing.

5

18.7 Legacy Series / OpenVPN setup, help plz!

« on: May 07, 2019, 10:05:36 am »

I'm trying to setup openvpn, but the fact is that I can't get it to work. I followed the guide but I found it so different, outdated I suppose, and the interface asks me for so many parameters that I don't know what to fill with. My problems are at the second half of parameters, server and client. The ones that doesn't have directions icon nor inline value suggestion.

I'm on 18.7 and I'm trying to setup a simple one client remote connection to my network for when I'll be out.
Thank you for any help you can give me!

6

18.7 Legacy Series / Firewall questions

« on: November 06, 2018, 02:25:40 pm »

I'm still trying to completely understand how the firewall configuration interface works. I thought that I had it already but happened something that I didn't expect. So if you're kind, I would like to ask two questions to better understand the base of it.

My network is composed of the three typical vlans plus wan interface. I'm using getty+stubby for tsl dns and content filtering through unbound. I've attached images of my defined basic rules for the trusted lan and the untrusted iot interfaces.

So my first question would be: Being at the TLAN interface, for example, the rules defined here are OUTPUT, or INPUT firewall rules (for the interface)? (And where they go?) I'm asking this because lately I've found myself writing more of output rules than input ones, that is the inverse I've done before.

The second question is: Taking into account the services I use, and looking at the image of the iot rules, the 3th rule should not be equal to the 4th+5th rules? If so, why I loose internet in the interface when I swap ones for the other? (like the image works)

Again, thanks.

7

18.7 Legacy Series / Stopped working

« on: October 20, 2018, 04:19:38 pm »

Today I left for a couple of hours and when a came back there was no network. I connected a screen to the router and it was showing something like the attached image. It was slightly different as the only thing shown was detecting the hard disk, then the memory cards reader, and nothing more.
I rebooted and changed the data cable port and the image is what I get now.

Is this normal?
Is also normal not counting at least with the ip leases for accessing the switch?

Enviado desde mi MI 5s Plus mediante Tapatalk

8

18.7 Legacy Series / Migration to VM

« on: October 16, 2018, 10:56:08 am »

I've been using opnsense in a dedicated box with two nics and multiple interface for vlans.
Now I'm trying to migrate to my esxi box. I created the VM with two nics for the moment, in the wan tagged group and in the management one. I installed it, updated, and after restoring the backup from the phys, it messes the esxi network. Not only the internal but even the machine ipmi. The rest of the network is ok.

So the question is, when doing such migration, how do you raise the opns VM services for the first time? and do you configure it the same as the physical one (ops, DHCP...)?
If I want to leave the physical as service backup, would leave it as an exact copy? Wouldn't then collide?
Thanks for your help.

9

18.7 Legacy Series / [SOLVED] ICAP protocol error

« on: September 24, 2018, 07:43:13 pm »

It seems that I messed the installation. I checked by error something (icap) in the Web proxy configuration and now I can't enter the GUI and slowly more and more inet pages show the "ICAP protocol error." page.
Is there anything I can change in the console so I can stop icap and bring back the system?
Help, please.

Edit: In console I see repeating "[bin/mongod] Preventing execution due to repeated segfaults" and the disk is continuously accessed. I dunno if that has relation.

10

18.7 Legacy Series / fwd: VLAN for IOT

« on: September 15, 2018, 11:20:52 am »

Hello, I posted the question bellow in the general forum, but as I was not having any reply (as with the previous one) I thought I could be posting in the wrong section. So I repost here in the hope someone can give me a hand.
For the lazy ones, the thing is that I'm trying to segregate the iot devices in a new vlan, and allow "some" communication with the lan so I can manage the iot server and he can do puntual things like ping a server to monitor it's status.

I'm in the process of replacing the network devices in my home lab. In my setup I have some servers, a big nas and a domotics system. I'm trying to setup a vlan for this system but I can't make it work. Perhaps someone can give me some advice on how to do it on opnsense.
This is my network structure:
Wan
ISP router (bridge) => opnsense box => dgs-1510 switch -> unifi AP / servers / workstations.
The network is 192.168.0.0/16 and I currently have the devices distributed in "sub networks" as 192.168.50.x for all the iot or ...1.x for the workstations.
I first created the vlan in the managed switch with tag 50 and then changed the WiFi ap so all the devices registered in the iot SSID would get tagged.
Then I came to the opnsense GUI and added the 50 vlan and created an interface linked to it derived from lan. I then went to DHCP and implemented another range for the vlan.

The problem I have is that all the iot WiFi devices don't get an ip. Does anyone knows if I'm forgetting a step? What can I do to diagnose the problem?
Thanks.

So far I managed to get the VLAN50 working. Previously I couldn't get the devices reach the dhcp, but it seems it was a mistake on the configuration for the AP and OPS switch ports. Now I have them like this:

Port	eth1/0/11
VLAN Mode	Trunk
Native VLAN	1 (Untagged)
Trunk Allowed VLAN	1-4094
Dynamic Tagged VLAN
Ingress Checking	Enabled
Acceptable Frame Type	Admit-All

I also changed the vlan50 interface net to 192.168.50.0/24 as after solving the above both dhcp servers started to collide.

So the problem I currently have is that I don't know how to make fw rules that allow communication between interfaces. I have tried anything from general to host rules and also floating with no success. Also I don't know how it's possible for devices that are in different sub-nets to talk between them.

I also think there is another issue in the vlan50 devices as I'm getting those entries in the fw log that seem that inet is being blocked even when I cloned the default allow rules of LAN into the vlan:

Interface   Time   Source   Destination   Proto   Label
IOT   Sep 15 11:19:19   192.168.50.52:44499   172.217.17.10:443   tcp   Default deny rule

11

General Discussion / Vlan for lot

« on: September 13, 2018, 11:46:01 pm »

I'm in the process of replacing the network devices in my home lab. In my setup I have some servers, a big nas and a domotics system. I'm trying to setup a vlan for this system but I can't make it work. Perhaps someone can give me some advice on how to do it on opnsense.
This is my network structure:
Wan
ISP router (bridge) => opnsense box => dgs-1510 switch -> unifi AP / servers / workstations.
The network is 192.168.0.0/16 and I currently have the devices distributed in "sub networks" as 192.168.50.x for all the iot or ...1.x for the workstations.
I first created the vlan in the managed switch with tag 50 and then changed the WiFi ap so all the devices registered in the iot SSID would get tagged.
Then I came to the opnsense GUI and added the 50 vlan and created an interface linked to it derived from lan. I then went to DHCP and implemented another range for the vlan.

The problem I have is that all the iot WiFi devices don't get an ip. Does anyone knows if I'm forgetting a step? What can I do to diagnose the problem?
Thanks.

Enviado desde mi MI 5s Plus mediante Tapatalk

12

Tutorials and FAQs / Wan over wifi

« on: September 05, 2018, 01:46:14 pm »

We're currently having thunderstorms in my location, that fried my ISP modem, the gateway/firewall and the switch. I already ordered a replacement managed switch and the ISP replaced the modem, that I'm temporally using as all-in-one.
I had a mini-pc laying around and have been trying to setup opnsense on it as the replacement gw/fw. As the rig only has one eth port and an internal wlan, I thought it could be used as a wifi bridge to the ISP modem. Only while storms continue and I receive all the replacement parts.

But I can't get it to work. It seems that the card is detected and I can assign it to the wan interface in the cli menu, but if I go through the gui, in interfaces->assign shows both LAN and WAN assigned to the wired adapter (re0) and I can't choose anything else on the WAN dropdown.
ath0_wlan0 clone is created from the ath0 device and I already setup the wpa_suplicant.

Code: [Select]

dmesg **************************************************************
ath0: <Atheros 9285> mem 0xfea00000-0xfea0ffff irq 16 at device 0.0 on pci2
[ath] AR9285 Main LNA config: LNA2
[ath] AR9285 Alt LNA config: LNA1
[ath] LNA diversity enabled, Diversity enabled
[ath] Enabling diversity for Kite
ath0: [HT] enabling HT modes
ath0: [HT] 1 stream STBC receive enabled
ath0: [HT] 1 RX streams; 1 TX streams
ath0: AR9285 mac 192.2 RF5133 phy 14.0
ath0: 2GHz radio: 0x0000; 5GHz radio: 0x00c0
...
wlan0: Ethernet address: xx:xx:xx:xx:xx:xx
wlan0: changing name to 'ath0_wlan0'
...
ath0: ath_legacy_rx_tasklet: sc_inreset_cnt > 0; skipping


Code: [Select]

log *************************************************************
Sep 4 22:11:10 sshd[73576]: Server listening on 192.168.0.2 port 22.
Sep 4 22:11:10 sshd[73576]: error: Bind to port 22 on fe80::5604:a6ff:fed1:ef0 failed: Can't assign requested address.
Sep 4 22:11:10 sshd[73576]: error: Bind to port 22 on fe80::5604:a6ff:fed1:ef0 failed: Can't assign requested address.
Sep 4 22:11:10 sshd[50160]: Received signal 15; terminating.
Sep 4 22:11:09 opnsense: /usr/local/etc/rc.initial.setports: ROUTING: skipping IPv6 default route
Sep 4 22:11:09 opnsense: /usr/local/etc/rc.initial.setports: ROUTING: creating /tmp/re0_defaultgw using '192.168.0.1'
Sep 4 22:11:09 opnsense: /usr/local/etc/rc.initial.setports: ROUTING: removing /tmp/re0_defaultgw
Sep 4 22:11:09 opnsense: /usr/local/etc/rc.initial.setports: ROUTING: setting IPv4 default route to 192.168.0.1
Sep 4 22:11:09 opnsense: /usr/local/etc/rc.initial.setports: ROUTING: no IPv6 default gateway set, assuming wan
Sep 4 22:11:09 opnsense: /usr/local/etc/rc.initial.setports: ROUTING: IPv4 default gateway set to lan
Sep 4 22:11:09 opnsense: /usr/local/etc/rc.initial.setports: ROUTING: entering configure using defaults
Sep 4 22:11:08 opnsense: /usr/local/etc/rc.initial.setports: Accept router advertisements on interface ath0_wlan0
Sep 4 22:11:08 opnsense: /usr/local/etc/rc.initial.setports: The command '/sbin/dhclient -c '/var/etc/dhclient_wan.conf' -p '/var/run/dhclient.ath0_wlan0.pid' 'ath0_wlan0'' returned exit code '1', the output was 'ath0_wlan0: no link .............. giving up'
Sep 4 22:11:00 kernel: ath0: ath_legacy_rx_tasklet: sc_inreset_cnt > 0; skipping
Sep 4 22:10:56 opnsense: /usr/local/etc/rc.initial.setports: The command '/sbin/ifconfig 'ath0_wlan0' up mode '' protmode '' -mediaopt hostap -mediaopt adhoc -hidessid -pureg -puren -apbridge -mediaopt turbo -wme authmode open wepmode off ' returned exit code '1', the output was 'ifconfig: unknown protection mode'
Sep 4 22:10:56 opnsense: /usr/local/etc/rc.initial.setports: The command '/sbin/ifconfig 'ath0_wlan0' mode ''' returned exit code '1', the output was 'ifconfig: SIOCSIFMEDIA (media): Device not configured'
Sep 4 22:10:56 opnsense: /usr/local/etc/rc.initial.setports: Accept router advertisements on interface re0
Sep 4 22:10:56 kernel: ifa_maintain_loopback_route: deletion failed for interface re0: 3
Sep 4 22:10:31 sshlockout[49679]: sshlockout/webConfigurator v3.0 starting up
Sep 4 22:10:31 opnsense: /usr/local/etc/rc.initial.setports: The command `/sbin/ifconfig 'ath0' up' failed to execute