Topics

Hi there,

Appreciate any thoughts on this ....

When PPPoE (running on a VLAN) disconnects, the only way for me to recover it is to restart the firewall.  The /var/logs/ppps files just shows:

Code Select Expand

"<30>1 2023-11-14T16:16:04+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="581"] Multi-link PPP daemon for FreeBSD
<30>1 2023-11-14T16:16:04+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="582"]
<30>1 2023-11-14T16:16:04+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="583"] process 1291 started, version 5.9
<30>1 2023-11-14T16:16:04+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="584"] web: web is not running
<30>1 2023-11-14T16:16:04+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="585"] [wan] Bundle: Interface ng1 created
<30>1 2023-11-14T16:16:04+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="586"] [undefined] GetSystemIfaceMTU: SIOCGIFMTU failed: Device not configured
<30>1 2023-11-14T16:16:04+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="587"] [wan_link0] Link: OPEN event
<30>1 2023-11-14T16:16:04+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="588"] [wan_link0] LCP: Open event
<30>1 2023-11-14T16:16:04+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="589"] [wan_link0] LCP: state change Initial --> Starting
<30>1 2023-11-14T16:16:04+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="590"] [wan_link0] LCP: LayerStart
<30>1 2023-11-14T16:16:04+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="591"] [wan_link0] PPPoE: Set PPP-Max-Payload to '1500'
<30>1 2023-11-14T16:16:04+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="592"] [wan_link0] PPPoE: Connecting to ''
<30>1 2023-11-14T16:16:13+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="593"] [wan_link0] PPPoE connection timeout after 9 seconds
<30>1 2023-11-14T16:16:13+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="594"] [wan_link0] Link: DOWN event
<30>1 2023-11-14T16:16:13+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="595"] [wan_link0] LCP: Down event
<30>1 2023-11-14T16:16:13+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="596"] [wan_link0] Link: reconnection attempt 1 in 4 seconds
<30>1 2023-11-14T16:16:17+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="597"] [wan_link0] Link: reconnection attempt 1
<30>1 2023-11-14T16:16:17+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="598"] [wan_link0] PPPoE: Set PPP-Max-Payload to '1500'
<30>1 2023-11-14T16:16:17+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="599"] [wan_link0] PPPoE: Connecting to ''
<30>1 2023-11-14T16:16:26+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="600"] [wan_link0] PPPoE connection timeout after 9 seconds
<30>1 2023-11-14T16:16:26+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="601"] [wan_link0] Link: DOWN event
<30>1 2023-11-14T16:16:26+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="602"] [wan_link0] LCP: Down event
<30>1 2023-11-14T16:16:26+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="603"] [wan_link0] Link: reconnection attempt 2 in 4 seconds
<30>1 2023-11-14T16:16:30+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="604"] [wan_link0] Link: reconnection attempt 2
<30>1 2023-11-14T16:16:30+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="605"] [wan_link0] PPPoE: Set PPP-Max-Payload to '1500'
<30>1 2023-11-14T16:16:30+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="606"] [wan_link0] PPPoE: Connecting to ''
<30>1 2023-11-14T16:16:39+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="607"] [wan_link0] PPPoE connection timeout after 9 seconds
<30>1 2023-11-14T16:16:39+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="608"] [wan_link0] Link: DOWN event
<30>1 2023-11-14T16:16:39+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="609"] [wan_link0] LCP: Down event
<30>1 2023-11-14T16:16:39+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="610"] [wan_link0] Link: reconnection attempt 3 in 1 seconds
<30>1 2023-11-14T16:16:40+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="611"] [wan_link0] Link: reconnection attempt 3
<30>1 2023-11-14T16:16:40+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="612"] [wan_link0] PPPoE: Set PPP-Max-Payload to '1500'
<30>1 2023-11-14T16:16:40+00:00 fw00.localdomain ppp 1291 - [meta sequenceId="613"] [wan_link0] PPPoE: Connecting to ''"

... over and over. 

It doesn't matter if I try to reload/reconnect the PPPoE interface, kill mpd5, reload all services, nothing will bring it back, other than a reboot. 

- All Intel NICS.
- em0 and the VLAN interfaces are up
- tcpdumping on the em0 interface just shows:

Code Select Expand

listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:24:59.901275 PPPoE PADI [Host-Uniq 0x8038224300F8FFFF] [Service-Name] [PPP-Max-Payload 0x05DC]
16:25:01.900485 PPPoE PADI [Host-Uniq 0x8038224300F8FFFF] [Service-Name] [PPP-Max-Payload 0x05DC]
16:25:05.900490 PPPoE PADI [Host-Uniq 0x8038224300F8FFFF] [Service-Name] [PPP-Max-Payload 0x05DC]
16:25:09.902668 PPPoE PADI [Host-Uniq 0x40DB564300F8FFFF] [Service-Name] [PPP-Max-Payload 0x05DC]
16:25:11.902485 PPPoE PADI [Host-Uniq 0x40DB564300F8FFFF] [Service-Name] [PPP-Max-Payload 0x05DC]

I've tried:

- Disconnect/reload PPPoE via the UI
- Killing mpd5
- Enabling/disabling promiscuous (although I don't believe this is required for me...)
- Enabling/disabling 'prevent interface removal'
- Tried leaving MTU default, rather than 1508 on the PPPoE (for 1500 calculated)
- Tried changing the physical WAN port on the firewall
- Tried unplugging and re-plugging the ethernet cable (to force link loss, recovery)
- Reset all tuneables

Is there anything that is generated differently, for example a 'fake' MAC or some form of ID (Host-Uniq?), that is different after a reboot but not different after a reload of PPPoE? 

I'm at the stage now, other than trying to source different hardware, where I'm wondering if it's the ISP that's dropping the connection and blocking reconnects - for whatever reason - but after a reboot, the ID/MAC appears different so succeeds?!? Clutching at straws now!

... guess next step is to hook up the ISP router again and run this in the DMZ behind it and see if it still occurs.

Good Morning,

So, I had some issues with one of my fibre providers and of course they wanted me to connect  their router - rather than using my own.

I re-configured the WAN interface on opnsense to be 'em0' and regular DHCP - rather than PPPoE over a VLAN - set a manual gateway to the ISP router, made some firewall changes.  Before doing so, I took a backup.

This morning I went to restore the original setup, as the ISP router didn't resolve the problem.  Shock horror.

However, upon restoring the config I took before, the firewall rebooted, I seem to end up with a 'mash up' of the changes I made and the original config?

For example, the WAN interface is still em0/DHCP (although PPPoE has been restored, it's not assigned to the WAN) and the various firewall rules are not back as they were (had some gateway rule changes).

If I view the saved backup XML file, the em0/DHCP WAN and the manual gateway I made, are NOT present in the backup config.  So I was hoping/assuming, they shouldn't be in the restored config!

Is there a restore log anywhere?  So I can see if it's puking on something in particular?  Nothing really 'fancy' just a basic firewall.

I tried a restore via the UI and also via option 13 on the CLI menu.  In a bit of a mess at the moment, my fail safe seems to have failed - anyone any ideas? 

Thanks,

Recently upgraded my Virgin Media connection to Gig1 and with it (unfortunately) I was provided with a Hub 4 - from day 1, it has been a PITA.  It took me hours and hours to finally get an IP address from modem mode to opnsense.

Running the device in Modem Mode, it is extremely picky about when/if it will let opnsense have an IP - or the DHCP servers are - certainly with the default DHCP settings, even though the opnsense defaults request more frequently than the FreeBSD defaults.

There are various reports of problems, with people using opnsense, pfSense, Asus, you name it.  It seems to be 'pot luck' whether the device behind the modem gets an IP address or not, the suggested steps of:

- Power off the modem
- Restart your main firewall/router
- Give it a few minutes
- Power on the modem

.. seems to work in some cases, but not others.  I do firmly believe there is still an element of luck to whether this works or not. 

I read somewhere that if the DHCP request doesn't complete in a 15 second window, after the modem has booted, it basically ignores all other requests after.  This was quoted from a Virgin engineer, so who knows....

I thought I had cracked it last time I had problems, when I filtered 192.168.100.254 in 'Reject Leases From' - otherwise opnsense ends up getting a 192.168.100.x address instead.  Note, previous modems you needed to filter 192.168.100.1 (I believe).

Running tcpdump, filtering for DHCP requests/replies, the requests are being made, but just no response or not a completed response:

Code Select Expand

tcpdump -i eth0 port 67 or port 68 -e -n -vv

Last night the UPS on my cable modem died, opnsense failed over beautifully to 4G and until I heard a whining noise on my way to bed coming from the garage I didn't realise it had failed (Muted PushOver notifications after 10PM).  So I removed the UPS, powered on the cable modem and went to bed - this morning, it still hadn't got an IP and was still running on 4G.

I tried restarting the modem, and opnsense, using the basic procedure above that is reported to (sometimes) work.  But it didn't.

In the end, I discovered that Asus now has some '12Hz DHCP' option - to supposedly fix similar issues, where the device doesn't get an IP - which requests an IP 12 times a second (12Hz)?!?!

https://www.asus.com/my/support/FAQ/1043591/

Frustrated, I ended up putting '1' in every box in the DHCP options for the WAN interface - along with filtering 192.168.100.254 from DHCP replies - to try to make DHCP requests as frequent as possible.  Rebooted the Hub 4....and it got an IP address first time.

... I don't really want to test this again, just yet, but will update this post with any further developments next time I come across it.

Aside from this, does anyone have any idea how to replicate the Asus 12Hz option?  I do see some mention that others with Starlink and other cable modems also need to use this feature, so it doesn't seem to be Virgin specific.

Just adding this here, for anyone in a similar situation...

Was previously running a Qotom J1900 for the last 7+ years, happily on 21.7.  Recently upgraded the firewall, now with an i7 9700 CPU.

Upgrade was easy enough, export the config, edit the interface names for the new device in the export file, import into the new box.  Initially started out with 21.7 on the new box.

But, whilst using PowerD with the J1900 was not a problem, with the i7 9700 it caused it to crash usually under a reasonably small amount of load - on 21.7 - fans would go to 100% and required physically power cycling. 

Turning off PowerD resolved the issue, but the CPU frequency (dev.cpu.0.freq) would always be at 1400 - even using 'stress' the CPU would seemingly not scale.

Upgrading to 22.1 which I believe uses hwpstate_intel instead for power management, resolves the issue. 

dev.cpu.x.freq when running stress shows it is also using up to the 'Turbo Boost' frequency '4497', so the CPU can work at maximum when required and drop back down (dev.cpu.0.freq: 897). 

Performance vs power can be tuned with dev.hwpstate_intel.*.epp

I did come across a FreeBSD post, where they were essentially saying it was legacy and should probably be removed - so I haven't attempted PowerD again, but will for testing again at some point.  So it seems 'ok' for older CPUs, but for more modern CPUs PState seems to be the way to go.

Anyone reading this, if you're running the Shaper, on 21.1.8, could you test something for me?

- Edit the Pipe
- Enable Advanced
- Set Queue slots, to a value other than 50 (the default)
- Apply

On the CLI, if you run:

Code Select Expand

ipfw pipe show

Do you still only see the default queue lot size, of 50?

Code Select Expand

q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail

I had a look in the /tmp directory, to see if I can see the ruleset, but it didn't seem to be there.

I'm pretty sure in 21.1.7 the queues were set correctly based on the value entered in the UI.

Hi there,

I've got a few ingress NAT rules, port forwards, however I can't see how I can specify 'synproxy' as part of this? 

The rules that are automatically created are not editable, to locate the Advanced setting - and potentially enable synproxy there - and it doesn't seem to be possible to set this on the parent NAT rule?

Cheers,

Hi there,

Running 19.1.3 making a change to Suricata/IDS, for example changing from Default -> Hypersync pattern matcher, the configuration does not automatically trigger an xmlrcsync of the config to the secondary node.   

I cannot see any attempt for it to do so, I see the Suricata config reload/regeneration in the logs, but no automatic sync.

If I go to Firewall -> HA -> Status -> Synchronize config to backup, the change is replicated.  But it does not seem to trigger automatically.  I would assume it should do?  Is anyone else seeing this?

Other changes, firewall rules for example, DO automatically trigger a change.

I have the following HA setup, with multi-WAN, I'm hoping one of your clever people might be able to suggest a workaround.  Albeit probably a hacky work around, as I realise this is a hacky setup!

Node A
WAN1: 10.0.0.1/30 (RFC1918)
WAN2: 37.x.x.1/27 (Default GW)

Node B
WAN1: 10.0.0.2/30 (RFC1918)
WAN2: 37.x.x.2/27 (Default GW)

CARP:
WAN1 VIP: 78.x.x.1/30 (Single IP from ISP)
WAN2 VIP: 37.x.x.3/27

IP Aliases:
80.x.x.1/27 - WAN1 (routed by ISP to 78.x.x.1). Gateway set to WAN1 gateway (78.x.x.2/30)
80.x.x.2/27 - WAN1 (routed by ISP to 78.x.x.1). Gateway set to WAN1 gateway (78.x.x.2/30)
80.x.x.3/27 - WAN1 (routed by ISP to 78.x.x.1). Gateway set to WAN1 gateway (78.x.x.2/30)
...
80.x.x.30/27 - WAN1 (routed by ISP to 78.x.x.1). Gateway set to WAN1 gateway (78.x.x.2/30)

Gateways, monitoring disabled for both:
WAN1: 78.x.x.2/30
WAN2: 37.x.x.30/27 (Default GW)

- Single IP from the ISP for WAN1, which is configured as a CARP VIP with RFC1918 on Node A and Node B
- A further /27 subnet is routed to the single CARP IP for WAN1
- IP aliases are set up for the same VHID as WAN1 CARP
- WAN2 has the default gateway, all IP addresses are externally reachable/routable (non-rfc1918)

When I perform a fail over from Node A -> Node B, with state synced, the IP aliases fail over correctly.  However, the egress packets for 80.x.x.0/27 IP aliases are routed out of the default WAN2 gateway once failed over to Node B.  If I clear the state on the firewall, things then sort themselves out.

Presumably as there is no routing table entry for the WAN1 CARP gateway, it takes the default route after failover - which is egress via WAN2.  When I clear the state, it then routes correctly via pf.  If I disable state sync, the fail over happens, state is lost, but routing is correct - i.e in and out of WAN1 for 80.x.x.0/27.

Is the work around to just not sync state?  Or is there a way for state to be synced, but for it to route correctly egress via WAN1 for 80.x.x.0/27 immediately after fail over?

Quick overview for installing the check-mk agent - brain dump whilst I still have it in my shell history - I saw this was mentioned once before some time ago:

https://forum.opnsense.org/index.php?topic=1310.0

1. Create a new directory:

Code Select Expand

mkdir -p /opt/bin

2. Download the agent:

Code Select Expand

curl "http://git.mathias-kettner.de/git/?p=check_mk.git;a=blob_plain;f=agents/check_mk_agent.freebsd;hb=HEAD" -o /opt/bin/check_mk_agent

3. Make it executable:

Code Select Expand

chmod +x /opt/bin/check_mk_agent

4. Install bash and statgrab

Code Select Expand

pkg install libstatgrab bash

5. Add the following to /etc/inetd.conf

Code Select Expand

check_mk  stream  tcp nowait  root  /opt/bin/check_mk_agent check_mk_agent

6. Add the following to /etc/services:

Code Select Expand

check_mk        6556/tcp   #check_mk agent

7. Add the following, modify monitoring.server.ip.address as required, to /etc/hosts.allow

Code Select Expand

# Allow nagios server to access us
check_mk_agent : monitoring.server.ip.address : allow
check_mk_agent : ALL : deny

8. Start inetd

Code Select Expand

/etc/rc.d/inetd onestart

9. Add firewall rules as required to access tcp 6556

To Do: Make it start on boot, investigate a potential plugin to make it survive (major?) upgrades

Hi there,

I have a cable connection with a dynamic IP, for that reason I also have an L2TP tunnel from another provider that provides static IP addresses over the tunnel.  At some point after 17.7.7 (I think, or there about) ICMP to the WAN IPs over the L2TP tunnel stopped providing a response, immediately after an upgrade.  Currently running 17.7.12.

I have the following rules permitting ICMP:

Code Select Expand

pass in  quick on l2tp1 reply-to ( l2tp1 c.c.c.c )  inet proto icmp from {any} to {(l2tp1)} icmp-type {echoreq} keep state label "USER_RULE"
pass in  quick on l2tp1 reply-to ( l2tp1 c.c.c.c )  inet proto icmp from {any} to $Host_Guest_WAN_IP icmp-type {echoreq} keep state label "USER_RULE"

The below is a dump from the l2tp1 interface, showing the response - a.a.a.a is an external server, b.b.b.b the WAN IP on the L2TP tunnel/interface:

Code Select Expand

12:05:24.304934 IP a.a.a.a > b.b.b.b: ICMP echo request, id 5384, seq 58, length 40
12:05:24.304959 IP b.b.b.b > a.a.a.a: ICMP echo reply, id 5384, seq 58, length 40
12:05:24.304970 IP a.a.a.a > b.b.b.b: ICMP echo request, id 5384, seq 59, length 40
12:05:24.304994 IP b.b.b.b > a.a.a.a: ICMP echo reply, id 5384, seq 59, length 40
12:05:24.305005 IP a.a.a.a > b.b.b.b: ICMP echo request, id 5384, seq 60, length 40
12:05:24.305030 IP b.b.b.b > a.a.a.a: ICMP echo reply, id 5384, seq 60, length 40

The below is a capture from the L2TP providers end (they provide the option to create 10 second pcaps from their portal), the response does not make it down the tunnel.

Code Select Expand

97 9.367539 a.a.a.a b.b.b.b ICMP 106 Echo (ping) request  id=0x10ea, seq=5/1280, ttl=57 (no response found!)

So far, I have tried:

- Disabling 'reply-to' on the rule.  When this happens the response does not go down the L2TP tunnel, i.e the response is not seen in the l2tp1 capture
- Tried setting a gateway to the L2TP tunnel on the rule, rather than default.  Did not resolve.

Only thing I haven't tried so far, as I need to get a maintenance window from the other half, is disabling shared forwarding - is this likely going to help? 

But as I say, this stopped post a 17.7.x upgrade, I think when I upgraded from 17.7.7 to 17.7.10.

Cheers,

Ed

So, followed a few of the FQ_Codel guides on here, I believe I had it working on an earlier 17.7 release - on the current, 17.7.7_1 I don't seem to be able to.

Something I'd just like to clarify, presumably I should see the Rules/ueues that I configure in the Traffic Shaper section, in 'ipfw -a list'?  I don't, if I should I can't for the life of me work out why.  ipfw rules below:

Code Select Expand

root@fw00:~ # ipfw -a list
00100       0          0 allow pfsync from any to any
00110       0          0 allow carp from any to any
00120       0          0 allow ip from any to any layer2 mac-type 0x0806,0x8035
00130       0          0 allow ip from any to any layer2 mac-type 0x888e,0x88c7
00140       0          0 allow ip from any to any layer2 mac-type 0x8863,0x8864
00150       0          0 deny ip from any to any layer2 not mac-type 0x0800,0x86dd
00200       0          0 skipto 60000 ip6 from ::1 to any
00201      44       9156 skipto 60000 ip4 from 127.0.0.0/8 to any
00202       0          0 skipto 60000 ip6 from any to ::1
00203       0          0 skipto 60000 ip4 from any to 127.0.0.0/8
01002      36       3560 skipto 60000 udp from any to 10.8.6.254 dst-port 53 keep-state
01002     117      13994 skipto 60000 ip from any to { 255.255.255.255 or 10.8.6.254 } in
01002     160      21192 skipto 60000 ip from { 255.255.255.255 or 10.8.6.254 } to any out
01002       0          0 skipto 60000 icmp from { 255.255.255.255 or 10.8.6.254 } to any out icmptypes 0
01002       0          0 skipto 60000 icmp from any to { 255.255.255.255 or 10.8.6.254 } in icmptypes 8
01003       0          0 skipto 60000 udp from any to 192.168.3.254 dst-port 53 keep-state
01003       0          0 skipto 60000 ip from any to { 255.255.255.255 or 192.168.3.254 } in
01003       0          0 skipto 60000 ip from { 255.255.255.255 or 192.168.3.254 } to any out
01003       0          0 skipto 60000 icmp from { 255.255.255.255 or 192.168.3.254 } to any out icmptypes 0
01003       0          0 skipto 60000 icmp from any to { 255.255.255.255 or 192.168.3.254 } in icmptypes 8
65535 9056022 8639833830 allow ip from any to any

I've follow the RickNY guide, below, multiple times, line for line, but I don't actually see any reduction in bufferbloat, nor in the downstream bandwidth (even if I set it to something stupidly low) suggesting something isn't matching.

https://forum.opnsense.org/index.php?topic=3758.0

Screenshot in the below post shows 'queue' rules in ipfw:

https://forum.opnsense.org/index.php?topic=4665.msg18072#msg18072

I don't seem to have these in my 'ipfw -a list' above, no matter what 'Rules' I configure in Firewall -> Traffic Shaper -> Settings -> Rules:

   

Code Select Expand

11 WAN ip 10.8.6.0/24 any DownQueue
21 WAN ip any 10.8.6.0/24 UpQueue