Messages

1

24.7 Production Series / Re: Gateway falsely marked as down after restart. Works again after clicking "apply"

« on: July 31, 2024, 02:20:22 pm »

It's not just about bad etiquette. Google seriously DOES rate limit these pings. I have been debugging multiple boxes where using 8.8.8.8 for monitoring caused intermittent "gateway down" issues. And I am not alone.

I have also seen a local ISP with couple of thousands customers redirecting all DNS queries to Google -- because their servers broke down. Well, it did not work exactly well. https://developers.google.com/speed/public-dns/docs/isp

Awesome, thank you for the links. Especially the one from Google.
Would you say that using the monitor IP of each of my routers' WAN gateway is a more reliable way to check uptime?
To illustrate that:
My master router has a WAN gateway to 'a' and also monitors it by pinging 'a'
My backup router has a WAN gateway to 'b' and also monitors it by pinging 'b'

If I want the Master router to have a gateway to backup router('s internet), I would then use 'b' as the monitoring IP and vice versa?
Or are there services more suitable for this in your experience?

2

24.7 Production Series / Re: Gateway falsely marked as down after restart. Works again after clicking "apply"

« on: July 31, 2024, 02:01:47 pm »

These are DNS servers, not ping servers. I would suggest to not (ab)use public DNS for ping tests. They will eventually rate-limit or block pings when you do this.

I've heard this said at times. If you could, is there an article/source where instances like google or cloudflare say that they don't want people to ping for uptime?
The most I could find describe it as "bad etiquette", never heard about them rate limiting or blocking public IPs.

I could change it to the Gateway Monitor IP I find on my router's automatic gateway for now. 

3

24.7 Production Series / Re: Gateway falsely marked as down after restart. Works again after clicking "apply"

« on: July 31, 2024, 12:27:11 pm »

Have you tried using a monitor IP on the gateway? If you set a gateway to "monitor" the IP of the opposite gateway, does it still do this?

I am using a monitor IP, but since this is a gateway to another router, I use an internet address (1.1.1.0) to verify that the gateway (internet access on the other router) works as intended.

If i understand you correctly, I now set the monitor ip to the address of the target gateway instead. This does come back online as expected after a reboot/poweroff on the other router.
But this will not tell me if the gateway (ISP of the other router) is online. So the failover group won't work. 


I am suspecting something is up with the routing table.
With the gateway down, the routing table also seems to mess with the monitor IP entry (1.1.1.0) that is supposed to statically use the gateway as next hop.
However even with the gateway reported down I can still reach 1.1.1.0 and the traceroute shows that it now uses the master router's ISP instead of the gateway I statically imposed in both the gateway monitor and Routes configuration.

4

24.7 Production Series / Gateway falsely marked as down after restart. Works again after clicking "apply"

« on: July 30, 2024, 11:55:41 am »

I have a gateway to another router to use its ISP in a gateway group.
A problem I found is that when either of the routers is rebooted, the other router marks the gateway (between the routers) as down even after reboot is finished.

When I hit "apply" on the router's Gateway configuration page (without changing anything) the gateway is back online.

Is this a bug? Does anyone have time to reproduce this?

5

24.7 Production Series / Gateway only works one way (hear me out)

« on: July 29, 2024, 04:30:42 pm »

Disclaimer & Goal
I'm a novice hobbyist so do tell me if I make false claims.
I hope to figure out why my gateway interface between my two routers won't work both ways.
Some of the pictures may be a bit wide and require sideways scrolling to see fully.

Setup
2 Routers, each have their own IP from a different ISP, running a CARP setup that is between the poor man's CARP and regular CARP (middle class CARP?). It's not seamless failover, but as close as I could get (switches could not be used to split the ISP connections to each router).


To make sure both ISP connections can be used by the Master, a gateway interface is made between the routers (separate from pfsync interface). Followed by a gateway group and firewall policy routing to use the group.

Since they are "local" gateways, I use DNS addresses as Gateway Group monitor IPs to figure out whether the ISP on the backup router is reachable through the master (picture shown in "Gateway Configuration"). 

Gateway configuration: success on one way config

In order for the master to use its own (gateway interface) source_ip when it communicates with the other router, I need to declare the interface as a gateway in the interface configuration.



This gives me the expected result, a gateway to the internet of backup router.


But because I did not configure the gateway to my master's router on my backup router's interface, his connection is down



The Issue: Gateways down when configured on both ends
While one way works, if I set the gateway rules on the backup when they already set on master, neither connection will now work.
They are both considered "Down" on the gateway monitoring.
If I remove the Gateway Rules option on either router's interface, that side's gateway will work.   

My questions to the forum
1. Is this a bug, or simply how it works?
2. Should I simply create a new physical connection, this time configuring the gateway on router1's side and call it a day?

6

24.7 Production Series / Re: Rebellion theme - Dashboard

« on: July 29, 2024, 03:37:58 pm »

+1 This request

Also did notice the ADD Widgets dropdown (while running the Rebellion Theme) seems to pop down a layer (CSS Issue?) , preventing the ability to add new cards

Interim Solution: Switched over to Cicada theme in the interim as a workaround.

I noticed you can still use the arrow keys to move and enter to select/deselect options. 

7

Hardware and Performance / Re: Inter VLAN routing Iperf results 0.00 bits per second when target has multi LAN

« on: July 21, 2024, 04:17:05 pm »

Solved on netgate forum since I asked the same question there,
https://forum.netgate.com/topic/189287/inter-vlan-routing-iperf-results-0-00-bits-per-second-when-target-has-multi-lan/2
TLDR: asymmetric routing and misconceptions on my end on how management should work.

8

Hardware and Performance / [SOLVED] Inter VLAN routing Iperf results 0.00 bits/s when target has multi LAN

« on: July 21, 2024, 02:01:29 pm »

Disclaimer and purpose
I have replicated the same behavior on the other *sense.
Firewall settings were kept on default and an Allow all rule was put in place on all interfaces (to rule out incorrect firewall rules).

What I am after is an explanation and whether a setup like this is non-standard.
I honestly feel pretty silly for not being able to find the explanation, the closest answer I found was on the OPNsense forum https://forum.opnsense.org/index.php?topic=35157.0 but the eventual conclusion was that the person just gave up on VLANs.

Setup
I ask you to follow me on this simple topology found below.
The server has public facing services on VLAN10.
Management of the server and the services is done on VLAN100


The problem: Zero upload speed, but normal download speed

I noticed secure copy (scp) could not upload files to VLAN10 despite being able to connect and use ssh.
Iperf3 then gave me a very weird result: 0.00 bits/s

Code: [Select]

iperf3 -c 192.168.10.10
Connecting to host 192.168.10.10, port 5201
[  5] local 192.168.1.10 port 58322 connected to 192.168.10.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   128 KBytes  1.05 Mbits/sec    2   1.41 KBytes       
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   128 KBytes   105 Kbits/sec    5             sender
[  5]   0.00-10.04  sec  65.0 KBytes  53.1 Kbits/sec                  receiver

iperf Done.
There are three ways I get normal speeds
1: Removing the VLAN100 interface from the dual LAN
This suggests to me that the upload is being (partially) incorrectly routed.
I also noticed that if I changed VLAN100 to a different VLAN (20) on the single-LAN or mult-LAN side (but not both), the speed also returns to normal. Meaning that as long as VLAN100 is present on both devices, the single VLAN device cannot upload to the server's VLAN10.

So multi LAN is not the real issue, the real issue is multi LAN where I try to connect from a device whose VLAN is also setup on the target machine but is not the target for the file transfer.   
What I just don't understand is why.
 
2: Reverse the iperf3 test (iperf3 -s)
Seems logical given the previous point. The target is the device with only one (V)LAN.

3: Turning off the firewall
Of course very silly, but it at least tells me that there is some traffic rejection going on, though I don't see anything in my Firewall Live View.

Conclusion
As and addition to the disclaimer, I think I just fundamentally misunderstand something that makes this setup act this way. Is it uncommon for a managment VLAN to still have access to a service VLAN?

9

High availability / [SOLVED] No upload speed to backup router in HA setup due to Firewall

« on: May 25, 2024, 04:21:25 am »

TLDR: Was not related to HA, but to asynchronous routing between servers.

Hello! I have a pretty unique HA firewall problem, I'm hoping someone can help me out with this.

The issue summarized
Iperf tests revealed that my upload speed to the backup router is 0.00 bits per second. Download speed is unaffected.
Turning off the firewall does give me the expected speeds.
So it must be a firewall issue. But I can't figure out the cause. I am suspecting state violations, but I am not experienced enough to know for sure.

The setup
I drew a basic overview of my network in the attachment.
I have two ISPs and therefore 2 IPs, not sufficient for a standard CARP, but sufficient for a "poor man's CARP".
In addition to the hardware redundancy,  I created a "gateway" between the routers for the Gateway group configuration, so that I can access and fall back to the ISP of the backup router.
Due to circumstances, I cannot place a dumb switch between the ISP connections to allow both routers to get each other's ISP address. Which is why I opted for this setup.

Setup disclaimer
I am 100% certain that my CARP and Gateway configurations are valid. All hardware has been switched out and works as intended.
If requested, I will provide screenshots of the configuration.

The odd behavior
When using the ISP of my backup router, I noticed that my upload speed went down the drain: 0.01 bits per second on online speedtests.
Checking locally with iperf, the results are similar. Connecting with my laptop (VLAN100) to any ip (except the VLAN100 ip) of the backup router gave the following result.

Code: [Select]

[  5] local 10.10.100.150 port 44652 connected to 10.10.10.254 port 40744
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   128 KBytes  1.05 Mbits/sec    2   1.41 KBytes       
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   128 KBytes   105 Kbits/sec    5             sender
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  receiver

From what I understand, iperf defaults to uploading data from the client to the server, thus testing upload speed.
If I use the -R (reverse) argument to test download speed, I get the expected speed.

 The cause: The firewall... somehow
I got the hint from an old forum post https://forum.opnsense.org/index.php?topic=35157.0
Sadly, the user never replied back with a solution. But indeed, when I turned off the Master router's firewall, the upload speed went back to normal.

It also explains why the speed is normal when performing iperf tests on the backup router's VLAN100 address, because the traffic does not pass through the master router's firewall. 

A new hint?
I have not created any firewall rules that prevent communication between VLANs 100 and 10. So I suspected some sort of state violation that may cause the packets to be dropped.
In the live view, I found 1 such state violation entry. But this entry only appeared in the Live View a couple seconds after the iperf test concluded.

10

Tutorials and FAQs / Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

« on: April 27, 2024, 01:34:05 am »

Has anyone tried this HAProxy setup in combination with CARP?
The short of my problem is that my Backup node refuses to start the HAProxy service after syncing the settings to it. The error I receive when trying to start it from the shell is a mere "Haproxy failed to start" without additional information.

Clearing the configuration from the Backup will make HAProxy run again, I thought of meticulously adding each setting one by one as it is on the Master until I see where it fails, but it's become quite the config.

Are there things that should be configured differently per system with HAProxy on CARP?
I should note that all other (CARP) functions works as intended.

EDIT: Fixed it,
Running haproxy -d -f /usr/local/etc/haproxy.conf on the shell revealed that my SNI couldn't listen on port 80 because my GUI still allowed connections to HTTP through port 80.
I disabled HTTP GUI in System: Settings: Administration and checking "Disable web GUI redirect rule"