Messages

It's been running good so far.

Nice! I see you're using DAC cables (which I couldn't), did you try any single SFP+ modules  (no DAC) ?!?!

Could conditional forwarders bet setup in Unbound and the DC so that domain clients can use the firewall as DNS unless they are looking for a service from the DC? It's hacky and backwards, but would this work?

Perfectly valid and prefered IMHO (but that's arbitrary, as almost all IT design ;-)). AD heavily depends on DNS, but "the Internet" was here decades before, so you don't have to hijack your main DNS infratructure by MS AD DNS, just as an add-on (Query Forwarding).

And that said, can Unbound use conditional forwarders? Is that the domain override tab? Setting these Windows to Windows is trivial but I'm not sure about Unbound or any of the other DNS server available in OPN.

Services: Unbound DNS: Query Forwarding would be the right place, be sure to not only forward your "primary" AD domain, but _all_ your reverse lookups too, so "100.168.192.in-addr.arpa", "101.168.192.in-addr.arpa", etc should be forwarded to the AD DNS (Kerberos, etc).

Beware that forwarding your AD DNS to your primary DNS infrastructure (OPNsense) assumes some redundancy, if your single forwarding OPNsense box is down, your (MS AD) DNS is down (from a client perspective).

Like Patrick M. Hausen already confirmed, LACP is working without any issues with latest OPNsense and any version I can remember (I'm using Juniper switches).

No experience with FS switches, but I noticed a inconsistency in your comparison with a working and non-working example in this post https://forum.opnsense.org/index.php?topic=44338.msg221412#msg221412 .

You've assigned your LAGG to the LAN interface of OPNsense, but it also looks like you have CARP enabled which isn't (or doesn't look like) with your PFsense config:

NON-working OPNSense:

...
carp: MASTER vhid 3 advbase 1 advskew 0
              peer 224.0.0.18 peer6 ff02::12
...

As you have a (low level) LAGG interface problem and it seems also HA (CARP), I would rule out the whole HA stuff first. In other words, try to configure a single vanilla OPNsense box without any bells and whistles and try to configure the LAGG in this setup (which should work), only after that continue with any HA stuff to keep a clear overview.

And now my question is: Does it have any downsides, from a security perspective, if I dont assign/create an Interface for the igb4 (step 1) AND the VLAN (step 2)?

No. The only reason you would assign the interface is if you want to change any property, like the MAC address or MTU.

Although there's already an option in OPNsense to calculate / compensate for the MTU size with PPP connections, it assumes PPPoE over a raw ethernet interface (1500 (ethernet) + 8 (ppp) = 1508). Where I like to set the raw parent at 1512 (1500 (ethernet) + 8 (ppp) + 4 (vlan) = 1512), for this I need to enable the parent to set these values. I didn't look into recent OPNsense versions regarding automagic PPP MTU settings (I beiieve introduced in 24.07), still using my original install from years ago (and upgraded it with every release instead of fresh install).


@cookiemonster

A slightly updated version with screenshots of a more recent OPNsense version (with the changed internal VLAN assignment logic)  is on my todo list. Will be busy for some more weeks, maybe a nice task when I try to avoid my mother-in-law during christmas dinner... ;-)

Any Qotom Q20331G9 owners who may share their experience with this unit ?

I've seen three units of the rack (1u) version, all failing to recognize a module in the top left SFP+ cage (ix3). Tried all usable suspects, genuine Intel, Juniper, Cisco and FS (Intel coding) modules and DAC, all no go in that single slot, other slots do work.

The platform is wel known, used for many years by SuperMicro in their SYS-E300-9A systems (A2SDI motherboard). Although Qotom using a slightly newer/faster Atom R variant, the platform itself is almost 6 years old. The idea is/was nice, a cheap (1/3 price of SuperMicro in 2019) multi 2.5/10Gb platform with decent power consumption.

Because the lack of documentation, a rather shady support page with some unspecified firmware versions and inconsistent SFP behaviour it's not a platform I would recommend.

What's your experience with the SFP+ cages ?!?!

From a security perspective: Is it important to leave one Interface assignment (in your case WAN_1) on the physical and untagged network interface (in your case igb4). Maybe to block untagged traffic or so?!
Or could you also reassign your [WAN] Interface to the pppoe device?

I'm not sure if I understand your question completley, but a pppoe device is assigned TO an interface. Because this HowTo is about PPPoE in combination with a provider based service VLAN, the pppoe device is assigned to the VLAN interface derived from the physical WAN interface. It's basically like a layered cake:

You start with a physical WAN interface, in this HowTo that's "igb4". Because multiple WAN interfaces are used (two different providers) this physical interface is renamed to WAN_1 (the other one WAN_2, but out of scope in this example). This interface is specifically enabled in OPNsense to assign custom MAC and MTU values, but at least the MTU can be calculated in other ways with recent OPNsense versions. (instructions are made on older version).

Because you need to match the VLAN of your provider you now need to create a (tagged) VLAN interface on top of the igb4 / WAN_1 interface AND name it too, in this example the specific VLAN interface is named WAN_1_FTTH for generic ISP IP services.

The actual PPPoE interface (pppoe0) is configured on top of this new VLAN interface (WAN_1_FTTH) and named/assigned automatically by OPNsense. In the example there's also a second VLAN interface on the same physical WAN interface for IPTV services, this doesn't use PPPoE but raw Ethernet/IP.

So the full example ends up with three interfaces that are ISP oriented, the raw ethernet WAN interface and two VLAN interfaces, one using PPPoE for generic IP services (ie. Internet) and one standard VLAN IP interface. Yes, the raw ethernet WAN interface can be seen as an untagged interface towards your ISP, but it's unnumbered (doesn't have an active IP address assigned) so can't be used in > Layer 3 communication. I can't think of a Layer 2 attack surface that compromises the security of the WAN interface in this scenario, except for some L2 DOS stuff, but that would compromise the core service of my ISP, they probably detect that earlier than I can/do.

So your layered PPPoE VLAN Cake looks like this:
Raw Ethernet Interface -> VLAN Interface -> PPPoE Interface -> IP

There is a sequence number, correct, but since you can "smoke ping" and there is no foreseeable sequence in which packets get transmitted, you cannot decide based on the sequence number (or, in other words: even if you used something akin to a "catch range", it would be just as good as an abitrary time range).

You can't decide anything indeed, it's the "related" behaviour baked into the ICMP network code. As explained in the link above, if your infrastructure somehow sends all ICMP with id=0x0001 you get exactly the behaviour described by the OP. So you need to fix your infra (ie the sequence id's) in that case, not something your firewall (OPNsense et all) can fix.

With ICMP, this is different. There is no port, only an ICMP subtype. Other than that, there are only (src_ip, dst_ip). Thus, you can only decide based on "soft" factors ("related"), like if within the last few seconds, you saw ICMP traffic between both parties that might explain why another ICMP packet is seen (and being passed).

Let's see the trace, OP is most probably hitting something similar as this:

https://forum.opnsense.org/index.php?topic=36152.msg176859#msg176859

As you already expained there's no state in ICMP buth there is relation based on sequence id, if these are non-unique you get the scenario as described. That's why it's interesting to see if this behaviour can also be seen with TCP (probably not).

I read on the different posts that with OPNSense it is actually default behaviour that when B is authorized to A, A can reply to B, and I could test it works well.

Correct, OPNsense is a stateful firewall.

However, my issue is the following when one does step by step :
1. first, A pings B : there is no answer - correct
2. second, B pings A : it works - correct
3. but now, if A pings B, A gets replies - NOT CORRECT

Actually what happens is obviously the following :
- step 1 : there is no rule to accept traffic from A to B so there is no reply
- step 2 : there is a rule to accept traffic from B to A, so as default OPNSense tracks the state of the connexion and replies from A are accepted back to B
- step 3 : when, at that point, A initiates traffic to B, OPNSense uses the previous state of the connexion at step 2 and it accepts the traffic !

Does it ? Can you share a packet trace of the scenario you tested ? (Interfaces: Diagnostics: Packet Capture).

Your example is about ping, which is ICMP, can you reproduce this with a TCP scenario also ?

So, in case there is regular communication from B to A, an attacker could suddenly usurpate the IP address of A to attack B through the firewall.

How can one definitely block traffic from A to B that is initiated by A ?

That is done by a default block rule in OPNsense, please share the packet trace to diag whats happening in your scenario.

It just feels "wrong" that my UDMP works with this module but OPNsense doesn't.

The first important thing, OPNsense doesn't (or does) support your SFP module, your NIC does. OPNsense (or FreeBSD) perfectly supports the Intel X553 network adapter, so if your module doesn't work, look at the adapter, not OPNsense.

Now your using the X5xx series from Intel, which is known to only support genuine Intel SFP's (hence the "hw.ix.unsupported_sfp=1" option), which is only found in the in-tree FreeBSD and Linux drivers. Third party SFP's shouldn't be a problem with this setting, but there are some specials like 10Gb RJ45 adapters (out-of-spec) and BiDi variants for AON Fiber like yours.

The other challenge is the Qotom device, got my hands on two of these (1U version) units a few weeks ago and both have problems with one of the SFP cages. The mini-pc and 1U versions have swapped fronts (so up&down is reverse for the two different models, see pictures on Qotom website for reference) and the SFP cages are _not_ numbered on the chassis (the copper ones are, but don't match OPNsense/FreeBSD assigment). If looking at the front of the 1U chassis the assignment in OPNsense 24.7 is this:

Code Select Expand


igc0 igc2 ix3 ix1

igc4 igc1 igc3 ix2 ix0


The ix3 port looks problematic for _any_ SFP with both the units I touched (supported or not), the interface is detected but the port won't power the module and so completly useless. I did read in several forums on the Interwebs (also this one), more people experiencing "one SFP isn't working" but lacks any details or orientation.

So back to your original question:

Is there any additional tests I can do?  Any additional settings I can try?  Maybe I am not waiting long enough for a connection? 

There're are a few things to check here: the SFP module, Qotom device (ix3 cage) and the Intel X553 NIC.

You're saying the module _does_ work in another device, so assume that's not your problem.

For the Qotom device, try and swap your SFP module in each and every SFP cage. Connect to OPNsense over SSH, select the "Shell" option and from the console type for every cage (interface) with the module inserted:

Code Select Expand

ifconfig -v "interface name"

With the -v option ifconfig on OPNsense shows you the module information (if correctly detected), something like this:

Code Select Expand


ix0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9216
options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,HWSTATS,MEXTPG>
ether a1:b2:c3:d4:e5:f6
media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
drivername: ix0
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: Intel Corp PN: ABC123 SN: XYZ789 DATE: yyyy-mm-dd
module temperature: 0 C voltage: 0 Volts
lane 1: RX power: 0.00 mW (-0.00 dBm) TX bias: 0.00 mA


If you see the module correctly in the ifconfig output but it still doesn't work or you don't see the module info at all (which explains why it doesn't work right away) use your Joker -> the Intel X722 NIC. I'm more familiar with the Intel E810 series, but there's a bigger change it's less picky and/or will support more (third party) modules than your X533.

At this stage these are the most important task you can try to get to the root of your problem.

I was able to upgrade to version 24.7 now, and still unable to ping or traceroute.

ICMP wasn't available for years in Azure (still not with things like Load Balancers), nowadays you need to specificaly allow ICMP. My first guess would be your NSG, is there an explicit ALLOW for ICMP ?

what other settings might I need to allow this?

Your cloud network infrastructure and/or NSG.

I finally have everything working as it should but one thing i cant find on the forum and in the docs is how to tel one device in my network to use always use one gateway or interface, so not using the gateway group.

In the specific Firewall Rule(s) select the WAN2 gateway (Gateway option) to use Policy Based Routing, the rule(s) will now use the WAN2 exclusively.

Yeah, it is not helping since that clearly does not work. There are other ACME mechanisms that work. HTTP-01 is not one of them.

DNS-01 is prefered indeed for a lot of reasons, HTTP-01 at scale is a disaster, but it does work perfectly fine over https tcp/443. In 99.9% of failures the redirection configuration is not done right (mostly NGINX), but that's Layer 8, not ACME or LetsEncrypt.

Not helping the OP issue, which is another story, but:

For HTTP-01 to work, you MUST NOT be redirecting the well-known URL to HTTPS.

Instead of a "MUST NOT" LE itself talks about "SHOULD"

Code Select Expand

Our recommendation is that all servers meant for general web use should offer both HTTP on port 80 and HTTPS on port 443. They should also send redirects for all port 80 requests, and possibly an HSTS header (on port 443 requests).

* https://letsencrypt.org/docs/allow-port-80/

Exactly. It will only be queried via HTTP, not HTTPS (obviously, otherwise the first verification would never work, since you do not have a certificate then).

The nice thing with LE is that they don't care what you provide as certificate (invalid, self-signed, etc) when doing the challenge. So even if you refreshing your cert 1 day too late, your expired certificate will be used for the refresh (ie, not checks or validation on the cert, only on a valid ACME challenge.

Code Select Expand


Our implementation of the HTTP-01 challenge follows redirects, up to 10 redirects deep. It only accepts redirects to "http:" or "https:", and only to ports 80 or 443. It does not accept redirects to IP addresses. When redirected to an HTTPS URL, it does not validate certificates (since this challenge is intended to bootstrap valid certificates, it may encounter self-signed or expired certificates along the way).

* https://letsencrypt.org/docs/challenge-types/