Messages

thanks again netnut (nice nick btw :) ) I can read/understand dutch, had a look, will check when I will start using the ONT/PON, once i got a local lan working

thanks, indeed with odido that should work as you say, though I wonder if I could also replace the PON and directly plug the fiber cable into the baremetal.
As for the spf+ "no carrier" problem, it seems to be a driver issue, tried freebsd 14 live and still not working but in ubuntu it does.
Also posted here https://forums.freebsd.org/threads/cant-get-sfp-nic-to-work-amd-driver-issue.93186/ let's see if it can be fixed somehow.

[4x2.5 gbps RJ45 sockets (igc0 to igc3)]

hi all :)

I'm having some trouble to get my Internet working with Odido in NL, so I thought to start this threat which hopefully when solved can be a guide for people to use their own modem/router as well for Odido in NL.
Some links from that might be helpful for own modem/router setup:

https://community.odido.nl/thuisnetwerk-539/how-to-edgerouter-iptv-internet-342141
https://community.odido.nl/bekabeld-internet-492/redelijk-eenvoudig-aansluiten-eigen-router-edgerouter-4-inclusief-tv-330666
https://community.odido.nl/thuisnetwerk-566/my-own-modem-357022
Official settings to be used provided by Odido:
https://assets.odido.nl/x/6ba57ca29d/eigen-modem-modem-instellingen.pdf

I just upgraded to a 8 gbps fiber WAN connection, earlier 1 gbps. (fyi there is no ipv6 support, so all only on ipv4)

fiberoptic cable comes into the apartment and goes into PON

Odido provides for speeds above 1 gbps the following PON:

OptiXstar HN8010Ts-20 XGS-PON Terminal (it is setup to run on ip 192.168.100.1)

It has a 10gbps RJ45 port that would go into the WAN port of the Odido provided Zyxel T-75 (which is a zyxel-ex7501-b0).

Earlier I had already replaced the provide modem when I was on 1 gbps. The RJ45 cable from the PON was plugged into a T2500G-10TS (tp-link 1G 8 port switch) on which I had created a VLAN on TAG ID 300.

Then another RJ45 cable from the T2500G-10TS went into my Ubiquiti edgerouter lite into eth0 which was configured to receive IP via dhcp (no other config on vlan or else on that nic).

eth1 was configured to run dhcp server and went into a switch for my local network etc.

That all works.

Now I got a baremetal machine:

4x2.5 gbps RJ45 sockets (igc0 to igc3)
igc0: <Intel(R) Ethernet Controller I226-IT> mem 0xfc900000-0xfc9fffff,0xfca00000-0xfca03fff at device 0.0 on pci4
igc0: Using 1024 TX descriptors and 1024 RX descriptors
igc0: Using 4 RX queues 4 TX queues
igc0: Using MSI-X interrupts with 5 vectors
igc0: Ethernet address: d0:63:b4:05:47:3f
igc0: netmap queues/slots: TX 4/1024, RX 4/1024

2xSFP+ sockets (ax0 and ax1)
ax0: <AMD 10 Gigabit Ethernet Driver> mem 0xfcb60000-0xfcb7ffff,0xfcb40000-0xfcb5ffff,0xfcb82000-0xfcb83fff at device 0.2 on pci10
ax0: Using 512 TX descriptors and 512 RX descriptors
ax0: Using 3 RX queues 3 TX queues
ax0: Using MSI-X interrupts with 7 vectors
ax0: Ethernet address: d0:63:b4:05:47:43
ax0: xgbe_config_sph_mode: SPH disabled in channel 0
ax0: xgbe_config_sph_mode: SPH disabled in channel 1
ax0: xgbe_config_sph_mode: SPH disabled in channel 2
ax0: RSS Enabled
ax0: Receive checksum offload Enabled
ax0: VLAN filtering Enabled
ax0: VLAN Stripping Enabled
ax0: Checking GPIO expander validity
ax0: GPIO configuration valid
ax0: SFP detected:
ax0:   vendor:    OEM
ax0:   part number:    SFP-H10GB-CU5M
ax0:   revision level: 09
ax0:   serial number:  CSC231202820014
ax0: netmap queues/slots: TX 3/512, RX 3/512

ax1: xgbe_phy_sfp_signals: port_sfp_inputs: 0x7
ax1: xgbe_phy_sfp_detect: mod absent
ax1: xgbe_phy_sfp_signals: port_sfp_inputs: 0x2
ax1: SFP detected:
ax1:   vendor:    Walsun
ax1:   part number:    HXSX-ATRI-1
ax1:   revision level: 1.0
ax1:   serial number:  H230S014913
ax1: xgbe_phy_sfp_signals: port_sfp_inputs: 0x0

1xwifi (iwlwifi0_wlan1)
2x5G SIM slots.

I installed OPNsense on it, no problems and it is running:

OPNsense 24.1-amd64
FreeBSD 13.2-RELEASE-p9
OpenSSL 3.0.12
FreeBSD OPNsense.localdomain 13.2-RELEASE-p9 FreeBSD 13.2-RELEASE-p9 stable/24.1-n254969-8659880248c SMP amd6

I want one of the sfp+ sockets to be the "WAN" socket, so I connected the RJ45 cable from the PON to this sfp+ port. For now I will configure one of the RJ45 (igc0) to have a static local lan IP (10.0.0.120) so I can reach opnsense machine to configure it, the other RJ45 (igc1) I will configure for now to dhcp. I set it in the following way:

In Interfaces-->Assignments-->I added 2 RJ45 lan interfaces (igc0 and igc1)

Then I enabled the igc0 interface and set a static ip from my local network (10.0.0.120) to it, that is also the IP through which I then start accessing the OPNsense box via my laptop in the local network to do the settings (the other end of the cable goes into a temporary wifi access point).

Odido splits up internet, voip, and iptv into three vlan's. I only use their internet which is on VLAN 300, so did the following:

Interfaces-->Other Types-->VLAN hit the plus to add a VLAN selected the SFP+ (ax1) as parent, set VLAN tag to 300 and saved it, then:

In Interfaces-->Assignments i assign device VLAN to sfp+ (ax1) interface and set it to dhcp.

This is not working, there is no IP being assigned.

This should work but does not, it shows as if no cable is connected and says "no carrier". Connecting via ssh to the opnsense box and checking with ifconfig the interfaces, the sfp+ (ax1) interface shows

media: Ethernet autoselect
status: no carrier

For testing purposes I configured on the baremetal opnsense machine as above but using one of the RJ45 as WAN port and not the sfp+ (ax1). That did work, the interface gets an IP assigned from the provider. So I assume there must be an issue with the sfp+

Decided to test the following, create a local network from the baremetal machine sfp+ with local server on 10.0.0.x network who also has sfp+ nic interface, which is already configured on static ip 10.0.0.15.

Configured on opensense new (removed vlan setting etc. from before) sfp+ (ax1) to static ip 10.0.0.5 connected the sfp+ on the opnsense baremetal and the local server sfp+ but opnsense shows no carrier and the machines cannot ping each other.
The LEDs on the physical NICs on both ends shows cables are plugged in.

fyi I do know the sfp+ interfaces, cables etc. are working, before I installed pfsense i had ubuntu on the baremetal running and had established successful connections with the sfp+ nics and the local server.

Tried the following on sfp+ on baremetal:

^ax0 is connected with a DAC cable 10Gtek 7meters CAP-10GSFP-P7M 10GBase-CU to a arch linux machine who has an Intel X710 NIC with two sfp+ ports.

Connection info (ifconfig/ethtool) on arch linux shows this:

enp7s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.133.120.3  netmask 255.255.255.0  broadcast 10.133.120.255
        inet6 fe80::3efd:feff:fe9e:cb7c  prefixlen 64  scopeid 0x20<link>
        ether 3c:fd:fe:9e:cb:7c  txqueuelen 1000  (Ethernet)
        RX packets 3380  bytes 612591 (598.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 647268  bytes 94844545 (90.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp7s0f1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet6 fe80::3efd:feff:fe9e:cb7e  prefixlen 64  scopeid 0x20<link>
        ether 3c:fd:fe:9e:cb:7e  txqueuelen 1000  (Ethernet)
        RX packets 67949  bytes 26850373 (25.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 76359  bytes 10329417 (9.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ethtool enp7s0f0
Settings for enp7s0f0:
   Supported ports: [ FIBRE ]
   Supported link modes:   10000baseT/Full
   Supported pause frame use: Symmetric Receive-only
   Supports auto-negotiation: No
   Supported FEC modes: Not reported
   Advertised link modes:  10000baseT/Full
   Advertised pause frame use: No
   Advertised auto-negotiation: No
   Advertised FEC modes: Not reported
   Speed: 10000Mb/s
   Duplex: Full
   Auto-negotiation: off
   Port: Direct Attach Copper
   PHYAD: 0
   Transceiver: internal
   Supports Wake-on: g
   Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
   Link detected: yes

ethtool enp7s0f1
Settings for enp7s0f1:
   Supported ports: [ FIBRE ]
   Supported link modes:   10000baseT/Full
   Supported pause frame use: Symmetric Receive-only
   Supports auto-negotiation: No
   Supported FEC modes: Not reported
   Advertised link modes:  10000baseT/Full
   Advertised pause frame use: No
   Advertised auto-negotiation: No
   Advertised FEC modes: Not reported
   Speed: Unknown!
   Duplex: Unknown! (255)
   Auto-negotiation: off
   Port: Direct Attach Copper
   PHYAD: 0
   Transceiver: internal
   Supports Wake-on: g
   Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
   Link detected: no

On the baremetal OPNsense machine I set up a static ip on sftp+ nic ax0.
Also on baremetal configured WAN device to get IP via dhcp4 and be ax1 (which is using the sfp+ Walsun with RJ45 cable cat6e, 1meter):

From dmesg on baremetal/opnsense:

ax0: xgbe_phy_sfp_signals: port_sfp_inputs: 0x0
ax0: SFP detected:
ax0:   vendor:    OEM
ax0:   part number:    SFP-H10GB-CU5M
ax0:   revision level: 09
ax0:   serial number:  CSC231202820016
ax1: xgbe_phy_sfp_signals: port_sfp_inputs: 0x7
ax1: xgbe_phy_sfp_detect: mod absent
ax1: xgbe_phy_sfp_signals: port_sfp_inputs: 0x2
ax1: SFP detected:
ax1:   vendor:    Walsun
ax1:   part number:    HXSX-ATRI-1
ax1:   revision level: 1.0
ax1:   serial number:  H230S014913
ax1: xgbe_phy_sfp_signals: port_sfp_inputs: 0x0

ifconfig on baremetal/opnsense:

ax0: flags=8867<UP,BROADCAST,DEBUG,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   description: OPT1 (opt1)
   options=4e0032b<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
   ether d0:63:b4:05:47:43
   inet 10.133.120.125 netmask 0xffffff00 broadcast 10.133.120.255
   media: Ethernet autoselect
   status: no carrier
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ax1: flags=8867<UP,BROADCAST,DEBUG,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   description: LANSFP (opt1)
   options=4e0032b<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
   ether d0:63:b4:05:47:44
   media: Ethernet autoselect
   status: no carrier
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

not sure what to try next, any suggestions please?