Setting up fiber PPPoE connection

[newsense]

Change the LAN to 192.168.11.0/24 with the LAN interface 192.168.11.1/24 and disable that rule

[meyergru]

Yes, change the LAN network range to anything but 192.168.1.0/24, because otherwise it will be impossible to route any packets from LAN to OPT2, however, you still need the NAT rule from LAN to there, because the ONT does not have a route and only answers to packets from its own network.

[newsense]

Yeah I was pondering that too actually, initially I thought that would be Lan to Wan traffic but NAT seems to be required here after all.

[markfree]

I've changed the LAN network to 192.168.7.0/24.
This is the new configuration.

Code Select Expand

LAN (igc1)      -> v4: 192.168.7.6/24
OPT2 (mlxen0)   -> v4: 192.168.1.1/24
PC -> 192.168.7.3 - GW: 192.168.7.6


If I understood correctly, this is how the outbound NAT should be:


Unfortunately, this way the 192.168.1.1 address only takes me back to OPNSense UI.

[meyergru]

Of course it does, since you have set the OPT2 interface address of the OpnSense to 192.168.1.1. So, if you access 192.168.1.1, you reach the OPT2 interface of the OpnSense, never anything else that is connected to it.

If 192.168.1.1 is the designated IP of your ONT, you must set the OPT2 interface to something else in the 192.168.1.0/24 network, say 192.168.1.3. This is basic networking.

Also, it seems like OPT2 state is not "up", so I wonder if there is anything actually connected to that interface, because then there should be no red X.

[markfree]

I see. I've set the OPT2 interface with address 192.168.1.2 now.
Still, I can't access the module. The interface only shows "no carrier" status.

Code Select Expand

mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT2 (opt2)
        inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
        status: no carrier

Also, the driver seems to be ok.

Code Select Expand

root@OPNsense:~ # pciconf -lv mlx4_core0
mlx4_core0@pci0:5:0:0:  class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1003 subvendor=0x15b3 subdevice=0x0113
    vendor     = 'Mellanox Technologies'
    device     = 'MT27500 Family [ConnectX-3]'
    class      = network
    subclass   = ethernet

root@OPNsense:~ # pciconf -a mlx4_core0
mlx4_core0: attached

How can I diagnose that?

[meyergru]

Some of the ONTs go online only when they have a valid GPON connect, otherwise they respond only on the serial connection. It may well be that you cannot connect via the network interface as long as there is no GPON online status.

The webpage for that ONT (https://hack-gpon.org/ont-odi-realtek-dfp-34x-2c2/) seems to indicate this in that is explains how to connect via a serial connection and how to set the various parameters.

Some ONT modules need a GPON connection at all (i.e. not a really connected one, just the GPON optical signal) in order to show an online status to the attached computer.

I bet you did not plug in the optical cable yet, did you?

You may have to do this and then access the ONT in order to set the serial # and/or PLOAM to get it fully working.

[markfree]

I did plug the fiber to the module but it wouldn't connect anyway.

Upon monitoring the dashboard, I noticed that the interface was oscillating. Briefly, it seemed to connect before disconnecting again.
During one of these short-lived connections, the interface status briefly showed as "active". I attempted to ping the interface, but unfortunately, I couldn't establish a route to the host. Simultaneously, the host console displayed a "link up" message, followed by a "link down" notification.

After a few reboots, I finally accessed the module. Once I configured an upstream gateway (192.168.1.1) and disabled the outbound NAT, the module was activated. This allowed me to successfully configure the GPON settings.

Now, with the fiber connected, VLAN set, GPON SN and PASS configured, I adjusted the interfaces to better match my environment.


I also configured the PPPoE interface as WAN, but it stubbornly refused to connect. The PPP logs only showed repeated reconnection attempts.

Code Select Expand

<30>1 2023-08-23T17:36:04-03:00 OPNsense.domain ppp 3289 - [meta sequenceId="1154"] caught fatal signal TERM
<30>1 2023-08-23T17:36:04-03:00 OPNsense.domain ppp 3289 - [meta sequenceId="1155"] [wan] IFACE: Close event
<30>1 2023-08-23T17:36:04-03:00 OPNsense.domain ppp 3289 - [meta sequenceId="1156"] [wan] IPCP: Close event
<30>1 2023-08-23T17:36:04-03:00 OPNsense.domain ppp 3289 - [meta sequenceId="1157"] [wan] IPV6CP: Close event
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 3289 - [meta sequenceId="1158"] [wan] Bundle: Shutdown
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 3289 - [meta sequenceId="1159"] [wan_link0] Link: Shutdown
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 3289 - [meta sequenceId="1160"] process 3289 terminated
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1161"] Multi-link PPP daemon for FreeBSD
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1162"]
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1163"] process 53835 started, version 5.9
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1164"] web: web is not running
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1165"] [wan] Bundle: Interface ng0 created
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1166"] [wan_link0] Link: OPEN event
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1167"] [wan_link0] LCP: Open event
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1168"] [wan_link0] LCP: state change Initial --> Starting
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1169"] [wan_link0] LCP: LayerStart
<30>1 2023-08-23T17:36:06-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1170"] [wan_link0] PPPoE: Connecting to ''
<30>1 2023-08-23T17:36:15-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1171"] [wan_link0] PPPoE connection timeout after 9 seconds
<30>1 2023-08-23T17:36:15-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1172"] [wan_link0] Link: DOWN event
<30>1 2023-08-23T17:36:15-03:00 OPNsense.domain ppp 53835 - [meta sequenceId="1173"] [wan_link0] LCP: Down event


The module seems to be receiving the correct signal,


yet I'm unsure of the best course of action from here.
Any thoughts?

[meyergru]

You are getting there...

1. The ONT's GPON status O5 is fine, you have a GPON connection.

2.  There seems to be something wrong with your WAN connection. Have you verified that under interface assigments, WAN is pointing to pppoe0 (xxxxx_vlan600), where xxxxx is the same physical interfaces as your FIB0? Are the pppoe credentials still set up correctly?

You still have to make sure that WAN -> pppoe0 -> xxxxx_vlan600, but last time, xxxxx was mlxen0 and now probably is not any more.

[markfree]

I must be messing up the configuration, but the PPPoE credentials are correct.
These are my current assignments.


This is the WAN configuration


I created a VLAN inteface with tag 600.


And this is PPPoE interface.

[meyergru]

Probably not, since you would see authentication errors if the credentials were wrong, but infact you see timeouts. There is a PPPoE log file, BTW. Also, because the same setup worked / works (?) with an external ONT, I doubt that the credentials are the culprit.

Are you sure that your ISP uses VLAN 600 and PPPoE / PPPoEv6? Many have PPPoE, but DHCPv6.
On the other hand, that seems to have worked with the other ONT as well...

[markfree]

Indeed, the provided credentials are accurate and work successfully on my external ONT (TP-Link XZ000-G3).
In configuring this TP-Link ONT, I focused on setting up the VLAN 600, as well as inputting the GPON SN and Pass, and it works seamlessly in bridge mode.
My current network setup further involves an Asus Router, which connects using PPPoE. It does not require any explicit VLAN configuration. I'm uncertain whether the VLAN setup does not exist in the router or if it is managed automatically, though.

Regarding the PPPoE log, I believe you're referring to the "/var/log/ppps/latest.log" file. This log file corresponds to the data presented under "Interfaces > Point-to-Point > Log File."
It contains only records of connection timeouts.

Code Select Expand

# tail -f /var/log/ppps/latest.log
<30>1 2023-08-25T17:15:59-03:00 OPNsense.domain ppp 86824 - [meta sequenceId="221"] process 86824 started, version 5.9
<30>1 2023-08-25T17:15:59-03:00 OPNsense.domain ppp 86824 - [meta sequenceId="222"] web: web is not running
<30>1 2023-08-25T17:15:59-03:00 OPNsense.domain ppp 86824 - [meta sequenceId="223"] [wan] Bundle: Interface ng0 created
<30>1 2023-08-25T17:15:59-03:00 OPNsense.domain ppp 86824 - [meta sequenceId="224"] [wan_link0] Link: OPEN event
<30>1 2023-08-25T17:15:59-03:00 OPNsense.domain ppp 86824 - [meta sequenceId="225"] [wan_link0] LCP: Open event
<30>1 2023-08-25T17:15:59-03:00 OPNsense.domain ppp 86824 - [meta sequenceId="226"] [wan_link0] LCP: state change Initial --> Starting
<30>1 2023-08-25T17:15:59-03:00 OPNsense.domain ppp 86824 - [meta sequenceId="227"] [wan_link0] LCP: LayerStart
<30>1 2023-08-25T17:15:59-03:00 OPNsense.domain ppp 86824 - [meta sequenceId="228"] [wan_link0] PPPoE: Connecting to ''
<30>1 2023-08-25T17:16:08-03:00 OPNsense.domain ppp 86824 - [meta sequenceId="229"] [wan_link0] PPPoE connection timeout after 9 seconds
<30>1 2023-08-25T17:16:08-03:00 OPNsense.domain ppp 86824 - [meta sequenceId="230"] [wan_link0] Link: DOWN event
<30>1 2023-08-25T17:16:08-03:00 OPNsense.domain ppp 86824 - [meta sequenceId="231"] [wan_link0] LCP: Down event
<30>1 2023-08-25T17:16:08-03:00 OPNsense.domain ppp 86824 - [meta sequenceId="232"] [wan_link0] Link: reconnection attempt 1 in 3 seconds

Previously, I configured the WAN interface with PPPoEv6. However, I really don't know if that's what my ISP uses. So, I switched to DHCPv6. Still, there is no successfull authentication with PPPoE.

[meyergru]

Did you try without the VLAN?

[markfree]

While doing a packet capture, I noticed that when the PPPoE interface is configured to VLAN600, it only sends PPPoE broadcasts and there is no response at all.
The packets are sent with vlan tag 600.

Code Select Expand

FIB0 mlxen1 2023-08-26 23:37:40.453668 00:02:c9:9b:77:89 ff:ff:ff:ff:ff:ff ethertype 802.1Q (0x8100), length 40: vlan 600, p 0, ethertype PPPoE D, PPPoE PADI [Host-Uniq 0xC0AB562B01F8FFFF] [Service-Name]
FIB0 mlxen1 2023-08-26 23:37:44.471702 00:02:c9:9b:77:89 ff:ff:ff:ff:ff:ff ethertype 802.1Q (0x8100), length 40: vlan 600, p 0, ethertype PPPoE D, PPPoE PADI [Host-Uniq 0xC0AB562B01F8FFFF] [Service-Name]

When I remove VLAN600 and set PPPoE on the mlxen1 (fiber) interface, it sends PPPoE discoveries with no vlan tag, but there is a remote response from two concentrators with vlan tag 600, and that's it. There is no subsequent PPPoE request from OPNsense.

Code Select Expand

FIB0 mlxen1 2023-08-26 23:41:51.475540 00:02:c9:9b:77:89 ff:ff:ff:ff:ff:ff ethertype PPPoE D (0x8863), length 36: PPPoE PADI [Host-Uniq 0x80D8092000F8FFFF] [Service-Name]
FIB0 mlxen1 2023-08-26 23:41:51.486668 04:b0:e7:c9:45:73 00:02:c9:9b:77:89 ethertype 802.1Q (0x8100), length 60: vlan 600, p 7, ethertype PPPoE D, PPPoE PADO [Host-Uniq 0x80D8092000F8FFFF] [Service-Name] [AC-Name "ME-BSA4A"]
FIB0 mlxen1 2023-08-26 23:41:51.490523 04:b0:e7:c9:44:38 00:02:c9:9b:77:89 ethertype 802.1Q (0x8100), length 60: vlan 600, p 7, ethertype PPPoE D, PPPoE PADO [Host-Uniq 0x80D8092000F8FFFF] [Service-Name] [AC-Name "ME-BSA4B"]

[meyergru]

That seems strange. It is almost as if VLAN tags are added automatically in only one direction (or stripped in one).

Probably, your ISP has some magic VLAN configuration in their provided external ONTs? I know for sure that such constructs exist in Huawei ONTs where you can have different profiles for specific services, like different VLANs.

Over here in germany, ISPs are required to be open w/r to what equipment is used to provide access, namely the network termination must be passive if the customer requires it. However, many providers do not like that specific law and deliberately do their best to maximize hurdles to actually make use of that freedom. Recently, there is a discussion going on to revise the law for reasons of potential disturbance of networks by customers who use non-certified equipment of some sort.

I do not think that there is a general problem with OpnSense or your configuration at this point. You can easily check if there is when you use the ISP-provided ONT with an otherwise unaltered configuration. If that works, either your GPON module has a problem with VLANs or there must be something special in the ISP's ONT.



P.S.: After looking at https://hack-gpon.org/ont-odi-realtek-dfp-34x-2c2/, I see some hints about VLAN problems with your specific GPON module in specific firmware revisions!

See https://github.com/Anime4000/RTL960x/tree/main/Firmware/DFP-34X-2C2, there seem to be some fixes out for those problems. It seems there are firmwares for different modes... I cannot help with this rabbit hole, though, since I have a different ISP and a different GPON module. Maybe it would be advisable to contact the hack gpon folks or to use a different module, preferably one that is known to work with your ISP.