Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - 2HgRyz13

#1
My local gigabit fiber ISP company was great for years but sold its residential business to another ISP about a year ago.

Unbeknownst to me, the new ISP uses CGNAT (Carrier-grade NAT: https://en.wikipedia.org/wiki/Carrier-grade_NAT). Its purpose is to conserve the ISP's IPv4 addresses to keep the ISP's costs down. The ISP should use IPv6 addresses instead.

CGNAT prevented my newly configured WireGuard VPN from working. This was my first time setting up WireGuard. I read that WireGuard is fast but difficult to set up, so I assumed I was botching the settings from the WireGuard Road Warrior Setup guide (https://docs.opnsense.org/manual/how-tos/wireguard-client.html#step-5-assignments-and-routing) and other guides, doc and video online.

My dynamicDNS address wasn't my DEC850's. It was the ISP's CGNAT address. The address difference was subtle because only the last octet was different (and just +-1) from my OpnSense WAN IP. I didn't notice that for a long time, dang it. I also tested using my WAN IP address in WireGuard instead of dynamicDNS but it wasn't routeable either.

I asked my ISP why my whatismyipaddress-dot-com address in my computer browser (and in dynamicDNS) was different from my firewall's WAN IP address.

They told they used CGNAT. I hadn't heard of it so read about it.

What a nightmare of wasted time. On and off for about 6 months, I slugged it out with OpnSense WireGuard, deleting all the configuration elements then adding them back using different advice online. I estimate I wasted 30-40 hours based on three to four sessions lasting 10 hours each.

The ISP's tech support said I could opt out of CGNAT for no extra cost if I submitted a webform explaining why. They couldn't guarantee it'd be approved and didn't give me a time estimate for their decision or implementation.

My ISP recently approved my request and took me off CGNAT, though they messed up their configuration and left me without internet for most of a day.

Today, after deleting all my WireGuard settings, including instance, peers, interface, firewall rules, I added them back cleanly from the WireGuard Road Warrior guide and everything worked the first time!

At least in my struggles I learned much more about WireGuard, OpnSense and CGNAT than I would have if WireGuard worked the first time.

This ISP has way more scheduled and unscheduled outages than the previous one. That's probably in part due to CGNAT, which is more complicated to maintain, I'm told.

I don't trust or like this small ISP. I'm switching back to my previous one as a business customer (they don't accept residential customers anymore). All their IP addresses are fully routable and they rarely had outages (maybe one a year).

DIRT ON CGNAT

From perplexity.ai...

The use of Carrier Grade NAT (CGNAT) by ISPs introduces several significant drawbacks for both providers and end-users:

## Performance Degradation 
- **Increased latency**: CGNAT adds an intermediary layer, causing delays in real-time applications like VoIP, video conferencing, and online gaming[1][4][8]. 
- **Network congestion**: Shared public IPs can lead to slower browsing and streaming during peak hours[2][4]. 

## Connectivity Limitations 
- **Broken peer-to-peer (P2P) functionality**: NAT traversal issues disrupt torrenting, gaming (e.g., Xbox), and direct device communication[1][3][6]. 
- **Remote access challenges**: Port forwarding restrictions prevent users from accessing home servers, security cameras, or NAS devices externally[2][4]. 
- **Hosting restrictions**: Self-hosted services (websites, email servers) become nearly impossible due to inbound traffic blocking[2][4]. 

## Security and Reputation Risks 
- **Shared IP vulnerabilities**: If one user engages in malicious activity, the entire shared IP may face blacklisting, affecting all users[2][4]. 
- **Expanded attack surface**: A DDoS attack on one IP can degrade service for multiple customers[5]. 

## Operational Challenges for ISPs 
- **Higher costs**: Implementation and maintenance fees, combined with IPv4 address scarcity, increase expenses[1][7]. 
- **Complex traffic monitoring**: Law enforcement and abuse tracking become difficult due to shared IPs[4][7]. 

## User Experience Issues 
- **Streaming quality degradation**: Buffering occurs on platforms like Netflix due to latency[2][4]. 
- **Smart home limitations**: Remote management of IoT devices often requires cloud workarounds[2][4]. 

These drawbacks highlight why CGNAT is often viewed as a temporary solution to IPv4 exhaustion, with IPv6 adoption being the long-term alternative[1][6].

Citations:
[1] https://ipv4connect.com/2023/06/pros-and-cons-deploying-carrier-grade-nat/
[2] https://ausgeek.net/viewtopic.php?t=284
[3] https://www.daryllswer.com/shortcomings-of-cgnat-and-potential-work-arounds/
[4] https://www.rapidseedbox.com/blog/cgnat
[5] https://www.corero.com/ddos-disadvantages-of-carrier-grade-nat/
[6] https://www.reddit.com/r/HomeNetworking/comments/s2ulh6/does_anyone_else_hate_it_when_isp_puts_you_behind/
[7] https://forum.universal-devices.com/topic/21946-ftth-fiber-to-the-home-isps-and-cgnat/
[8] https://www.snbforums.com/threads/pros-and-cons-of-static-ip.87208/


#2
I also decided to leave switching to external switches and stopped trying to bridge two DEC 850 LAN interfaces.
#3
Yay! I figured it out.

1. Booted into Single-user mode in the console then reset the root password.
2. Logged into the console as root.
3. Selected Restore from backup (#13)
4. Rebooted.
Yay!

Logged into the web gui as an admin (not root) on igc0 lan0. Yay!

The magic trick for me was accidentally learning that while in console view but not logged in, I could reboot the DEC 850 with a paperclip in the reset hole next to the light, which initially turns off the DEC 850 but another press turns it back on, then watch the boot info on the console. So the reset button is really a shutdown-off/on button. Unplugging the power cord probably would have done the same thing but less gracefully. I didn't realize the console view remains ready on restart.

Then I quickly pressed the spacebar to bring up boot options and poked around before settling on Single-User mode. It looked promising.

I searched online with perplexity dot ai for "what can I do in single-user mode in opnsense" and it answered, "reset the root password," among other things. Then I searched how to reset the root password in single-user mode and found and used these ZFS-based instructions
# /sbin/mount -u /
# /sbin/zfs mount -a
# opnsense-shell password

which worked! yay!

Then I could log into the console as root and use the easy menu that includes factory reset, restore from backup and other options.

I selected restore from backup (#13), which worked. Yay!

If that hadn't worked, I could have booted into the console as root again and selected restored it to factory defaults.

Chernobyl aborted.

Carry on.

Morals of the story for me:
• create another interface (icg3 lan3) just for admin
• no matter how messed up a config gets, it's always possible to log into the console and restore from backup or restore to factory defaults
• security-wise, it's best to keep the firewall locked up without access to the mini-usb console port on the front, since anyone with access could reset root, etc.
#4
Newbie.

Configured the Dec 850 and updated to latest community edition (instead of Business ed it came with for a year).

In the web interface, instead of configuring igc0 lan0 and igc2 lan2 to communicate with each other just with rules, I tried to "bridge" them.

While bouncing between settings for igc0 lan0, igc2 lan2 and the new bridged interface, I messed up and disabled the igc0 lan interface Chernobyl style (I had bad feeling but kept going as I unchecked the box, saved and clicked apply). Late night config is too dangerous for me.

FAILED with CONSOLE

• I used the serial console to login, but "This account is currently not available" appeared. When I typed a wrong password intentionally, "Login incorrect" appeared instead.

• I also tried root with the password I had written down and the default password, but couldn't login ("Login incorrect").

SSH DISABLED
I recently enabled and used it to check the microcode version then disabled it again, so I know it's not an option.

Setting a manual IP for my computer and connecting to igc2, hoping the bridge might work, didn't help.

RESET HOLE
Apparently it doesn't restore to factory defaults (tried 60 seconds, but the DEC 850 just turned off). Maybe there's a button-press pattern that restores it factory defaults.

Somehere here posted in Oct 2024 that they might open their DEC 850 to reset the BIOS, which isn't my goal, but that makes me wonder if I can restore the DEC 850 to factory defaults by opening it and maybe replacing, erasing or rewritting its SSD, if removable.

I saved a backup config before I messed up the settings. But it won't do me any good if I can't get back in.

Fortunately, I have another firewall I can use.

#5
After updating my DEC850 's (AMD EPYC) firmware and installing the new microcode plugin, I used SSH to see the running microcode version:

0x800126f

Unfortunately, I forgot to check the microcode version before I installed the plugin.

Is 0x800126f the latest?

When was it released?

If released recently, did it fixed the SinkClose vulnerability?

I didn't find the answers in numerous online searches.

I'm still waiting for the sinkclose fix before I deploy my new DEC850.


UPDATE: AMD's bulletin lists microcode version 0x0800126F 2024-05-03 as "Mitigation Option 2" for fixing CVE-2023-31315 (sinkclose) for 1st Gen AMD EPYC processors (including 3201 in DEC 850, I assume) listed in the Data Center section. So it looks like  0x800126f fixed the flaw a long time ago. Also, 0x800126f is listed here (https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/amd-ucode/README) under microcode_amd_fam17h.bin in the line Family=0x17 Model=0x01 Stepping=0x02: Patch=0x0800126f and appears to be the latest version. I found 0x0800126e (the previous version) mentioned here (https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/amd-ucode/README) posted on 9-19-2022.
#6
FYI — version 24.7.2 (August 21, 2024), opnsense release doc:

"As a special note we now have native CPU microcode update plugins for either AMD or Intel to install from the GUI. Apart from a reboot these plugins require no further user interaction and will keep the applicable microcode at the latest known version as shipped in the packages repository."
#7
I, too, would like to know the estimated fix date for the SinkClose vulnerability in the EPYC 3201 in my OpnSense DEC850.

My DEC850 is new. I configured it but haven't deployed it yet. I'll wait until SinkClose is patched.

AMD's advisory seems to list the Embedded EPYC 3000 [series] "target" as Oct 2024, if I read it correctly:
https://www.amd.com/en/resources/product-security/bulletin/amd-sb-7014.html

Then I assume OpnSense will need X weeks to incorporate the fix.

So maybe Oct at the earliest but Nov or Dec more likely.

I can wait but was looking forward to installing the DEC850 this week. :(