Messages

I just pick up the following section and explain what he have done

Code Select Expand

    # ACL: Websocket_Upgrade
    acl acl_61bfe65d4066b0.98106842 hdr(Connection) -i Upgrade
    # ACL: Websocket
    acl acl_61bfe67f211b51.96043088 hdr(Upgrade) -i WebSocket
    # ACL: Websocket_WS
    acl acl_61c0e62bc10459.22208791 hdr_beg(host) -i ws
    # ACL: Emby_Webserver
    acl acl_5e811fc593d624.87923872 hdr_sub(host) -i example.com

    # ACTION: Emby_Websocket
    use_backend Emby_Webserver if acl_61bfe65d4066b0.98106842 acl_61bfe67f211b51.96043088 acl_61c0e62bc10459.22208791
    # ACTION: Emby_WebServer
    use_backend Emby_Webserver if acl_5e811fc593d624.87923872

he already have a websock backend called "Emby_Webserver"

He created 4 condition
1. Name=Websocket_Upgrade, Condition type=Custom condition (Option pass-through), Option pass-through=hdr(Connection) -i Upgrade
2. Name=Websocket, Condition type=Custom condition (Option pass-through), Option pass-through=hdr(Upgrade) -i WebSocket
3. Name=Websocket_WS, Condition type=Custom condition (Option pass-through), Option pass-through=hdr_beg(host) -i ws
4. Name=Emby_Webserver, Condition type=Custom condition (Option pass-through), Option pass-through=hdr_sub(host) -i example.com

He created 2 rules
1. Name=Emby_Websocket, Test type=IF, Select conditions=Websocket_Upgrade+Websocket+Websocket_WS, Logical operator for conditions=AND, Execute function=Use Specified Backend Pool, Use backend pool=Emby_Webserver
2. Name=Emby_WebServer, Test type=IF, Select conditions=Emby_Webserver, Execute function=Use Specified Backend Pool, Use backend pool=Emby_Webserver

He put these 2 rule into Frontend: HTTPS (Front End HTTPS), with the following order
Emby_Websocket in front, then Emby_WebServer

A frontend is a listener, listening a specific post of a specific IP.
However, one port of a IP can bind with one frontend at a time.
When you have 2 frontend listening to the the same port of same IP,  only one of them will work.
That's why you find only one of them work.

The process to make a simple HAProxy work:
(Assume all real server are http services, and you don't need to redirect port 80 to 443)
I won't write too detail, you may need to fill in other stuffs like cert etc.
1. Create real servers
2. Create backends, each real server should have at least one backend
3. Create conditions to check sni (example: SNI TLS extension matches (TCP request...), aaa.website.com), until you make all condition
4. Create rules to redirect to a backend if specific condition is true, one rule for each condition
Example:
Name=aaa_ru, if, aaa_cond, use specific backend, aaa_back
5. Create 1 frontend listening to "WAN IP"*#:443, put all rules you created in 4 to rules part.

#If you have dynamic IP, you might need to create a Loopback VIP, frontend bind to that VIP, firewall port forward set to that VIP.

If you need more advanced setting, you may check
https://forum.opnsense.org/index.php?topic=23339.0

@theHellSite
Hello
When I overide the dns Server will the haproxy still be used or not?

It depends on how you override the dns record.
Assume that you set your SNI frontend with 0.0.0.0:80 and 0.0.0.0:443

For example, you added a DNS record in Cloudflare "abc.domain.com" pointing to your WAN IP, and your tested it and found HAProxy working both locally and externally.
Then you removed the DNS record from Cloudflare, and add one in unbounded "abc.domain.com" pointing to your OpnSense IP (either LAN or WAN, doesn't metter)

Then your HAProxy should work locally, but failed externally.
*If you want to override DNS record in unbound, always point to SNI frontend. If you set it to either HTTP frontend or HTTPS frontend, it will fail.

Strange, there shouldn't be something related to cloudflare
Maybe try to create condition: "SNI TLS extension matches (locally deciphered)" with your full sni, prism.website.com
Create another rule that
Use specified Backend Pool: Prism_backend
When
Source IP matches a specific IP...
And
SNI TLS extension matches (locally deciphered)

Use this rule to replace the one with local map file first

mineos.website.com > works locally and externally
prism.website.com > 503 error locally and externally

Sorry, I haven't read the error 503.

Try not to use "src_is_local"
I remember that there is issue with this function, especially under proxy protocol. (It's hyproxy issue since 1.7?)
Try to use "Source IP matches a specific IP" instead

The issue is really similar to this one
I guess the issue should be solved in 22.1.4.
I run the patch in 22.1.3, so I don't know rather it is really fixed. But I don't have such problem anymore (currently 22.1.5)

If your are in 22.1.4 or 22.1.5, having similar issue. But general log don't have something like

Code Select Expand

/firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '192.168.64.1' -alias' failed to execute
Please create another thread under 22.1 Production Series

If you have exactly the same log and you are in 22.1.4 or 22.1.5. Please try to reply to the thread

0_SNI_frontend > Listen Addresses:0.0.0.0:80, 0.0.0.0:443
should this need to be the Virtual IP as opnsense runs on 192.168.1.1

^^fyi thankyou for the tips on tracing

For this question, lets clear the package path that OP wants to do first.
Assume the following IP config:
WAN IP: 1.2.3.4
Firewall IP: 192.168.1.1
VIP: 192.168.64.1
Server IP:port and SNI: 192.168.1.2:80, the.website.com

A browser try to access https://the.website.com from internet.
It asked system to resolve from DNS server: the.website.com. DNS server replies it is 1.2.3.4
Browser try to access 1.2.3.4 with port 443, sending TLS package with SNI=the.website.com

Since haproxy SNI_frontend is listening to 0.0.0.0:443, that means it is listening to port 443 that all IP can represent the firewall. In this case, it is 1.2.3.4:443 and 192.168.1.1:443

SNI_frontend catches the TLS package in 1.2.3.4:443 and passes to SSL_backend(VIP) without changing port

Since SSL_frontend is listening to 192.168.64.1:443, it takes the TLS package and knows that it try to access 192.168.1.2:80. SSL_frontend communicate to the browser, exchanging the SSL cert and keys according to ciphers.

Session to webserver_backend start, SSL_frontend redirect remaining packages to webserver_backend.

I don't know why your VIP won't work, maybe missing opnsense update, misconfig of VIP or another new bug. (You can try to ping the VIP in LAN to check rather it reply. If the VIP is working normally, it should reply)
And SNI frontend wont use VIP unless you use hopey's method (He still need to add NAT rule to redirect packages to VIP as I mentioned in another thread)

I usually trace the session in Services: HAProxy: Statistics, Counters
When saving HAProxy setting, it will reset the session stat.
So, we can check the session stop at which server if we try to access immediately after a reset.
For example, I expect a session goes through
1. Frontend: TCP_front
2. Frontend: SSL_front
3. Backend: opn_back (usually don't have problem if you type correctly)
4. Server: opn

If all of them have session counts, but you still failed to access the website.
That means, it should be problem between browser and server, but you can access to the page when you type the server IP and port directly. Then, there might be issue about your ciphers, certs, OCSP settings, etc. You might find handshake error in server log too.

If it stop at 2. That means, your haproxy is not recognizing SNI correctly. Check your map file, or you can try to create condition with "SNI TLS extension matches (locally deciphered)" and your full SNI (the.domain.com), then create a rule to "Use specified Backend Pool" when condition matches.
Removing all rules and set default backend to test server first can also be a choice. (At least you will know rather it fails only in SNI part or more parts suffers)

If it stop at 1. There might be issue with the VIP again. (The bug similar to this)

If no session count. The listener is not working. TCP_frontend have wrong Listen Addresses. If you try it from WAN, check your firewall rules too, or maybe DNS record issue.

Nice work.
But, do you only need to access your services in LAN?

The key word: bind
means you assigns a listener to a specific port of an IP
*You cannot bind 2 services in same port of same IP

Obviously you can create a VIP and bind to it, then create a local DNS record that points to the VIP.
Let say your Firewall LAN IP is 10.0.0.1, WAN IP is 1.2.3.4, website you host is 10.0.0.2:443, VIP you created is 10.1.0.1, SNI you want to redirect to the website is the.website.com

In LAN environment, you tried to access https://the.website.com
You browser asked the system about the IP of the.website.com. PC asked DNS server about the IP of the.website.com
Unbounded DNS server replies that it is 10.1.0.1.
Your browser tried to access 10.1.0.1:443 and sent a TLS package with SNI=the.website.com
Haproxy frontend handled TLS etc. and forward packages to 10.0.0.2:443.
So, In LAN environment, it should work.

In WAN environment.
Assume that you use DDNS service, and set the.website.com points to 1.2.3.4 in public. You opened port 443 to public.
As opnsense is listening to 0.0.0.0:443, which means it is listening to port 443 of ANY IP which can represent your firewall.
(10.0.0.1:443 and 1.2.3.4:443 in this case)
Your mobile device using 2.3.4.5 asked GoogleDNS to resolve the.website.com, and it resolves with 1.2.3.4.
You tried to assess 1.2.3.4:443 and send a TLS package with SNI: the.website.com.
It will give you opnsense page without asking haproxy, which is dangerous.

Recommend you add a NAT rule if you need WAN access
Interface: WAN
Source: Any
Destination: WAN address
Port 443
Redirect target IP: Your VIP

I mean

# Backend: SSL_backend ()
backend SSL_backend
website.com    mode tcp

And

# Backend: club_backend ()
backend club_backend
website.com    mode http

Usually the wouldn't have SNI or domain names hard code in backends

You are missing "code 301" in HTTPupgrade_rule
(Part 5-9)

Explanation of the code can be found here

This technique will only work when using mode http because it redirects at the HTTP layer using a 302 Found HTTP response status, which is known as a temporary redirect. Once you're fully committed to using HTTPS and have tested it thoroughly on your website, you may wish to instruct the browser to cache the redirect, which will save one round trip between the browser and HAProxy, speeding up page load times. Set the code parameter to 301 to send a 301 Moved Permanently status back, which browsers can cache:

Although it should work with 302, just give it a try

BTW, I don't know why there is website.com in the code of both backend. There shouldn't be something like this.

Can you find your new cert in System: Trust: Certificates?

Strange,  have just tested one of my backend
Create user->Enable Basic Auth and select Allowed users in backend
It works as expected.

Edit: Just tested a bit deeper, for some pages like unifi controller, it will always redirect to wrong page
For some pages like opnsense web UI, in chrome(PC), it will keep prompting for auth, but in firefox(PC & mobile), everything works normally

BTW, nothing more can be done in haproxy too, as some site in some browser works normally. Thus, the problem is due to webserver and browser.

I don't think you need to create another ipv6 real server, as long as it is the same sever in ipv4
You only need to add :::443 and :::80 to frontend listener (in frontend, [::]:80 is the same as :::80, in case you confused with the syntax)
That will be ipv6 to 4 setup.

If you add 2 real server to the same backend, you are load balancing them.

Check and change Services: HAProxy: Settings: Default Parameters
Check Tuning Options of your frontends rather you overwrite the setting too.
If it doesn't fix the issue, then it is Nginx setting problem
Check this link too for Nginx setting