[solved] nginx auth issues with Exchange 2016/IIS 401 loop

[klamath]

Hello,

I have been investing a good deal of time getting IDS working on opnsense for SSL inspection.  At first I went the route of using HAproxy with decrypt/encrypt however I was told that the IDS system for opnsense requires an interface to monitor and act upon.  I was recommended to use nginx since it offers a WAF that should fit my needs.  I got all the endpoints going, everything seems fine, however when running Microsoft's connectivity checker it fails validating the connection back to my firewall.  I am not seeing this issue with HAproxy that is currently serving up access to Exchange.  I ran into a few fixes online to address the issue however I am at a loss as to where I can plumb the fixes into opnsense's GUI for nginx. 

The problems line up exactly with these posts [1,2,3], authentication loop when trying to reach "autodiscover.domain.com/Autodiscover/Autodiscover.xml"  I can verify an auth loop by trying to login via a web browser to this URL with nginx fronting the requests.

Thank you,
Tim

[1] https://forum.opnsense.org/index.php?topic=12939.msg59935#msg59935
[2] https://stackoverflow.com/questions/14839712/nginx-reverse-proxy-passthrough-basic-authenication
[3] https://community.synology.com/enu/forum/1/post/132310

[Fright]

Hi
I made such a setup, although there was additionally used wap/adfs between nginx and Exch.
I will try to test it when there is time. And what username format you use for authentication (just username or domain\username)?

[klamath]

Thank you for the response, the auth format that was use is ad\username.  The MS exchange test that i used is located here: https://testconnectivity.microsoft.com/tests/Ola/input

Tim

[Fright]

Hi.
it seems to me that there are some erroneous statements on the links, which then began to be copied on the forums.
can you check the connection again and look to the HTTP headers given in the analyzer report about the POST request error?
is there an NTLM? like:

Code Select Expand


Server: nginx
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
WWW-Authenticate: Basic realm="****"

afaik nginx still wants commercial subscription for ntlm support on upstreams.
so imho you need to leave only the basic auth on autodiscovery app on Exch.
it works for me in test environment (with Exch2k7 but i don't think there is a difference): all works via nginx without 3d-party modules.

and if you look in nginx docs for Exch balancing they also leave the basic auth only:

Code Select Expand


C:\> Set-AutodiscoverVirtualDirectory
-LiveIdNegotiateAuthentication 0
-WSSecurityAuthentication 0 -LiveIdBasicAuthentication 0
-BasicAuthentication 1 -DigestAuthentication 0
-WindowsAuthentication 0 -OAuthAuthentication 0
-AdfsAuthentication 0

https://docs.nginx.com/nginx/deployment-guides/load-balance-third-party/microsoft-exchange/

[klamath]

Thank you for all the investigative work, I really appreciate it!

I think you might be on to something:

An HTTP 401 Unauthorized response was received from the remote Unknown server. This is usually the result of an incorrect username or password. If you are attempting to log onto an Office 365 service, ensure you are using your full User Principal Name (UPN).
HTTP Response Headers:
Connection: keep-alive
request-id: 8283fc28-3fc3-4e8c-a634-af087592ed43
Content-Length: 0
Date: Fri, 29 Jan 2021 14:29:00 GMT
Server: nginx
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
WWW-Authenticate: Basic realm="autodiscover.xxx.com"
X-FEServer: EXCHANGE2016
Referrer-Policy: same-origin

Would you recommend me changing auth options to basic?  Is there a way in opnsense to get the paid for version of nginx?


Many Thanks,

Tim

[fabian]

AFAIK you can only buy that version from the vendor and maybe you can install that manually. This version cannot be used with the OPNsense Plugin UI as you will miss some required compiled in extensions (for example the WAF)

[Fright]

Would you recommend me changing auth options to basic?

at least i would try.
the stackoverflow solution actually enforces the use of the basic auth by stripping the "WWW-Authenticate: Negotiate" and "WWW-Authenticate: NTLM" headers from the backend response on 401 code
(this may be necessary if you do not have access to the Exch config. but if there is access, then it doesn't make much sense. especially since it requires the use of 3d-party module).

Is there a way in opnsense to get the paid for version of nginx?

I don't think there is a way to convert the plugin to this. everything will be on you
and I wouldn't do it because of autodiscovery. works fine with basic auth over TLS

[klamath]

So today was eventful, I remember how delicate Exchange is.   I ended up implementing the majority of the nginx recommendations you linked, but I was chasing an Address Book endpoint issue after switching over, I ended up turning too many knobs and I was reminded Exchange is built on bubblegum and hope :)

I ended reinstalling all the virtual IIS directories, got a clean bill of health via Exchange connectivity test running under haproxy and then switched over to nginx. 

I am now stuck at this error message with Exchange connectivity test [1]  If you can see anything wrong please let me know, I configured Basic and NTLM auth via IIS for "Default Web Site/mapi" and still receive the error message.

Thanks,
Tim

[1]

Code Select Expand


Testing the MAPI Address Book endpoint on the Exchange server.
An error occurred while testing the address book endpoint.
Test Steps

Testing the address book "Check Name" operation for user tim@xxx.com against server mail.xxx.com.An error occurred while attempting to resolve the name.
Additional Details
A protocol layer error occured. HttpStatusCode: 401

Failure LID: 47372

Failure Information:



###### REQUEST [2021-01-30T01:45:25.6194431Z] [ResolvedIPs: 24.xxx.xxx.41] ######



POST /mapi/nspi/?mailboxId=39a98ba9-9188-4474-9811-0ef5db77cf19@xxx.com HTTP/1.1

Content-Type: application/octet-stream

User-Agent: MapiHttpClient

X-RequestId: bb9a9e89-7bcb-48d8-8195-956d0fa21720:1

X-ClientInfo: 1215cce7-aa7c-4990-b850-1e2e98b589e5:1

client-request-id: a0ed4144-7731-4228-a2a6-3de91083488d

X-ClientApplication: MapiHttpClient/15.20.3391.4

X-RequestType: Bind

Authorization: Negotiate [truncated]

Host: mail.xxx.com

Content-Length: 0



--- REQUEST BODY [+0.103] ---

..[BODY SIZE: 45]



--- REQUEST SENT [+0.104] ---



###### RESPONSE [+0.155] ######



HTTP/1.1 401 Unauthorized

Connection: keep-alive

request-id: bf929243-8299-4de9-beb7-5c08bfd6ecf8

X-FailureContext: FrontEnd;401;VW5hdXRob3JpemVk;;;;

X-FEServer: EXCHANGE2016

Content-Length: 0

Date: Sat, 30 Jan 2021 01:45:25 GMT

Server: nginx

WWW-Authenticate: Basic [truncated]



--- RESPONSE BODY [+0.155] ---



--- RESPONSE DONE [+0.155] ---



###### EXCEPTION THROWN [+0.155] ######





HTTP Response Headers:

Connection: keep-alive

request-id: bf929243-8299-4de9-beb7-5c08bfd6ecf8

X-FailureContext: FrontEnd;401;VW5hdXRob3JpemVk;;;;

X-FEServer: EXCHANGE2016

Content-Length: 0

Date: Sat, 30 Jan 2021 01:45:25 GMT

Server: nginx

WWW-Authenticate: Basic realm="mail.xxx.com"



HTTP Status Code: 401 Unauthorized


[Fright]

well i don't have Exch2K16 infrastructure on hand to test what i have to say. so everything else is pure theory  ;)

firstly, I see that you turned on ntlm on the mapi virtual dir and I see "Authorization: Negotiate" in the response. So try to leave the basic authentication only.

AFAIK basic authentication is still supported (even on more recent versions) although M$ is going to abandon this and send everyone to oauth2.
so in theory everything should work - just a matter of config.
BUT, you need to understand that changes in settings concern not only external, but also internal clients, and you should always check how changing the settings will affect the behavior of all clients.
it is difficult to talk about the details without knowing all the details: which clients are used in the local network, which protocols are used in this case (mapi, mapi over http, rpc over http or some) etc.
after some changes, changes may be required on the client side. for example, if I remember correctly, some versions of Outlook do not store passwords for basic authentication and the user will have to enter credentials each time Outlook starts or it will be necessary to turn off the mapi over http and leave only the Outlook Anywhere via GPO. in general, this is a rather complex issue if corporate infrastructure is involved.
and different solutions are possible

but returning to your question - try to leave the basic auth only for mapi virtual dir

[klamath]

I appreciate all the help so far, but I for the life of me cant figure out the knob to turn to allow basic auth without negotiate on /mapi/nspi without breaking the exchange install.  Is proxy_pass an option?  I see it referenced a bit to allow NTLM auth with nginx? [1,2,3]

Tim
[1] https://gist.github.com/enoch85/573dac9005f0c8f1b826cc22e520e0ae
[2] https://stackoverflow.com/questions/14839712/nginx-reverse-proxy-passthrough-basic-authenication
[3] http://blog.manton.im/2016/04/configure-nginx-with-exchange-2010-2013.html

[Fright]

to turn to allow basic auth without negotiate on /mapi/nspi without breaking the exchange install

what breaks on Exch in this case?

I see it referenced a bit to allow NTLM auth with nginx?

ntlm is a part of commercial subscription
we can try to strip negotiate and ntlm header with proxy_hide_header and test if you want.
but you will need to edit the template by hand and add configuration files

[klamath]

to turn to allow basic auth without negotiate on /mapi/nspi without breaking the exchange install

what breaks on Exch in this case?

I see it referenced a bit to allow NTLM auth with nginx?

ntlm is a part of commercial subscription
we can try to strip negotiate and ntlm header with proxy_hide_header and test if you want.
but you will need to edit the template by hand and add configuration files

I was wondering about proxy_pass and the more_headers [1], it seems there is a good amount of luck with using that with Exchange.  As for what breaks, OWA and EAP wont allow login anymore and the exchange remote tester still fails in the same spot. I can have both basic and ntlm enabled at the same time but I have that negotiate header still reported via the remote connectivity tester. 

[1]https://forum.opnsense.org/index.php?topic=16595.0

[Fright]

As for what breaks, OWA and EAP wont allow login anymore

hm.Yes, sorry, can't comment on this without seeing the servers, logs etc

I was wondering about proxy_pass and the more_headers

proxy_pass is standard directive that sets the protocol and address of a backend server (or upstream). already used in each location )
headers_more is a 3d-party module not included in nginx plugin for OPNSense.
as i said we can try to do the same (strip ntlm and negotiate auth headers from response so external clients will not try to use ntlm. internal clients still can use ntlm in this case) with standard proxy_hide_header directive from http_proxy module.
i will try to test this method with my test-exch2k7 within a couple of days

[Fright]

Hi
I tested 3 options (actually 4, but in one with error_page it strangely removes not all headers on nginx) with a check on the M$ connectivity test. All works:
1. NGINX. Removing all WWW-Authenticate headers and adding one WWW-Authenticate header with forcing basic authentication in the location block:

Code Select Expand


    proxy_hide_header WWW-Authenticate;
    add_header WWW-Authenticate "Basic realm=mail.xxx.com" always;

  pros: no 3d-party modules involved
  cons:   you will need to modify template (to add a hook for location extra config) and add one file in dir.
technically this is not entirely correct. the "WWW-Authenticate Basic realm = mail.xxx.com" header will be sent with every server response. it will probably be ignored in cases where it is not needed, but nevertheless it is not entirely correct ..

2. NGINX. using the map directive in the http block, removing the headers and adding a new with forcing basic authentication in the location block:
add to http block:

Code Select Expand


map $status $forceBasic{
        401    'Basic realm=mail.xxx.com';
}

location block:

Code Select Expand


    proxy_hide_header WWW-Authenticate;
    add_header WWW-Authenticate $forceBasic always;

  pros: no 3d-party modules involved. more correct way to add WWW-Authenticate header (only on 401 status response)
  cons: even more manual labor

3. HAProxy chain. you can make a proxy chain. with HAProxy between NGINX and Exch (haproxy and nginx on OPN). in this case you can delete WWW-Authenticate headers and insert new one with basic auth forcing on HAProxy.
  pros: everything can be done through a GUI. you can use the HAProxy for additional traffic manipulation.
  cons: installing a plugin just for the sake of removing the header


   

[klamath]

Thank you for all the research on this matter, is there any option you would recommend if I am serving up multiple https endpoints from the name instance of nginx?  If you can provide some basic steps to fit your recommended choice into my config I would appreciate it.


Tim