Opnsense NGINX R-Proxy w/ Apache help?

[ThyOnlySandman]

Really struggling with getting a few Apache virtual servers to work consistently behind Opnsense NGINX.
I know its more of Apache2 + NGINX thing here - But I'm bit confused with the Opnsense wizardry not using standard nginx configs that I reference online.

Which is my first question.  Possible to completely override the opnsense NGINX plugin config - and use basic config?  Is that supported or must use GUI?
-----

I have a single page website - index.html being served via port 80.  It has a javascript that connects to a external yellowpages API server also on port 80.

yp.mydomain1.com.conf
<VirtualHost *:80>
        ServerName yp.mydomain1.com
        DocumentRoot /home/www/yp.mydomain1.com
        DirectoryIndex index.html
        ErrorLog ${APACHE_LOG_DIR}/wp.mydomain1.com-error.log
        CustomLog ${APACHE_LOG_DIR}/wp.mydomain1.com-access.log combined
</VirtualHost>

Setup the 4 NGINX components. 
Server, Local IP, 80
Location:  /
HTTP Server:  yp.mydomain.com1 , 80

It works.  THEN.
I'm also wanting apache multi-site wordpress.  Doing the following gets me my first working TLS offload wordpress site, but then breaks my yellowpages site.

wordpress confg
                                                                                                                                                        mydomain2.com.conf                                                                                                                                                                         
<VirtualHost *:80>

        ServerName mydomain2.com
        ServerAlias www.mydomain2.com

        DocumentRoot /home/www/mydomain2.com
        DirectoryIndex index.html index.php
       
        <Directory /home/www/mydomain2.com/>
        Options FollowSymLinks
        AllowOverride Limit Options FileInfo
        DirectoryIndex index.php
        Require all granted
        </Directory>
        <Directory /home/www/mydomain2.com/wp-conent>
         Options FollowSymLinks
         Require all granted
        </Directory>

 ErrorLog ${APACHE_LOG_DIR}/mydomain2.com-error.log
 CustomLog ${APACHE_LOG_DIR}/mydomain2.com-access.log combined
</VirtualHost>

--------

wp-config.php  (A LOT of time)

define('WP_HOME', 'http://mydomain2.com');
define('WP_SITEURL', 'http://mydomain2.com');

/** Make sure WordPress understands it's behind an SSL terminator */
define('FORCE_SSL_ADMIN', true);
define('FORCE_SSL_LOGIN', true);

if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')
$_SERVER['HTTPS']='on';
---------------
.htaccess

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
----------------

Opnsense NGINX:
Server:  LOCAL IP, Access 80
Location:  URL /
HTTP server:  mydomain2.com, X-Forward, 443 + 80, Cert
---------

So now I have my first TLS offload wordpress with wp-admin functional (although port 80 to 443 redirect rule not working yet - must provide https) , however now my yellowpages site now always returns 404. I've played with URL pattern.  I've played with NGINX specifying root dir + index.html - No go.  I'm lost with http header troubleshooting.  Not a web dev...but this shouldn't be so difficult. 
Any suggestions welcome.  Thanks.

[Fright]

please describe opn\nginx config in more detail

[ThyOnlySandman]

Well - this project is forcing me to address my high degree of apache ignorance.

Believe 90% all my troubles were do apache misconfigurations.  After reading and adjusting my apache.conf + 000-default.conf + site configs + /etc/hosts + DNS. I got both working together.  I also played around with NameVirtualHost *:8080 and testing NGINX / Apache on other listening ports.

The default wordpress .htaccess I posted must get loaded.
(AllowOverride Limit Options FileInfo)

And for the wordpress TLS offload / X-Forward the wp-config.php and/or .htaccess must get configured.  I'm using the php method I shared.

Also believe I encountered NGINX getting into a funk state with my troubleshooting / changing configs around.  Requiring me to disable entirely - stop / restart via lobby.  Re-enable. 

Please - can someone clarify on the NGINX plugin
1)  Can /usr/local/etc/nginx/nginx.conf be manually configured in or must one use GUI / upstream logic?
2)  Clarification on the restart / reload.  Green restart in plugin same as Green restart in lobby?  Red refresh just to re-load config without restarting / breaking existing connections?
3)  Is there a particular reason that even after NGINX plugin is disabled the service remains running?

NGINX config.  Defaults + :

Upstream servers
yp.mydomain1.com (server) --> 172.16.0.10  --> 80
mydomain2.com (server) --> 172.16.0.10 --> 80

Upsteams
yp.mydomain1.com (LB) --> yp.mydomain1.com (server)
mydomain2.com (LB)  --> mydomain2.com (server)

Locations: 
yp.mydomain1.com (Location)
URL Paturn:  /
Upstream Servers:  yp.mydomain1.com (LB)

mydomain2.com (Location)
URL Patturn:  /
Upstream Servers:  mydomain2.com (LB)

HTTP Server (yp.mydomain1.com)
HTTP Listen:  80
HTTPS Listen: 
Server Name:  yp.mydomain1.com
Locations:  yp.mydomain1.com (Location)

HTTP Server (mydomain2.com)
HTTP Listen:  80
HTTPS Listen:  443
Real IP Source:  X-Forwarded-For
Locations:  mydomain2.com (Location)
TLS Cert:  mydomain2.com
HTTPS Only

---------------

Also when playing with different apache listening ports upstream but leaving NGINX http server on 80.  The following inside ,htaccess would re-write URL properly.  I've yet to learn how to do this on NGINX rules - which I'm sure is more preferable.

RewriteCond %{SERVER_PORT} =8080
RewriteRule ^my_specified_string http://yp.mydomain1.com:8080%{REQUEST_URI} [NC,L,R]
-------

I have a lot more to learn on web / R-proxy  / backend ...

[Fright]

Can /usr/local/etc/nginx/nginx.conf be manually configured in or must one use GUI / upstream logic?

it will be overwritten next time templates applied (Apply button pressed)

Clarification on the restart / reload.  Green restart in plugin same as Green restart in lobby?  Red refresh just to re-load config without restarting / breaking existing connections?

afaik Green is just start
"refresh" - is Restart (without templates\config applying)

I got both working together

so everything is working now?

[ThyOnlySandman]

Yes both working.  I believe the apache config had misconfig with my default website / DNS.  Names were not matching up.  NGINX would give not found.

I am going to look at adding proxy proto to the yp.mydomain.com and understand more on a logging perspective.  Also need to understand more what's happening with source / destination IPs.  My end goal being to lock down my L3 switch ACLs to DMZ vlan so only opn/Nginx LAN INT can speak to apache.
My opnsense WAN rules BTW being
Source USA geo-ip + any port , des (this firewall) 80
Source USA geo-ip + any port , des (this firewall) 443

it will be overwritten next time templates applied (Apply button pressed)

Got it.  Would be useful IMO if plugin devs may consider nginx.conf referencing another config file that wouldn't reset. 

afaik Green is just start
"refresh" - is Restart (without templates\config applying)

THIS may have also been source of confusion...
See screenshot.  So top is start nginx, middle refresh WITHOUT applying new config changes.  Bottom is reload config without restarting nginx?
If I click stop on nginx plugin page it does nothing.  I must go to lobby and shutdown there...
Similarly - If I go to config - uncheck enable Nginx.  It does stop traffic / forwarding.  Yet service appears to remain running tho.  Must to go lobby to shutdown.

Thank you.

[ThyOnlySandman]

So after reading and learning.  Proxy Proto is not what I wanted.  In conclusion what I'll be using is X-Forward-For on my non wordpress sites as well  Following works.

a2enmod remoteip

<VirtualHost *:80>
     
        RemoteIPHeader X-Forwarded-For
        RemoteIPInternalProxy x.x.x.x
       
</VirtualHost>

-------
apache.conf

#Log real client IP
LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

-----

Perhaps one may find my verbal learning / sharing useful.
TTFN

[Fright]

Would be useful IMO if plugin devs may consider nginx.conf referencing another config file that wouldn't reset

the plugin uses hooks for this - static files loaded via the include directives in various nginx.conf places

Bottom is reload config without restarting nginx?

yes.interesting. never paid attention to this button)

It does stop traffic / forwarding.  Yet service appears to remain running tho

yes. it just applies an 'empty' template if I understand correctly

If I click stop on nginx plugin page it does nothing.  I must go to lobby and shutdown there...

yes. lobby uses different script for this (status_services.php). nginx plugin page uses API for this and the maintainer disabled this feature on purpose. I honestly don't really understand why

Code: [Select]

  /**
    *  override parent method - stopping nginx is not allowed because otherwise you would loose
    *  access to the web interface
    */
    public function stopAction()
    {
        return array('status' => 'failed');
    }



[ThyOnlySandman]

the plugin uses hooks for this - static files loaded via the include directives in various nginx.conf places

Thanks info.  Interesting.  I'll spend some time investigating these includes.

Unrelated but related - In past I have manually configured the NTOPNG plugin to keep my vlans / subnets organized for network view.  Plugin lacks this in GUI and adding [--local-networks|-m] <local nets> /usr/local/etc/ntopng.conf resets on opnsense reboot.  Really would like that to stick if one knows how.

[ThyOnlySandman]

The answer to my last NTOPNG resetting config / -m networks option is here:

https://www.ntop.org/guides/ntopng/third_party_integrations/opnsense.html

I was using the old...
Uninstall old
pkg add https://packages.ntop.org/FreeBSD/FreeBSD:12:amd64/latest/ntop-1.0.txz
Install os-ntopng-enterprise
Local Networks (-m) right in GUI.  Perfect!

Edit:  Apparently there are some inconsistencies in the GUI. (I'm guessing due to opnsense 21.1?)
The license tab appears to be missing to either specify enterprise license or community mode.
Adding " --community " to bottom of /usr/local/etc/ntopng.conf appears to work.  Don't know if it will stay following reboot yet...

Edit:  Upon reboot the license tab appeared and can execute NTOPNG in community mode.  As with old version -  Custom edits on /usr/local/etc/ntopng.conf will stay for service restart but not opnsense reboot.