16
20.7 Legacy Series / Re: NGINX plugin does not copy cert to /keys for TCP stream proxy
« on: January 15, 2021, 02:31:12 pm »
Opened a issue for this https://github.com/opnsense/plugins/issues/2189
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
drwxr-x--- 2 root wheel 512 Jan 11 10:39 .this is what shows up in the general log:
drwxr-xr-x 6 root wheel 512 Jan 11 10:22 ..
-rw------- 1 root wheel 0 Jan 11 11:58 trust_upstream_123dd4dd-f4c7-4e86-bce7-85b2817d5096.pem
2021-01-11T22:47:22 configd.py[20102] [cf2d62f8-6b50-41c4-96ad-32fde3196537] returned exit status 1and the result from the shell is:
2021-01-11T22:47:21 configd.py[20102] [cf2d62f8-6b50-41c4-96ad-32fde3196537] restarting nginx
2021-01-11T22:46:57 configd.py[20102] [4b91bdca-a882-417e-b71e-c578a7fb58d6] returned exit status 1
2021-01-11T22:46:57 configd.py[20102] [4b91bdca-a882-417e-b71e-c578a7fb58d6] starting nginx
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //etc/newsyslog.conf.d/nginx
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //usr/local/etc/php-fpm.d/webgui.conf
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //usr/local/etc/php-fpm.d/www.conf
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //etc/rc.conf.d/php_fpm
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //usr/local/etc/nginx/mime.types
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //usr/local/etc/nginx/nginx_web.conf
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //usr/local/etc/nginx/nginx.conf
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //etc/rc.conf.d/nginx
2021-01-11T22:46:55 configd.py[20102] [531046f7-c20b-48ea-9a1d-8c50c13ef04a] trigger config changed event
2021-01-11T22:46:55 configd.py[20102] generate template container OPNsense/Nginx
2021-01-11T22:46:55 configd.py[20102] [2ef28f04-4391-4987-a910-a2111951eb69] generate template OPNsense/Nginx
# configctl nginx restartNothing real helpful I am afraid..
Error (1)
cannot load certificate "/usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
...
stream {
...
# UPSTREAM SERVERS
upstream upstream123dd4ddf4c74e86bce785b2817d5096 {
hash $remote_addr consistent;
server IP:636 weight=1 max_conns=5 max_fails=2 fail_timeout=20;
server IP:636 weight=1 max_conns=5 max_fails=2 fail_timeout=20;
}
# upstream maps
include opnsense_stream_vhost_plugins/*.conf;
# servers
server {
listen 63636 ssl;
listen [::]:63636 ssl;
access_log /var/log/nginx/stream_f5e949f2-0d6b-42a8-8c52-9706945f9454.access.log main;
error_log /var/log/nginx/stream_f5e949f2-0d6b-42a8-8c52-9706945f9454.error.log info;
ssl_client_certificate /usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454_ca.pem;
ssl_verify_client off;
ssl_certificate_key /usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454.key;
ssl_certificate /usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_dhparam /usr/local/etc/dh-parameters.4096;
ssl_ciphers 'ECDHE-ECDSA-CAMELLIA256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CAMELLIA256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-CAMELLIA128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CAMELLIA128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-CAMELLIA256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-CAMELLIA256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-CAMELLIA128-SHA256:ECDHE-RSA-AES128-SHA256';
ssl_session_timeout 1d;
ssl_session_cache shared:sslcachef5e949f20d6b42a88c529706945f9454:50m;
ssl_session_tickets off;
ssl_prefer_server_ciphers on;
proxy_ssl on;
proxy_pass upstream123dd4ddf4c74e86bce785b2817d5096;
proxy_protocol off;
}
}
...
nc -v -s SECONDARY-IP PIMARY-IP 519
the same commands fails on the non-working interfaces. Anetstat -nl |grep .519
on the primary and secondary shows only established connections on the working interfaces and SYN_SENTs on the secondary on the non-working interfaces. A packet-capture on the primary non-working interfaces on port 519 show a lot of retransmissions.2020-06-15T17:09:57 configd.py: [0fb34a55-97c6-43c2-9970-93d830c6d2fa] Show log
2020-06-15T17:09:55 configd.py: [015978b3-82c5-4622-8312-01211854eb95] New IPv6 on pppoe0
2020-06-15T17:09:53 configd.py: [95fa9549-993f-4f2b-a9cb-7e92071cf70e] New IPv6 on pppoe0
2020-06-15T17:09:51 configd.py: [ff2cd119-f604-4f41-b1aa-89bf1688a329] New IPv6 on pppoe0
2020-06-15T17:09:48 configd.py: [e20f47d0-cc9d-46d3-ae5e-277cd3b2af38] New IPv6 on pppoe0
2020-06-15T17:09:46 configd.py: [33785606-da95-456f-b719-3129a66f03f6] New IPv6 on pppoe0
2020-06-15T17:09:44 configd.py: [346acd6c-067a-4101-b4fd-91f1e6485ab0] New IPv6 on pppoe0
2020-06-15T17:09:42 configd.py: [bdd02580-61d6-4edd-9d21-db5369272253] New IPv6 on pppoe0
2020-06-15T17:09:39 configd.py: [99b9c282-5327-4ced-9e97-df2b373f6201] New IPv6 on pppoe0
2020-06-15T17:09:37 configd.py: [1abd0b63-ddfd-4a2e-9466-c163a626f458] New IPv6 on pppoe0
2020-06-15T17:09:35 configd.py: [1c38df23-60ee-4227-8e63-5ed593e3a729] New IPv6 on pppoe0
2020-06-15T17:09:33 configd.py: [138e2c9e-ff33-4ae3-91d5-dffbf24fc6d8] New IPv6 on pppoe0
2020-06-15T17:09:30 configd.py: [c47db9e4-4388-43e2-b2b5-f5349a90d31d] New IPv6 on pppoe0
2020-06-15T17:09:28 configd.py: [2ca23137-9a2f-4a98-93aa-1c6ebd08289c] New IPv6 on pppoe0
2020-06-15T17:09:26 configd.py: [89e19975-4ef6-4cf7-b9a1-652d0f1e3549] New IPv6 on pppoe0
2020-06-15T17:09:23 configd.py: [a986bd9d-e382-4f12-a629-965ee5203860] New IPv6 on pppoe0
2020-06-15T17:09:21 configd.py: [6322d5c4-2763-4557-b9e6-61747be608d8] New IPv6 on pppoe0
2020-06-15T17:09:19 configd.py: [d3ee9140-5772-4564-b550-97ac466e8ebb] New IPv6 on pppoe0
2020-06-15T17:09:17 configd.py: [1beb8c8a-e22e-4079-8693-603eb2c55491] New IPv6 on pppoe0
2020-06-15T17:09:14 configd.py: [77e5b4df-99a8-44dc-8b88-1c90ad3ff1bc] New IPv6 on pppoe0
2020-06-15T17:09:12 configd.py: [82c360e8-364d-42a9-95f3-4cbc0c691bee] New IPv6 on pppoe0
2020-06-15T17:09:10 configd.py: [e6791ef5-eaeb-41d1-a5f2-e0330b8b5fda] New IPv6 on pppoe0
2020-06-15T17:09:08 configd.py: [4ccfe9a9-4e6c-43ba-a03e-b18bb283c1f8] New IPv6 on pppoe0
2020-06-15T17:09:05 configd.py: [11fb3e74-bc05-4097-9388-b18b50d7cc73] New IPv6 on pppoe0
2020-06-15T17:09:03 configd.py: [b100a94e-8308-46be-a47b-b46fb27603a1] New IPv6 on pppoe0
2020-06-15T17:09:01 configd.py: [59a2f606-187b-4378-9f37-cb1c3ca9f96a] New IPv6 on pppoe0
2020-06-15T17:08:58 configd.py: [c54ccfb1-a452-41c8-bcad-635f6b860b3a] New IPv6 on pppoe0
2020-06-15T17:08:56 configd.py: [0adc9d30-7d24-476f-94b2-97377a89e440] New IPv6 on pppoe0
2020-06-15T17:08:54 configd.py: [227d4883-ee15-41c9-a5f4-541d98847686] New IPv6 on pppoe0
2020-06-15T17:10:43 dhcp6c[94655]: reset timer for pppoe0 to 0.982245
2020-06-15T17:10:43 dhcp6c[94655]: server ID: 00:02:00:00:05:83:64:63:3a:33:38:3a:65:31:3a:31:30:3a:62:66:3a:63:30:00:00:00, pref=-1
2020-06-15T17:10:43 dhcp6c[94655]: get DHCP option DNS, len 32
2020-06-15T17:10:43 dhcp6c[94655]: status code: no addresses
2020-06-15T17:10:43 dhcp6c[94655]: get DHCP option status code, len 43
2020-06-15T17:10:43 dhcp6c[94655]: IA_NA: ID=0, T1=0, T2=0
2020-06-15T17:10:43 dhcp6c[94655]: get DHCP option identity association, len 59
2020-06-15T17:10:43 dhcp6c[94655]: DUID: 00:02:00:00:05:83:64:63:3a:33:38:3a:65:31:3a:31:30:3a:62:66:3a:63:30:00:00:00
2020-06-15T17:10:43 dhcp6c[94655]: get DHCP option server ID, len 26
2020-06-15T17:10:43 dhcp6c[94655]: DUID: 00:01:00:01:25:bf:1e:ec:dc:58:bc:e0:0a:3a
2020-06-15T17:10:43 dhcp6c[94655]: get DHCP option client ID, len 14
2020-06-15T17:10:43 dhcp6c[94655]: receive advertise from fe80::de38:e1ff:fe10:bb8b%pppoe0 on pppoe0
2020-06-15T17:10:43 dhcp6c[94655]: reset a timer on pppoe0, state=SOLICIT, timeo=0, retrans=1030
2020-06-15T17:10:43 dhcp6c[94655]: send solicit to ff02::1:2%pppoe0
2020-06-15T17:10:43 dhcp6c[94655]: set option request (len 4)
2020-06-15T17:10:43 dhcp6c[94655]: set elapsed time (len 2)
2020-06-15T17:10:43 dhcp6c[94655]: set identity association
2020-06-15T17:10:43 dhcp6c[94655]: set client ID (len 14)
2020-06-15T17:10:43 dhcp6c[94655]: a new XID (256156) is generated
2020-06-15T17:10:43 dhcp6c[94655]: Sending Solicit
2020-06-15T17:10:43 dhcp6c[94655]: got an expected reply, sleeping.
2020-06-15T17:10:43 dhcp6c[94655]: removing server (ID: 00:02:00:00:05:83:64:63:3a:33:38:3a:65:31:3a:31:30:3a:62:66:3a:63:30:00:00:00)
2020-06-15T17:10:43 dhcp6c[94655]: removing an event on pppoe0, state=REQUEST
2020-06-15T17:10:43 dhcp6c[94655]: script "/var/etc/dhcp6c_wan_script.sh" terminated
2020-06-15T17:10:43 opnsense: plugins_configure hosts (execute task : unbound_hosts_generate())
2020-06-15T17:10:43 opnsense: plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2020-06-15T17:10:43 opnsense: plugins_configure hosts ()
2020-06-15T17:10:43 opnsense: /usr/local/etc/rc.newwanipv6: The command '/sbin/route add -host -'inet6' '2606:4700:4700::1001' 'fe80::de38:e1ff:fe10:bb8b%'' returned exit code '71', the output was 'route: fe80::de38:e1ff:fe10:bb8b%: hostname nor servname provided, or not known'
2020-06-15T17:10:43 opnsense: /usr/local/etc/rc.newwanipv6: The command '/sbin/route add -host -'inet6' '2606:4700:4700::1111' 'fe80::de38:e1ff:fe10:bb8b%'' returned exit code '71', the output was 'route: fe80::de38:e1ff:fe10:bb8b%: hostname nor servname provided, or not known'
2020-06-15T17:10:43 opnsense: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
2020-06-15T17:10:43 opnsense: plugins_configure dhcp (,inet6)
2020-06-15T17:10:43 opnsense: /usr/local/etc/rc.newwanipv6: On (IP address: 2003:cd:efff:4b13:de58:bcff:fee0:a3a) (interface: WAN[wan]) (real interface: pppoe0).
2020-06-15T17:10:43 opnsense: /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0'
2020-06-15T17:10:42 dhcp6c: dhcp6c REQUEST on pppoe0 - running newipv6
2020-06-15T17:10:42 dhcp6c[94655]: executes /var/etc/dhcp6c_wan_script.sh
2020-06-15T17:10:42 dhcp6c[94655]: reset a timer on pppoe0, state=INIT, timeo=0, retrans=196
2020-06-15T17:10:42 dhcp6c[94655]: remove an IA: NA-0
2020-06-15T17:10:42 dhcp6c[94655]: IA NA-0 is invalidated
2020-06-15T17:10:42 dhcp6c[94655]: status code for NA-0: no addresses
2020-06-15T17:10:42 dhcp6c[94655]: make an IA: NA-0
2020-06-15T17:10:42 dhcp6c[94655]: nameserver[1] 2003:180:2::53
2020-06-15T17:10:42 dhcp6c[94655]: nameserver[0] 2003:180:2:6000::53
2020-06-15T17:10:42 dhcp6c[94655]: Received REPLY for REQUEST
2020-06-15T17:10:42 dhcp6c[94655]: get DHCP option DNS, len 32
2020-06-15T17:10:42 dhcp6c[94655]: status code: no addresses
2020-06-15T17:10:42 dhcp6c[94655]: get DHCP option status code, len 43
2020-06-15T17:10:42 dhcp6c[94655]: IA_NA: ID=0, T1=0, T2=0
2020-06-15T17:10:42 dhcp6c[94655]: get DHCP option identity association, len 59
2020-06-15T17:10:42 dhcp6c[94655]: DUID: 00:02:00:00:05:83:64:63:3a:33:38:3a:65:31:3a:31:30:3a:62:66:3a:63:30:00:00:00
2020-06-15T17:10:42 dhcp6c[94655]: get DHCP option server ID, len 26
2020-06-15T17:10:42 dhcp6c[94655]: DUID: 00:01:00:01:25:bf:1e:ec:dc:58:bc:e0:0a:3a
2020-06-15T17:10:42 dhcp6c[94655]: get DHCP option client ID, len 14
2020-06-15T17:10:42 dhcp6c[94655]: receive reply from fe80::de38:e1ff:fe10:bb8b%pppoe0 on pppoe0
2020-06-15T17:10:42 dhcp6c[94655]: reset a timer on pppoe0, state=REQUEST, timeo=0, retrans=963
2020-06-15T17:10:42 dhcp6c[94655]: send request to ff02::1:2%pppoe0
2020-06-15T17:10:42 dhcp6c[94655]: set option request (len 4)
2020-06-15T17:10:42 dhcp6c[94655]: set elapsed time (len 2)
2020-06-15T17:10:42 dhcp6c[94655]: set identity association
2020-06-15T17:10:42 dhcp6c[94655]: set status code
2020-06-15T17:10:42 dhcp6c[94655]: set server ID (len 26)
2020-06-15T17:10:42 dhcp6c[94655]: set client ID (len 14)
2020-06-15T17:10:42 dhcp6c[94655]: a new XID (b7a5b) is generated
2020-06-15T17:10:42 dhcp6c[94655]: Sending Request
2020-06-15T17:10:42 dhcp6c[94655]: picked a server (ID: 00:02:00:00:05:83:64:63:3a:33:38:3a:65:31:3a:31:30:3a:62:66:3a:63:30:00:00:00)
2020-06-15T17:10:41 dhcp6c[94655]: reset timer for pppoe0 to 0.981837
2020-06-15T17:10:41 dhcp6c[94655]: server ID: 00:02:00:00:05:83:64:63:3a:33:38:3a:65:31:3a:31:30:3a:62:66:3a:63:30:00:00:00, pref=-1
2020-06-15T17:10:41 dhcp6c[94655]: get DHCP option DNS, len 32
2020-06-15T17:10:41 dhcp6c[94655]: status code: no addresses
2020-06-15T17:10:41 dhcp6c[94655]: get DHCP option status code, len 43
2020-06-15T17:10:41 dhcp6c[94655]: IA_NA: ID=0, T1=0, T2=0
2020-06-15T17:10:41 dhcp6c[94655]: get DHCP option identity association, len 59
2020-06-15T17:10:41 dhcp6c[94655]: DUID: 00:02:00:00:05:83:64:63:3a:33:38:3a:65:31:3a:31:30:3a:62:66:3a:63:30:00:00:00
2020-06-15T17:10:41 dhcp6c[94655]: get DHCP option server ID, len 26
2020-06-15T17:10:41 dhcp6c[94655]: DUID: 00:01:00:01:25:bf:1e:ec:dc:58:bc:e0:0a:3a
2020-06-15T17:10:41 dhcp6c[94655]: get DHCP option client ID, len 14
2020-06-15T17:10:41 dhcp6c[94655]: receive advertise from fe80::de38:e1ff:fe10:bb8b%pppoe0 on pppoe0
2020-06-15T17:10:41 dhcp6c[94655]: reset a timer on pppoe0, state=SOLICIT, timeo=0, retrans=1075
2020-06-15T17:10:41 dhcp6c[94655]: send solicit to ff02::1:2%pppoe0
2020-06-15T17:10:41 dhcp6c[94655]: set option request (len 4)
2020-06-15T17:10:41 dhcp6c[94655]: set elapsed time (len 2)
2020-06-15T17:10:41 dhcp6c[94655]: set identity association
2020-06-15T17:10:41 dhcp6c[94655]: set client ID (len 14)
2020-06-15T17:10:41 dhcp6c[94655]: a new XID (7f2be7) is generated
2020-06-15T17:10:41 dhcp6c[94655]: Sending Solicit
2020-06-15T17:10:41 dhcp6c[94655]: got an expected reply, sleeping.
2020-06-15T17:10:41 dhcp6c[94655]: removing server (ID: 00:02:00:00:05:83:64:63:3a:33:38:3a:65:31:3a:31:30:3a:62:66:3a:63:30:00:00:00)
2020-06-15T17:10:41 dhcp6c[94655]: removing an event on pppoe0, state=REQUEST
2020-06-15T17:10:41 dhcp6c[94655]: script "/var/etc/dhcp6c_wan_script.sh" terminated
2020-06-15T17:10:41 opnsense: plugins_configure hosts (execute task : unbound_hosts_generate())
2020-06-15T17:10:41 opnsense: plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2020-06-15T17:10:41 opnsense: plugins_configure hosts ()
2020-06-15T17:10:41 opnsense: /usr/local/etc/rc.newwanipv6: The command '/sbin/route add -host -'inet6' '2606:4700:4700::1001' 'fe80::de38:e1ff:fe10:bb8b%'' returned exit code '71', the output was 'route: fe80::de38:e1ff:fe10:bb8b%: hostname nor servname provided, or not known'
2020-06-15T17:10:41 opnsense: /usr/local/etc/rc.newwanipv6: The command '/sbin/route add -host -'inet6' '2606:4700:4700::1111' 'fe80::de38:e1ff:fe10:bb8b%'' returned exit code '71', the output was 'route: fe80::de38:e1ff:fe10:bb8b%: hostname nor servname provided, or not known'
2020-06-15T17:10:40 opnsense: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
2020-06-15T17:10:40 opnsense: plugins_configure dhcp (,inet6)
2020-06-15T17:10:40 opnsense: /usr/local/etc/rc.newwanipv6: On (IP address: 2003:cd:efff:4b13:de58:bcff:fee0:a3a) (interface: WAN[wan]) (real interface: pppoe0).
2020-06-15T17:10:40 opnsense: /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0'
2020-06-15T17:10:40 dhcp6c: dhcp6c REQUEST on pppoe0 - running newipv6
2020-06-15T17:10:40 dhcp6c[94655]: executes /var/etc/dhcp6c_wan_script.sh
2020-06-15T17:10:40 dhcp6c[94655]: reset a timer on pppoe0, state=INIT, timeo=0, retrans=635