ssh/sftp-Verbindungen mal [pass], mal [block] bei Ansible-Modul apt

[MrTomtom]

Hallo zusammen

ich habe folgendes Problem auf meiner opnsense-Firewall.

Auf einem Server läuft unter Docker das Tool semaphoreui zur Verwaltung von Ansible-Skripten/Jobs. Einer meiner Jobs prüft auf bestimmten Servern, ob neue OS-Packete vorhanden sind. Dazu wird das Ansible-Modul apt verwendet. Das hat auch bis zum Update (23.7.10 --> 24.7.6) der opnsense am 12.1. funktioniert.

Nun zur genauen Problem-Beschreibung:

Semaphore-Host ist 10.20.1.1 (VLAN ID 20) (Netzwerk 10.20.0.0/16)
Remote-Host ist 10.1.3.209 (Hostname docker301-lxc) (VLAN ID 1) (Netzwerk 10.1.0.0/16)

Aufruf nur den einen Ansible-Moduls mit Dubug-Option:
ansible docker103-lxc -m ansible.builtin.apt -a "update_cache=yes" --become --private-key ~/ansible_id_ed25519 -i inventories/pve -J -vvv

Ausgabe des Skripts:

cba315ae59be:/tmp/semaphore/inventory_251$ ansible docker103-lxc -m ansible.builtin.apt -a "update_cache=yes" --become --private-key ~/ansible_id_ed25519 -i inventories/pve -J -vvv
ansible [core 2.18.1]
  config file = /tmp/semaphore/inventory_251/ansible.cfg
  configured module search path = ['/home/semaphore/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/semaphore/apps/ansible/11.1.0/venv/lib/python3.11/site-packages/ansible
  ansible collection location = /home/semaphore/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/semaphore/apps/ansible/11.1.0/venv/bin/ansible
  python version = 3.11.11 (main, Dec  6 2024, 11:15:21) [GCC 13.2.1 20231014] (/opt/semaphore/apps/ansible/11.1.0/venv/bin/python3)
  jinja version = 3.1.5
  libyaml = True
Using /tmp/semaphore/inventory_251/ansible.cfg as config file
Vault password:
host_list declined parsing /tmp/semaphore/inventory_251/inventories/pve/hosts.yml as it did not pass its verify_file() method
script declined parsing /tmp/semaphore/inventory_251/inventories/pve/hosts.yml as it did not pass its verify_file() method
Parsed /tmp/semaphore/inventory_251/inventories/pve/hosts.yml inventory source with yaml plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
<docker103-lxc> ESTABLISH SSH CONNECTION FOR USER: ansible
<docker103-lxc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/semaphore/ansible_id_ed25519"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o 'ControlPath="/home/semaphore/.ansible/cp/cbf443803c"' docker103-lxc '/bin/sh -c '"'"'echo ~ansible && sleep 0'"'"''
<docker103-lxc> (0, b'/home/ansible\n', b'OpenSSH_9.6p1, OpenSSL 3.1.7 3 Sep 2024
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 22: include /etc/ssh/ssh_config.d/*.conf matched no files
debug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/home/semaphore/.ssh/known_hosts\'
debug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/home/semaphore/.ssh/known_hosts2\'
debug1: auto-mux: Trying existing master at \'/home/semaphore/.ansible/cp/cbf443803c\'
debug1: Control socket "/home/semaphore/.ansible/cp/cbf443803c" does not exist
debug2: resolving "docker103-lxc" port 22
debug3: resolve_host: lookup docker103-lxc:22
debug3: channel_clear_timeouts: clearing
debug3: ssh_connect_direct: entering
debug1: Connecting to docker103-lxc [10.1.3.209] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 10000 ms remain after connect
debug1: identity file /home/semaphore/ansible_id_ed25519 type 3
debug1: identity file /home/semaphore/ansible_id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.2p1 Debian-2+deb12u4
debug1: compat_banner: match: OpenSSH_9.2p1 Debian-2+deb12u4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to docker103-lxc:22 as \'ansible\'
debug3: record_hostkey: found key type ED25519 in file /home/semaphore/.ssh/known_hosts:28
debug3: record_hostkey: found key type RSA in file /home/semaphore/.ssh/known_hosts:33
debug3: record_hostkey: found key type ECDSA in file /home/semaphore/.ssh/known_hosts:34
debug3: load_hostkeys_file: loaded 3 keys from docker103-lxc
debug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: zlib@openssh.com,zlib,none
debug2: compression stoc: zlib@openssh.com,zlib,none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512,sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,kex-strict-s-v00@openssh.com
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug3: kex_choose_conf: will use strict KEX ordering
debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:psCPM8+NwIxlX0VEP6GcEniXs5YxWJkAiO5Mw5UcI8Y
debug3: record_hostkey: found key type ED25519 in file /home/semaphore/.ssh/known_hosts:28
debug3: record_hostkey: found key type RSA in file /home/semaphore/.ssh/known_hosts:33
debug3: record_hostkey: found key type ECDSA in file /home/semaphore/.ssh/known_hosts:34
debug3: load_hostkeys_file: loaded 3 keys from docker103-lxc
debug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host \'docker103-lxc\' is known and matches the ED25519 host key.
debug1: Found key in /home/semaphore/.ssh/known_hosts:28
debug3: send packet: type 21
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug3: kex_input_ext_info: extension server-sig-algs
debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com,ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>
debug3: kex_input_ext_info: extension publickey-hostbound@openssh.com
debug1: kex_ext_info_check_ver: publickey-hostbound@openssh.com=<0>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred: ,gssapi-keyex,hostbased,publickey
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Will attempt key: /home/semaphore/ansible_id_ed25519 ED25519 SHA256:VofkRfc+b7TzPcux4/dKXzDzL/eFlVeG48C1yl0Gvq8 explicit
debug2: pubkey_prepare: done
debug1: Offering public key: /home/semaphore/ansible_id_ed25519 ED25519 SHA256:VofkRfc+b7TzPcux4/dKXzDzL/eFlVeG48C1yl0Gvq8 explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /home/semaphore/ansible_id_ed25519 ED25519 SHA256:VofkRfc+b7TzPcux4/dKXzDzL/eFlVeG48C1yl0Gvq8 explicit
debug3: sign_and_send_pubkey: using publickey-hostbound-v00@openssh.com with ED25519 SHA256:VofkRfc+b7TzPcux4/dKXzDzL/eFlVeG48C1yl0Gvq8
debug3: sign_and_send_pubkey: signing using ssh-ed25519 SHA256:VofkRfc+b7TzPcux4/dKXzDzL/eFlVeG48C1yl0Gvq8
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Enabling compression at level 6.
Authenticated to docker103-lxc ([10.1.3.209]:22) using "publickey".
debug1: setting up multiplex master socket
debug3: muxserver_listen: temporary control path /home/semaphore/.ansible/cp/cbf443803c.PPm8BN2fHAQMtuW4
debug2: fd 4 setting O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug1: channel 0: new mux listener [/home/semaphore/.ansible/cp/cbf443803c] (inactive timeout: 0)
debug3: muxserver_listen: mux listener channel 0 fd 4
debug2: fd 3 setting TCP_NODELAY
debug3: set_sock_tos: set socket 3 IP_TOS 0x20
debug1: control_persist_detach: backgrounding master process
debug2: control_persist_detach: background process is 97677
debug2: fd 4 setting O_NONBLOCK
debug1: forking to background
debug1: Entering interactive session.
debug1: pledge: id
debug3: client_repledge: enter
debug2: set_control_persist_exit_time: schedule exit in 60 seconds
debug1: multiplexing control connection
debug2: fd 5 setting O_NONBLOCK
debug3: fd 5 is O_NONBLOCK
debug1: channel 1: new mux-control [mux-control] (inactive timeout: 0)
debug3: channel_post_mux_listener: new mux channel 1 fd 5
debug3: mux_master_read_cb: channel 1: hello sent
debug2: set_control_persist_exit_time: cancel scheduled exit
debug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4
debug2: mux_master_process_hello: channel 1 client version 4
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4
debug2: mux_master_process_alive_check: channel 1: alive check
debug3: mux_client_request_alive: done pid = 97681
debug3: mux_client_request_session: session request sent
debug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 78
debug2: mux_master_process_new_session: channel 1: request tty 0, X 0, agent 0, subsys 0, term "xterm", cmd "/bin/sh -c \'echo ~ansible && sleep 0\'", env 0
debug3: mux_master_process_new_session: got fds stdin 6, stdout 7, stderr 8
debug2: fd 7 setting O_NONBLOCK
debug2: fd 8 setting O_NONBLOCK
debug1: channel 2: new session [client-session] (inactive timeout: 0)
debug2: mux_master_process_new_session: channel_new: 2 linked to control channel 1
debug2: channel 2: send open
debug3: send packet: type 90
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: client_input_hostkeys: received RSA key SHA256:JUXXn0obbvutUtF5DR+Lr1p1rrFUQkM45SW1Cz0RlOE
debug3: client_input_hostkeys: received ECDSA key SHA256:l8g7wsgFjwLn9/CHdB1gV55ipyxX3hXOsF3vxtG9JTk
debug3: client_input_hostkeys: received ED25519 key SHA256:psCPM8+NwIxlX0VEP6GcEniXs5YxWJkAiO5Mw5UcI8Y
debug1: client_input_hostkeys: searching /home/semaphore/.ssh/known_hosts for docker103-lxc / (none)
debug3: hostkeys_foreach: reading file "/home/semaphore/.ssh/known_hosts"
debug3: hostkeys_find: found ssh-ed25519 key at /home/semaphore/.ssh/known_hosts:28
debug3: hostkeys_find: found ssh-rsa key at /home/semaphore/.ssh/known_hosts:33
debug3: hostkeys_find: found ecdsa-sha2-nistp256 key at /home/semaphore/.ssh/known_hosts:34
debug3: hostkeys_find: found ssh-ed25519 key under different name/addr at /home/semaphore/.ssh/known_hosts:37
debug3: hostkeys_find: found ssh-ed25519 key under different name/addr at /home/semaphore/.ssh/known_hosts:38
debug1: client_input_hostkeys: searching /home/semaphore/.ssh/known_hosts2 for docker103-lxc / (none)
debug1: client_input_hostkeys: hostkeys file /home/semaphore/.ssh/known_hosts2 does not exist
debug3: client_input_hostkeys: 3 server keys: 0 new, 3 retained, 0 incomplete match. 0 to remove
debug1: client_input_hostkeys: no new or deprecated keys from server
debug3: client_repledge: enter
debug3: receive packet: type 4
debug1: Remote: /home/ansible/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 4
debug1: Remote: /home/ansible/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 2: callback start
debug2: client_session2_setup: id 2
debug1: Sending command: /bin/sh -c \'echo ~ansible && sleep 0\'
debug2: channel 2: request exec confirm 1
debug3: send packet: type 98
debug3: client_repledge: enter
debug3: mux_session_confirm: sending success reply
debug2: channel_input_open_confirmation: channel 2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug1: mux_client_request_session: master session id: 2
debug2: channel 2: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: exec request accepted on channel 2
debug3: receive packet: type 96
debug2: channel 2: rcvd eof
debug2: channel 2: output open -> drain
debug2: channel 2: obuf empty
debug2: chan_shutdown_write: channel 2: (i0 o1 sock -1 wfd 7 efd 8 [write])
debug2: channel 2: output drain -> closed
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
debug3: mux_exit_message: channel 2: exit message, exitval 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0
debug2: channel 2: rcvd eow
debug2: chan_shutdown_read: channel 2: (i0 o3 sock -1 wfd 6 efd 8 [write])
debug2: channel 2: input open -> closed
debug3: receive packet: type 97
debug2: channel 2: rcvd close
debug3: channel 2: will not send data after close
debug2: channel 2: send close
debug3: send packet: type 97
debug2: channel 2: is dead
debug2: channel 2: gc: notify user
debug3: mux_master_session_cleanup_cb: entering for channel 2
debug2: channel 1: rcvd close
debug2: channel 1: output open -> drain
debug2: chan_shutdown_read: channel 1: (i0 o1 sock 5 wfd 5 efd -1 [closed])
debug2: channel 1: input open -> closed
debug2: channel 2: gc: user detached
debug2: channel 2: is dead
debug2: channel 2: garbage collecting
debug1: channel 2: free: client-session, nchannels 3
debug3: channel 2: status: The following connections are open:
  #1 mux-control (t16 [mux-control] nr0 i3/0 o1/16 e[closed]/0 fd 5/5/-1 sock 5 cc -1 io 0x03/0x00)
  #2 client-session (t4 [session] r0 i3/0 o3/0 e[write]/0 fd -1/-1/8 sock -1 cc -1 io 0x00/0x00)

debug2: channel 1: obuf empty
debug2: chan_shutdown_write: channel 1: (i3 o1 sock 5 wfd 5 efd -1 [closed])
debug2: channel 1: output drain -> closed
debug3: mux_client_read_packet_timeout: read header failed: Broken pipe
debug2: Received exit status from master 0
debug2: channel 1: is dead (local)
debug2: channel 1: gc: notify user
debug3: mux_master_control_cleanup_cb: entering for channel 1
debug2: channel 1: gc: user detached
debug2: channel 1: is dead (local)
debug2: channel 1: garbage collecting
debug1: channel 1: free: mux-control, nchannels 2
debug3: channel 1: status: The following connections are open:
  #1 mux-control (t16 [mux-control] nr0 i3/0 o3/0 e[closed]/0 fd 5/5/-1 sock 5 cc -1 io 0x00/0x00)

debug2: set_control_persist_exit_time: schedule exit in 60 seconds
')
<docker103-lxc> ESTABLISH SSH CONNECTION FOR USER: ansible
<docker103-lxc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/semaphore/ansible_id_ed25519"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o 'ControlPath="/home/semaphore/.ansible/cp/cbf443803c"' docker103-lxc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/ansible/.ansible/tmp `"&& mkdir "` echo /home/ansible/.ansible/tmp/ansible-tmp-1737888680.0324144-97675-279371810568143 `" && echo ansible-tmp-1737888680.0324144-97675-279371810568143="` echo /home/ansible/.ansible/tmp/ansible-tmp-1737888680.0324144-97675-279371810568143 `" ) && sleep 0'"'"''
<docker103-lxc> (0, b'ansible-tmp-1737888680.0324144-97675-279371810568143=/home/ansible/.ansible/tmp/ansible-tmp-1737888680.0324144-97675-279371810568143\n', b"OpenSSH_9.6p1, OpenSSL 3.1.7 3 Sep 2024
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 22: include /etc/ssh/ssh_config.d/*.conf matched no files
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/semaphore/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/semaphore/.ssh/known_hosts2'
debug1: auto-mux: Trying existing master at '/home/semaphore/.ansible/cp/cbf443803c'
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 97681
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet_timeout: read header failed: Broken pipe
debug2: Received exit status from master 0
")
Using module file /opt/semaphore/apps/ansible/11.1.0/venv/lib/python3.11/site-packages/ansible/modules/apt.py
<docker103-lxc> PUT /home/semaphore/.ansible/tmp/ansible-local-97666k28ns3ut/tmpqqvf6d5l TO /home/ansible/.ansible/tmp/ansible-tmp-1737888680.0324144-97675-279371810568143/AnsiballZ_apt.py
<docker103-lxc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/semaphore/ansible_id_ed25519"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o 'ControlPath="/home/semaphore/.ansible/cp/cbf443803c"' '[docker103-lxc]'
[WARNING]: sftp transfer mechanism failed on [docker103-lxc]. Use ANSIBLE_DEBUG=1 to see detailed information
<docker103-lxc> SSH: EXEC scp -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/semaphore/ansible_id_ed25519"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o 'ControlPath="/home/semaphore/.ansible/cp/cbf443803c"' /home/semaphore/.ansible/tmp/ansible-local-97666k28ns3ut/tmpqqvf6d5l '[docker103-lxc]:/home/ansible/.ansible/tmp/ansible-tmp-1737888680.0324144-97675-279371810568143/AnsiballZ_apt.py'

Im Live-Log der opnsense sehe ich folgende Einträge:

Interface      Time   Source   Destination   Proto   Label   
[block] VLAN20_Server      2025-01-26T11:52:13   10.20.1.1:56308   10.1.3.209:22   tcp   VL20_Server: default block IPv4   
[block] VLAN20_Server      2025-01-26T11:52:13   10.20.1.1:56308   10.1.3.209:22   tcp   VL20_Server: default block IPv4   
[pass] VLAN20_Server      2025-01-26T11:51:19   10.20.1.1:56308   10.1.3.209:22   tcp   VL20_Server: Allow traffic to local subnets

Meine Rule "VL20_Server: Allow traffic to local subnets" erlaubt den Traffik auf Port 22 für TCP/UDP zu allen internen Netzwerken. Die Rule "VL20_Server: default block IPv4" ist meine letzte Rule, die jeglichen Traffik verhindert und protokolliert.

Es scheint so, als funktioniere der erste Aufruf von ssh problemlos, jedoch alle weiteren Verbindungsversuche über den Controller scheitern irgendwann mit timeout und es werden von der Firewall (aus meiner Sicht) die gleichen Verbindungen nun geblockt. Bei den Prozessen auf dem Semaphore-Host sehe ich jeweils:

98080 semaphor  0:00 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o IdentityFile="/home/semaphore/ansible_id_ed25519" -o KbdInteract
98085 semaphor  0:00 ssh: /home/semaphore/.ansible/cp/cbf443803c [mux]

Der SKript mit dem apt-Modul lief bis zum FW-Update ohne Probleme.


Was ich nicht verstehe ist, dass ein erster Verbindungsversuch erfolgreich ist, die nachfolgenden aber geblockt werden.

Hat da jemand eine Erklärung/Lösung.

Besten Dank
Thomas

[Patrick M. Hausen]

So was riecht nach asymmetrischem Routing. Mal doch bitte mal dein Netzwerk auf und poste außerdem die relevanten Regeln.

[MrTomtom]

Besten Dank Patrick für die rasche Rückfrage.

Gerne die weiteren Informationen.

Das Netzwerk sieht so aus für die relevanten Objekte:
You cannot view this attachment.

Und die Filewall-Rules beim VLAN20:
You cannot view this attachment.

als Detail (keine advanced features):
You cannot view this attachment.

Im Alias LOCAL_SUBNETS sind alle lokalen Subnetze zusammengefasst, im Alias Allowed_IN_Ports_to_LOCALNET sind einige Ports (darunter 22) zusammengefasst.

Asymetrisches routing könnte sein --> weshalb gings aber vor dem Update?

Und klar könnte ich das Ganze übers VLAN20 machen, das geht nun auch. Aber ich möchte den Fehler verstehen und ggf. meine Rules richtig anpassen.

[Patrick M. Hausen]

Was macht die Verbindung vom Proxmox-Host zum VLAN 20 da? Der Linux-Container hat Adressen in beiden Netzen? Einmal 10.1.3.209 und einmal 10.20.3.209?

Dann ist doch klar, was passiert. Der Docker-Host 10.20.1.1 schickt seine Pakete über die OPNsense zum Ziel, das Zielsystem aber antwortet direkt, da es ja ein Interface im VLAN 20 hat. Das ist Standard-Routing.

Das triggert bei der OPNsense die "default deny / out of state" Regel.

Zwei Lösungen:

- Routing fixen, jedes System nur ein Interface in einem VLAN - oder
- auf dem LAN Interface ausgehend NAT auf die LAN-Adresse der OPNsense, wenn die Quelle VLAN 20 ist (Firewall > NAT > Outbound)

HTH,
Patrick

[MrTomtom]

Hallo Patrick

deine Erklärung ist nachvollziehbar und verständlich.

Ja, der Pwoxmox-Host hat zwei Interfaces, eines ins LAN und eines ins VLAN20. Auch bekommen die Linux-Container beide Interfaces mit eignene Mac-Adressen. Das Default-Routing geht beim docker103-lxc über 10.20.0.1 (VLAN20):

thomas@docker103-lxc:~$ ip r
default via 10.20.0.1 dev eth0
10.1.0.0/16 dev eth1 proto kernel scope link src 10.1.3.209
10.20.0.0/16 dev eth0 proto kernel scope link src 10.20.3.209

Ich benötige aber auch beide Interfaces am Proxmox, da ich z.B. auch den Unifi-Controller darauf laufen lasse, bei dem die Verwaltung der Unifi-Komponenten über das 10.1.0.0-Netzwerk geht.

Für mich sind aber noch zwei Dinge nicht schlüssig:

• Warum das Problem erst seit dem Upgrade?
• Wenn ich den ssh-Befehl oder den sftp-Befehl auf dem Docker-Host (im Semaphore-Container) ausführe, dann gehen die problemlos:
  

  cba315ae59be:/tmp/semaphore/inventory_251$ ssh -v -o StrictHostKeyChecking=no -o 'IdentityFile="/home/semaphore/ansible_id_ed25519"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' 10.1.3.209 exit
  OpenSSH_9.6p1, OpenSSL 3.1.7 3 Sep 2024
  debug1: Reading configuration data /etc/ssh/ssh_config
  debug1: /etc/ssh/ssh_config line 22: include /etc/ssh/ssh_config.d/*.conf matched no files
  debug1: Connecting to 10.1.3.209 [10.1.3.209] port 22.
  debug1: Connection established.
  debug1: identity file /home/semaphore/ansible_id_ed25519 type 3
  debug1: identity file /home/semaphore/ansible_id_ed25519-cert type -1
  debug1: Local version string SSH-2.0-OpenSSH_9.6
  debug1: Remote protocol version 2.0, remote software version OpenSSH_9.2p1 Debian-2+deb12u4
  debug1: compat_banner: match: OpenSSH_9.2p1 Debian-2+deb12u4 pat OpenSSH* compat 0x04000000
  debug1: Authenticating to 10.1.3.209:22 as 'ansible'
  debug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts2: No such file or directory
  debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
  debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
  debug1: SSH2_MSG_KEXINIT sent
  debug1: SSH2_MSG_KEXINIT received
  debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com
  debug1: kex: host key algorithm: ssh-ed25519
  debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
  debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
  debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
  debug1: SSH2_MSG_KEX_ECDH_REPLY received
  debug1: Server host key: ssh-ed25519 SHA256:psCPM8+NwIxlX0VEP6GcEniXs5YxWJkAiO5Mw5UcI8Y
  debug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts2: No such file or directory
  debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
  debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
  debug1: Host '10.1.3.209' is known and matches the ED25519 host key.
  debug1: Found key in /home/semaphore/.ssh/known_hosts:37
  debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
  debug1: rekey out after 134217728 blocks
  debug1: SSH2_MSG_NEWKEYS sent
  debug1: expecting SSH2_MSG_NEWKEYS
  debug1: ssh_packet_read_poll2: resetting read seqnr 3
  debug1: SSH2_MSG_NEWKEYS received
  debug1: rekey in after 134217728 blocks
  debug1: SSH2_MSG_EXT_INFO received
  debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com,ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>
  debug1: kex_ext_info_check_ver: publickey-hostbound@openssh.com=<0>
  debug1: SSH2_MSG_SERVICE_ACCEPT received
  debug1: Authentications that can continue: publickey,password
  debug1: Next authentication method: publickey
  debug1: Will attempt key: /home/semaphore/ansible_id_ed25519 ED25519 SHA256:VofkRfc+b7TzPcux4/dKXzDzL/eFlVeG48C1yl0Gvq8 explicit
  debug1: Offering public key: /home/semaphore/ansible_id_ed25519 ED25519 SHA256:VofkRfc+b7TzPcux4/dKXzDzL/eFlVeG48C1yl0Gvq8 explicit
  debug1: Server accepts key: /home/semaphore/ansible_id_ed25519 ED25519 SHA256:VofkRfc+b7TzPcux4/dKXzDzL/eFlVeG48C1yl0Gvq8 explicit
  Authenticated to 10.1.3.209 ([10.1.3.209]:22) using "publickey".
  debug1: channel 0: new session [client-session] (inactive timeout: 0)
  debug1: Requesting no-more-sessions@openssh.com
  debug1: Entering interactive session.
  debug1: pledge: filesystem
  debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
  debug1: client_input_hostkeys: searching /home/semaphore/.ssh/known_hosts for 10.1.3.209 / (none)
  debug1: client_input_hostkeys: searching /home/semaphore/.ssh/known_hosts2 for 10.1.3.209 / (none)
  debug1: client_input_hostkeys: hostkeys file /home/semaphore/.ssh/known_hosts2 does not exist
  debug1: client_input_hostkeys: host key found matching a different name/address, skipping UserKnownHostsFile update
  debug1: Remote: /home/ansible/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
  debug1: Remote: /home/ansible/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
  debug1: Sending command: exit
  debug1: pledge: fork
  debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
  debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
  debug1: channel 0: free: client-session, nchannels 1
  Transferred: sent 3436, received 3900 bytes, in 0.1 seconds
  Bytes per second: sent 32205.8, received 36554.9
  debug1: Exit status 0
  

• Ebenfalls geht ja der erste Aufruf von Ansible apt durch und aktiviert das SSH-Multiplexing. Die nachfolgenden Verbindungen über das Multiplexing haben ein Problem.

Ich finde zu dem Verhalten einfach noch keine logische Erklärung. Die von dir angesprochene Regel "default deny / out of state" scheint bei mir gar nicht aktiv zu sein:
You cannot view this attachment.

Kannst du dir das erklären?

Besten Dank

[Patrick M. Hausen]

Wieso nicht aktiv? Das rote Kreuz sagt, dass das eine "deny" und keine "allow" Regel ist. Die ist immer aktiv, die kannst du auch nicht weg konfigurieren oder deaktivieren. Außerdem hat sie nicht "quick" gesetzt, das sind die kleinen gelben Blitze bei den anderen Regeln. Das bedeutet, sie greift immer nach allen anderen Regeln, die den Blitz haben.

[MrTomtom]

Stimmt, mein Fehler. Macht auch Sinn bei einer automatisch generierten Regel...

Hast du aber noch eine Erklärung, weshalb die ssh-Verbindung ohne Multiplex geht und bei der zweiten Verbindung (in Ansible) mit Multiplex nicht?

[Patrick M. Hausen]

Dazu müsste man sich einmal einen tcpdump auf allen Interfaces auf dem Weg angucken, was da genau wo lang läuft.

Wenn der Docker-Container ein Bein im VLAN 20 hat, was für einen Sinn hat es dann, die Verbindung über die Firewall zu routen? Weshalb nimmst du nicht gleich die 10.20.3.209 als ansible_host? Da ist doch sowieso keinerlei Schutz vorhanden bei einer Verbindung ins selbe Netz.

[meyergru]

Ich benötige aber auch beide Interfaces am Proxmox, da ich z.B. auch den Unifi-Controller darauf laufen lasse, bei dem die Verwaltung der Unifi-Komponenten über das 10.1.0.0-Netzwerk geht.

Das habe ich nicht verstanden. Der Proxmox-Host selbst braucht doch nur ein logisches Interface, unter dem er erreichbar ist. Natürlich sollen die VMs und Container auf verschiedene Netze zugreifen können, aber das regelt man doch über VLANs, auf die der Proxmox selbst gar nicht zugreifen können muss.

Ich mache das auch so und da habe ich beispielsweise folgende Konfiguration in /etc/network/interfaces:

Code Select Expand

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
#10 Gbit/s Trunk

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eth0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        bridge-mcsnoop 0
#Network bridge

auto lan0
iface lan0 inet static
        address 192.168.1.5/24
        gateway 192.168.1.1
        vlan-id 10
        vlan-raw-device vmbr0
        accept_ra 2
        autoconf 1
        pre-up echo 2 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra
        post-up echo 2 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra
#pvehost lan

source /etc/network/interfaces.d/*

Somit kann sich jede VM an ein beliebiges VLAN anklemmen, indem man in der virtuellen Schnittstelle die Bridge vmbr0 und das VLAN angibt.
Wie man oben sieht, gilt das auch für den Proxmox-Host selbst, er hat halt ein Bein (lan0) an der Bridge, in meinem Fall im VLAN 10 (untagged würde man keins angeben, aber bekanntlich kann ein Mix ja Probleme verursachen).

[MrTomtom]

Coole Ideen, besten Dank...

Da mach ich mich doch nochmals hinter das Netzwerkdesign. Das mit dem "Auseinandernehmen" von den Interfaces und dem Durchreichen an die Container tönt sehr spannend.

Ich werde wieder berichten.

Schönes Restwochenende
Thomas