device not pinging when accessed from opnsense box

[stormy]

I got a pretty simple reproducible case.

1) opnsense box is at 192.168.1.2

2) connected to a swtich, which holds a cisco voip phone with DHCP/IP set on the opnsense to always be 192.168.1.82

3) The cisco phone cannot make the vpn connection, unless an out-NAT rule is added, that is working fine.

4) upon ISP outage, the opnsense, re-establishes internet connection, HOWEVER, the cisco voip fails to re-connect/authenticate despite the fact that a) it was running days before that w/o any issues, and b) all network is restored to all other devices c) the outbound NAT rule is in place.

5) I've debugged that for few months, and narrowed the issue to the fact that the opnsense box (192.168.1.2) cannot PING the cisco phone 192.168.1.82, after such outage, despite the fact that OTHER machines on the same switch can ping the cisco's IP just fine, before, and after outage..  Also, the 192.168.1.82 IP is marked as allocated to the cisco phone, and 'arp' output on the opnsense and another random PC agree on the cisco's MAC address, e.g.  opnsense:

Code: [Select]

? (192.168.1.82) at 08:cc:68:xx:xx:xx on bridge0 expires in 1176 seconds [bridge]
?
From random PC that can ping it:

Code: [Select]

192.168.1.82          08-cc-68-xx-xx-xx     dynamic

6) The "workaround" of course is to reboot the cisco voip, and then it works fine, however, that was not the case with my prior (hiding: tomato) firmware..

Is there any chance someone can help debug this? why is the ping working for the random PC, yet it does not for the opnsense box, which i suspect is the reason for the failure of the cisco phone to build the vpn tunnel..

Appreciate any tips.
Stormy.

[guest15389]

Running the latest? I'm not sure as I don't see a bridge0 interface on my install.

I have my 2 interfaces set at em0 and em1 and all the arps look to be correct.

If you can get a shell and you tcpdump that interface, are you seeing the traffic?

You can also do a floating rule to capture any traffic (log) for that IP to see if something is being blocked, but that would seem odd.

[stormy]

Started using O/S since 17.1, then updated to latest, and most recently running 17.1.2. It reproduced at least 10 times, each time got a bit closer to diagnosis

tcpdump from the opnsense box is as follows:

Code: [Select]

21:53:27.743148 IP OPNsense.localdomain > 192.168.1.82: ICMP echo request, id 12037, seq 0, length 64
21:53:27.746797 ARP, Request who-has OPNsense.localdomain tell 192.168.1.82, length 50
21:53:28.746825 ARP, Request who-has OPNsense.localdomain tell 192.168.1.82, length 50
21:53:28.747657 IP OPNsense.localdomain > 192.168.1.82: ICMP echo request, id 12037, seq 1, length 64

whereas on the PC sitting on that switch, it can ping the 192.168.1.82, and also ssh to the opnsense box (192.168.1.2) just fine.

the 'arp' output has 'bridge0', that is expected, part of initial configuration, it is not trivial to setup, or understand from the GUI, but ip 192.168.1.2 belongs to this bridge0, and that's how i ssh to the opnsense box from the lan.

the point is that it cannot ping the 192.168.1.82, and i've now rebooted the opnsense into 17.1.3, and same thing, although one can argue that the dhcp lease has expired due to reboot.. I bet if I reboot the PHONE, then it will connect just fine...  I know networking, but never got down to the bits like that, so not sure how to further debug..

Stormy.

[guest15389]

If you run from the OPNsense shell:

Code: [Select]

tcpdump -i bridge0 host 192.168.1.2

and you try the pings, do you get any output back?

[stormy]

on opnsense box the arp table shows it as:

? (192.168.1.82) at 08:cc:68:xx:xx:xx on bridge0 expires in 1065 seconds [bridge]

ping does not return when issued on the opnsense box, it seems the cisco phone cannot "find" the opnsense for some reason:

Code: [Select]

22:40:37.036722 IP OPNsense.localdomain > 192.168.1.82: ICMP echo request, id 34087, seq 0, length 64
22:40:37.044966 ARP, Request who-has OPNsense.localdomain tell 192.168.1.82, length 50
22:40:37.044971 ARP, Reply OPNsense.localdomain is-at 02:b2:2d:xx:xx:xx (oui Unknown), length 32
22:40:37.142473 ARP, Reply OPNsense.localdomain is-at 02:b2:2d:xx:xx:xx (oui Unknown), length 28
22:40:38.037821 IP OPNsense.localdomain > 192.168.1.82: ICMP echo request, id 34087, seq 1, length 64
22:40:38.045163 ARP, Request who-has OPNsense.localdomain tell 192.168.1.82, length 50
22:40:38.045172 ARP, Reply OPNsense.localdomain is-at 02:b2:2d:xx:xx:xx (oui Unknown), length 32
22:40:39.045122 ARP, Request who-has OPNsense.localdomain tell 192.168.1.82, length 50
22:40:39.045131 ARP, Reply OPNsense.localdomain is-at 02:b2:2d:xx:xx:xx (oui Unknown), length 32
22:40:39.045189 IP OPNsense.localdomain > 192.168.1.82: ICMP echo request, id 34087, seq 2, length 64
22:40:39.804002 IP OPNsense.localdomain > 192.168.1.82: ICMP echo request, id 13867, seq 0, length 64
22:40:40.857483 IP OPNsense.localdomain > 192.168.1.82: ICMP echo request, id 13867, seq 1, length 64
22:40:40.865206 ARP, Request who-has OPNsense.localdomain tell 192.168.1.82, length 50
22:40:40.865214 ARP, Reply OPNsense.localdomain is-at 02:b2:2d:xx:xx:xx (oui Unknown), length 32
22:40:41.865232 ARP, Request who-has OPNsense.localdomain tell 192.168.1.82, length 50
22:40:41.865241 ARP, Reply OPNsense.localdomain is-at 02:b2:2d:xx:xx:xx (oui Unknown), length 32

maybe that .localdomain is confusing it? not sure...  Thanks for the help!

[guest15389]

Hmm, is there any VLANs or anything else that in between the router and your cisco device? Almost seems like there is something else that is answer for it. like a proxy arp or some NAT'ing that is happening.

[stormy]

Like I said, been tracking this for several weeks now..

I know this may sound outrageous, but I think there is a bug/issue or misconfiguration on my part on the opnsense, which I probably stand no chance in figuring out on my own, but really I stuck to a very basic setup at first, no MultiWAN (which i had working on tomato firmware).

net topology is very simple/basic:

ISP -> opnsense (17.1.3) -> 8-port non-managed switch ->  (PC1, PC2, Cisco Phone, etc.)

When the condition happens, the cisco phone (.82) can be pinged from PC1, PC2, but, not from opnsense box.  This means that the cisco phone is not unable to connect no matter what, until it is rebooted, and somehow connection is re-established with opnsense, then vpn works, etc.

I've been thinking it is the firewall rule, and  played around with it, but now i'm pretty sure it is OK, just the reboot on the cisco fixes it.

running wireshark on PC1 and pinging from that node to 192.168.1.82 works fine, and shows request/reply right away, but not from the opnsense.  even tried FLOOD pinging from the opnsense box, did:

ping -f 192.168.1.82

this is tcpdump at that time, and we even see .82 is asking where the opnsense box is, but for some reason, absolutely no replies back:

Code: [Select]

09:12:06.119746 IP OPNsense.localdomain > 192.168.1.82: ICMP echo request, id 34144, seq 281, length 64
09:12:06.119801 IP OPNsense.localdomain.ssh > 192.168.1.195.10973: Flags [P.], seq 10293:10329, ack 108, win 513, length 36
09:12:06.120132 IP 192.168.1.195.10973 > OPNsense.localdomain.ssh: Flags [.], ack 10329, win 63144, length 0
09:12:06.123524 ARP, Request who-has OPNsense.localdomain tell 192.168.1.82, length 50
09:12:06.123533 ARP, Reply OPNsense.localdomain is-at 02:e4:c2:xx:xx:xx (oui Unknown), length 32
09:12:06.130492 IP OPNsense.localdomain > 192.168.1.82: ICMP echo request, id 34144, seq 282, length 64
09:12:06.130584 IP OPNsense.localdomain.ssh > 192.168.1.195.10973: Flags [P.], seq 10329:10365, ack 108, win 513, length 36
09:12:06.141206 IP OPNsense.localdomain > 192.168.1.82: ICMP echo request, id 34144, seq 283, length 64
09:12:06.141299 IP OPNsense.localdomain.ssh > 192.168.1.195.10973: Flags [P.], seq 10365:10401, ack 108, win 513, length 36
09:12:06.141644 IP 192.168.1.195.10973 > OPNsense.localdomain.ssh: Flags [.], ack 10401, win 63072, length 0
09:12:06.151922 IP OPNsense.localdomain > 192.168.1.82: ICMP echo request, id 34144, seq 284, length 64
09:12:06.152016 IP OPNsense.localdomain.ssh > 192.168.1.195.10973: Flags [P.], seq 10401:10437, ack 108, win 513, length 36


The cisco phone and opnsense connect to the 8-port switch, and i've changed ports, all of them are good, still no change.. PCs can all ping the cisco (b/c they only go on the switch), but, the opnsense cannot go beyond the switch and ping that phone.

Note, this is only for the cisco phone, the opnsense can ping just fine PC1, PC2, etc. so it must be something to do with the NAT/outbound rule..

It includes two rules for 192.168.1.0, one specifically for the ADSL modem, and another for the Cisco phone (rest are auto-generated):

[stormy]

Just to clarify, the FIRST rule is supposed to allow access for the cisco vpn phone.

the SECOND rule, allows the opnsense to ping 11.0.0.11 (the ISP modem's internal IP).

So, in summary, the opnsense can ping ALL machines, EXCEPT the cisco phone, 192.168.1.82.

I might be able to setup access if someone wants

Maybe this can help?

Code: [Select]

root@OPNsense:~ # pfctl -sr | grep -i 192
block drop in log on ! bridge0 inet from 192.168.1.0/24 to any
block drop in log inet from 192.168.1.2 to any
block drop in log quick on pppoe0 inet from 192.168.0.0/16 to any label "Block private networks from 3WANC"
block drop in log quick on igb3 inet from 192.168.0.0/16 to any label "Block private networks from 3WANCS"
block drop in log quick on igb1 inet from 192.168.0.0/16 to any label "Block private networks from 4WANH"
pass in quick on bridge0 inet from 192.168.1.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule"

Final thoughts: I would have no clue how to setup a "proxy arp" or anything beyond basic stuff.. I'm a Linux guy, but not much freebsd exposure until now

Also, i've powered OFF all other network gear (that was hanging from that switch), and then rebooted all remaining (including the opnsense), except the cisco phone, and condition remains..

[stormy]

ok, confirmed, rebooting the cisco phone immediately allows opnsense to ping it (192.168.1.82), and also phone can connect/establish vpn tunnel.

The question, why so drastic measure, power cycle a phone, why opnsense cannot ping a simple IP on the LAN...

right now issue is resolved, however, next time ISP goes down, then I'm 99% sure the opnsense will stop being able to ping the cisco phone

[stormy]

Maybe there's an alternative way (not DMZ) to allow setting up the cisco vpn phone behind opnsense box that will not have this issue?

[stormy]

One more piece of info. Once cisco VPN phone is rebooted, it is pingable on 192.168.1.82 from both PCs on LAN as well as opnsense box. However, after VPN tunnel is established by cisco phone (it connects to service), then it is only pingable from opnsense box, and NOT from all the PCs.. I think this is expected, but just funny, that before opnsense could not ping, and now PCs cannot ping, so it must be related to the rule/handling?

Does anyone have such a cisco VPN phone that requires NAT outbound rule and can test it?

[stormy]

Confirmed , this "loss-of-contact" happens only after an ISP outage.. network reconnects OK (adsl pppoe) by opnsense, but, from that point on, the opnsense box cannot ping the cisco phone (192.168.1.82), however, all other LAN members can ping that cisco phone just fine.  On the phone's menu it shows the correct IP.

The only workaround i found is to reboot the CISCO PHONE, rebooting everything else does not work.

I know with 99.99% certainty that this is not an issue with the phone, b/c it worked for ~3 years with linksys/tomato firmware..  also, phone responds fine to other pings.. I think somehow the "outbound NAT" rules got confused in the opnsense and prevent it from getting, or seeing the packets from the cisco phone..  I'm just not knowledgable enough to prove it..

also, is there any way to put the phone (192.168.1.82) in a DMZ instead of the outbound NAT? that may help to debug further, if that does not show the symptom, then we know it's the outbound NAT rule(s) I have (see above).

OR, if there's another form of workaround instead of physically being by the phone and pulling it from power source, and reconnecting, that would be cool.

Thanks for any tips.

[stormy]

btw, if someone is curious enough, i may be able to setup some login rules so you can sniff around via gui or ssh, really interested in getting to bottom of this, not urgent, but a puzzle I'd love to solve

[stormy]

I guess I'm the only one hitting such an issue..  here's a summary, network is basic:

ciscophone -> 8-port-switch -> opnsense -> adsl-router
          PC1   ->
          PC2   ->

phone at: 192.168.1.82, opnsense at 192.168.1.2.  I've setup a NAT outbound rule to allow the cisco to create a VPN connection (never had to do this with tomato or other firmwares), I think this is the source of the issue, but was told , this is a must w/opnsense.

The issue is simple, once all is powered on fresh, all PCs and opnsense can ping 192.168.1.82 (ciscophone).
As soon as VPN tunnel is established by phone, then only opnsense box can ping it, PCs cannot - this is probably expected.

Next, an event takes place, ISP outage, where by the ADSL modem is either powered off, or no network for some time.  The VPN tunnel by phone expires as expected.   At this point all PCs can ping the phone, EXCEPT, opnsense box cannot ping 192.168.1.82.

The ONLY workaround is to physically power cycle the phone.

Tried rebooting the opnsense box, pulling wires, etc. nothing helps restore the pinging abilities of opnsense to the phone.

It sorts of points to the phone as the issue, however, that config worked 3+ years w/tomato/tt-wrt and no such issues, further, phone does return the pings to PCs.

I think something with the tunnel gets mixed up in the opnsense box that tells it the phone is now in a different "area", and thus cannot ping it.  I've posted traces above in post, if anyone is interested, it's now in this state, if not, maybe some other time

Any ideas on how to debug OR, configure cisco phone without NAT outbound rules, would be appreciated.  As far as I'm concerned any IP from "inside" opnsense side should be allowed to do ANYTHING it wants, is there a way to disable this extra security that requires NAT outbound? and allow any internal IP to connect in any way it wants to outside world?

Sorry for asking what may sound like silly questions   Just being brave, maybe others like me are shy to ask the "obvious" ?

[stormy]

So, this super friendly djGrrrrrrrr contacted over IRC, what can I say... he spent a few good HOURS, had me pull cables and reboot everything except the cisco phone, he can comment further, but bottom line he thinks the phone is not acting nicely, in any event, he added this FLOATING rule:

IPv4 *      LAN net      *      *      *      *           Allow LAN to Any

also removed all the DHCP lease times on the opensense box, and we'll see. A quick test showed it OK, but only time will tell....    I've never had any such issues w/tomato/ddwrt, but then maybe they "accomodate" more for such mis-behaviors.. basically the phone did not "see" the arps sent from the opnsense, even though we plugged it DIRECTLY into the LAN port on the opnsense box itself.

Time to wait, hope is high.. new SW 8 Trailer is out!!

Stormy..