OPNsense
  • Home
  • Help
  • Search
  • Login
  • Register

  • OPNsense Forum »
  • Profile of Gunni »
  • Show Posts »
  • Messages
  • Profile Info
    • Summary
    • Show Stats
    • Show Posts...
      • Messages
      • Topics
      • Attachments

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

  • Messages
  • Topics
  • Attachments

Messages - Gunni

Pages: [1]
1
Hardware and Performance / Re: [solved] Network performance problems OPNsense 10 times slower
« on: October 30, 2021, 03:47:16 pm »
I solved it.
Short explanation: dumb
Long explanation: duuuuuuuuuuuuumb ;)

I messed up the gateways. In the LAN interface the IPv4 upstream gateway was set to the wrong gateway.
Setting it to "auto-detect" solved my problem.
I do not remember if I  set it or why, it may be in the beginning where I tested to fit OPNsense in our infrastructure.

2
Hardware and Performance / Re: Network performance problems OPNsense 10 times slower http(s) via web proxy
« on: October 29, 2021, 10:18:00 am »
I do not think your problem is the same.
Someone else seems to have had my problem, but he gave up and had no solution:
https://forum.opnsense.org/index.php?topic=13391.0

3
Hardware and Performance / [solved] Network performance problems OPNsense 10 times slower
« on: October 26, 2021, 03:05:28 pm »
I have a network problem and maybe some expert here is able to help me. I hope I provide enough information, but if more information is needed I will provide it.
It would be nice if someone has an idea?

Problem:
I want to replace an old IPFire with the OPNSense, and when trying to test the Webproxy of OPNsense I run into massive performance problems.
For now I think it is a general network and not a proxy problem.

Environment:
A simplified topology is in the attachment image.

OPNsense version: OPNsense 21.7.3_3 (amd64/OpenSSL)
Connected two Gbit networks (igb) as LAGG with VLANs on it. In this example only one VLAN is used.

I have a parent proxy out of my control (but seems to be a squid):
Code: [Select]
admintools:~$ telnet [PARENTPROXY] 80
Trying [PARENTPROXY]...
Connected to [PARENTPROXY].
Escape character is '^]'.
sdfaasfg
HTTP/1.1 400 Bad Request
Server: squid/4.15
Mime-Version: 1.0
Date: Tue, 26 Oct 2021 12:43:44 GMT
...
My IPFire is the gateway with NATing and I defined it as LAN gateway in the OPNsense.
I can not activate the WAN interface on the OPNsense yet.

Testing:

For testing purpose I tried different curl downloads. The Linux (Ubuntu) and the OPNsense are in the same vlan.

OPNsense via the IPFire squid
OPNsense via the ParentProxy
Linux via IPFire squid
Linux via Parentproxy

As you can see below OPNsense via ParentProxy is 10 times slower than all other tests.
Any Ideas?


OPNsense via the IPFire squid
Code: [Select]
OPNsense3:~ # curl -x "http://[IPFIRE]:800" https://download.checkmk.com/checkmk/2.0.0p11/check-mk-raw-2.0.0p11_0.focal_amd64.deb --output test.deb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  112M  100  112M    0     0  14.3M      0  0:00:07  0:00:07 --:--:-- 14.2M
OPNsense via the ParentProxy (that is the slow one)
Code: [Select]
OPNsense3:~ # curl -x "http://[PARENTPROXY]:80" https://download.checkmk.com/checkmk/2.0.0p11/check-mk-raw-2.0.0p11_0.focal_amd64.deb --output test.deb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  112M  100  112M    0     0  1906k      0  0:01:00  0:01:00 --:--:-- 2029k
Linux via IPFire squid
Code: [Select]
admintools:~$ curl -x "http://[IPFIRE]:800" https://download.checkmk.com/checkmk/2.0.0p11/check-mk-raw-2.0.0p11_0.focal_amd64.deb --output test.deb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  112M  100  112M    0     0  21.2M      0  0:00:05  0:00:05 --:--:-- 23.9M
Linux via Parentproxy
Code: [Select]
admintools:~$ curl -x "http://[PARENTPROXY]:80" https://download.checkmk.com/checkmk/2.0.0p11/check-mk-raw-2.0.0p11_0.focal_amd64.deb --output test.deb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  112M  100  112M    0     0  18.7M      0  0:00:06  0:00:06 --:--:-- 21.4M
admintools:~$



tcpdumps:
I only provide the last rows of the dumps for now, but if more or special parameters are needed, please ask.
Maybe there are already hints about the problem. On the first view the OPNsense sends much more ACKs

OPNsense via the ParentProxy
Code: [Select]
12:25:57.901319 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118204373:118205671, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849161], length 1298
12:25:57.901325 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118205671:118206969, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849161], length 1298
12:25:57.901330 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118206969:118208267, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849161], length 1298
12:25:57.901334 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118208267:118209565, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849161], length 1298
12:25:57.901373 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118209565:118210863, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849161], length 1298
12:25:57.901384 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118210863:118212161, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849161], length 1298
12:25:57.901390 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118212161:118213459, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849161], length 1298
12:25:57.901396 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118213459:118214757, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849161], length 1298
12:25:57.901416 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118214757:118216055, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 1298
12:25:57.901492 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118203075, win 8693, options [nop,nop,TS val 3525849175 ecr 3328889438], length 0
12:25:57.901499 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118204373, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901614 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118216055:118217353, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 1298
12:25:57.901620 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118217353:118218651, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 1298
12:25:57.901625 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118218651:118219949, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 1298
12:25:57.901629 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118219949:118221247, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 1298
12:25:57.901716 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118221247:118222545, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 1298
12:25:57.901723 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118222545:118223843, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 1298
12:25:57.901745 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118205671, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901763 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118223843:118225141, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 1298
12:25:57.901768 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118225141:118226439, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 1298
12:25:57.901773 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118226439:118227737, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 1298
12:25:57.901781 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118206969, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901786 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118208267, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901790 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118209565, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901794 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118210863, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901810 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118227737:118229035, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 1298
12:25:57.901815 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], seq 118229035:118230333, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 1298
12:25:57.901819 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [P.], seq 118230333:118231311, ack 892, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849162], length 978
12:25:57.901835 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118212161, win 8692, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901852 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118213459, win 8682, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901858 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118214757, win 8672, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901862 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118216055, win 8662, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901878 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118217353, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901888 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118218651, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901903 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118219949, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901908 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118221247, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.901917 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118222545, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.902094 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118223843, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.902110 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118225141, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.902115 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118226439, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.902121 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118227737, win 8693, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.902125 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118229035, win 8689, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.902138 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118230333, win 8679, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.902187 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118231311, win 8671, options [nop,nop,TS val 3525849176 ecr 3328889438], length 0
12:25:57.902761 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [P.], seq 892:916, ack 118231311, win 8703, options [nop,nop,TS val 3525849177 ecr 3328889438], length 24
12:25:57.905826 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [F.], seq 916, ack 118231311, win 8703, options [nop,nop,TS val 3525849180 ecr 3328889438], length 0
12:25:57.916387 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [.], ack 917, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849177], length 0
12:25:57.916643 IP [PARENTPROXY].80 > [OPNSENSE].1475: Flags [F.], seq 118231311, ack 917, win 4096, options [nop,nop,TS val 3328889438 ecr 3525849177], length 0
12:25:57.916938 IP [OPNSENSE].1475 > [PARENTPROXY].80: Flags [.], ack 118231312, win 8703, options [nop,nop,TS val 3525849191 ecr 3328889438], length 0
^C
130693 packets captured
174070 packets received by filter
43373 packets dropped by kernel
[ipfire ~]#

Linux via Parentproxy
Code: [Select]
12:24:53.689743 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118179069:118180367, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689746 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118180367:118181665, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689785 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118181665:118182963, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689790 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118182963:118184261, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689794 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118184261:118185559, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689797 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118185559:118186857, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689835 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118186857:118188155, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689842 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118188155:118189453, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689846 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118189453:118190751, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689849 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118190751:118192049, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689852 IP [PARENTPROXY].80 > [LINUX].60130: Flags [P.], seq 118192049:118192684, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 635
12:24:53.689881 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118192684:118193982, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689885 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118193982:118195280, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689888 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118195280:118196578, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689891 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118196578:118197876, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689928 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118197876:118199174, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689935 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118199174:118200472, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689986 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118200472:118201770, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689990 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118201770:118203068, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689993 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118203068:118204366, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.689996 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118204366:118205664, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690000 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118205664:118206962, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690027 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118206962:118208260, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690030 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118208260:118209558, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690033 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118209558:118210856, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690080 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118210856:118212154, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690089 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118212154:118213452, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690095 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118213452:118214750, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690101 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118214750:118216048, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690106 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118216048:118217346, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690127 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118217346:118218644, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690131 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118218644:118219942, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690134 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118219942:118221240, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690143 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118221240:118222538, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169164], length 1298
12:24:53.690563 IP [LINUX].60130 > [PARENTPROXY].80: Flags [.], ack 118222538, win 24112, options [nop,nop,TS val 1106169179 ecr 1706858100], length 0
12:24:53.694721 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118222538:118223836, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169171], length 1298
12:24:53.694819 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118223836:118225134, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169171], length 1298
12:24:53.694899 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118225134:118226432, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169171], length 1298
12:24:53.694945 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118226432:118227730, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169171], length 1298
12:24:53.694952 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118227730:118229028, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169171], length 1298
12:24:53.694957 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], seq 118229028:118230326, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169171], length 1298
12:24:53.694962 IP [PARENTPROXY].80 > [LINUX].60130: Flags [P.], seq 118230326:118231311, ack 892, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169171], length 985
12:24:53.695086 IP [LINUX].60130 > [PARENTPROXY].80: Flags [.], ack 118226432, win 24552, options [nop,nop,TS val 1106169183 ecr 1706858100], length 0
12:24:53.695094 IP [LINUX].60130 > [PARENTPROXY].80: Flags [.], ack 118231311, win 24528, options [nop,nop,TS val 1106169183 ecr 1706858100], length 0
12:24:53.762258 IP [LINUX].60130 > [PARENTPROXY].80: Flags [P.], seq 892:916, ack 118231311, win 24568, options [nop,nop,TS val 1106169250 ecr 1706858100], length 24
12:24:53.763031 IP [LINUX].60130 > [PARENTPROXY].80: Flags [F.], seq 916, ack 118231311, win 24568, options [nop,nop,TS val 1106169251 ecr 1706858100], length 0
12:24:53.773864 IP [PARENTPROXY].80 > [LINUX].60130: Flags [.], ack 917, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169250], length 0
12:24:53.774575 IP [PARENTPROXY].80 > [LINUX].60130: Flags [F.], seq 118231311, ack 917, win 4096, options [nop,nop,TS val 1706858100 ecr 1106169250], length 0
12:24:53.774661 IP [LINUX].60130 > [PARENTPROXY].80: Flags [.], ack 118231312, win 24568, options [nop,nop,TS val 1106169263 ecr 1706858100], length 0
^C
45195 packets captured
114205 packets received by filter
68993 packets dropped by kernel
[ipfire ~]#

OPNsense via the IPFire squid
Code: [Select]
12:31:23.852641 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112353851, win 885, options [nop,nop,TS val 346827788 ecr 636760515], length 0
12:31:23.852644 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112355299, win 885, options [nop,nop,TS val 346827788 ecr 636760515], length 0
12:31:23.852646 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112356747, win 874, options [nop,nop,TS val 346827788 ecr 636760515], length 0
12:31:23.852648 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112358195, win 863, options [nop,nop,TS val 346827789 ecr 636760515], length 0
12:31:23.852651 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112359193, win 855, options [nop,nop,TS val 346827789 ecr 636760515], length 0
12:31:23.865488 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112359193:112364385, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827789], length 5192
12:31:23.865556 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112364385:112371077, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827789], length 6692
12:31:23.865627 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112360641, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865632 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112362089, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865637 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112371077:112380163, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827789], length 9086
12:31:23.865659 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112363537, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865670 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112364385, win 890, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865674 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112365833, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865674 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112380163:112382759, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827802], length 2596
12:31:23.865718 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112367281, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865728 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112382759:112389249, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827802], length 6490
12:31:23.865731 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112368729, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865735 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112370177, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865741 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112389249:112393143, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827802], length 3894
12:31:23.865757 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112371077, win 890, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865819 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112372525, win 879, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865826 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112393143:112400931, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827802], length 7788
12:31:23.865829 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112373973, win 868, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865833 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112375421, win 856, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865838 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112376869, win 845, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865846 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112400931:112403527, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827802], length 2596
12:31:23.865860 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112378317, win 834, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865864 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112379765, win 852, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865870 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112380163, win 894, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865879 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112381611, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865890 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112382759, win 888, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865927 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112403527:112407421, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827802], length 3894
12:31:23.865944 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112384207, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865950 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112385655, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865963 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112407421:112415209, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827802], length 7788
12:31:23.865991 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112387103, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.865999 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112415209:112419103, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827802], length 3894
12:31:23.866001 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112388551, win 882, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.866005 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112389249, win 876, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.866010 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112390697, win 865, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.866014 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112392145, win 869, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.866020 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112419103:112421699, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827802], length 2596
12:31:23.866231 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112393143, win 889, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.866238 IP [IPFIRE].800 > [OPNSENSE].53641: Flags [P.], seq 112421699:112426891, ack 892, win 252, options [nop,nop,TS val 636760519 ecr 346827802], length 5192
12:31:23.866240 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112394591, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.866245 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112396039, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.866250 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112397487, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.866254 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112398935, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
12:31:23.866258 IP [OPNSENSE].53641 > [IPFIRE].800: Flags [.], ack 112400383, win 885, options [nop,nop,TS val 346827802 ecr 636760519], length 0
^C
28839 packets captured
111529 packets received by filter
82677 packets dropped by kernel
[ipfire ~]#

Linux via IPFire squid
Code: [Select]
12:32:59.599869 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112177893, win 13299, options [nop,nop,TS val 2765744465 ecr 636789239], length 0
12:32:59.599875 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112180391, win 13315, options [nop,nop,TS val 2765744465 ecr 636789239], length 0
12:32:59.599878 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112183287, win 13315, options [nop,nop,TS val 2765744465 ecr 636789239], length 0
12:32:59.599881 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112186183, win 13315, options [nop,nop,TS val 2765744465 ecr 636789239], length 0
12:32:59.599884 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112189079, win 13315, options [nop,nop,TS val 2765744465 ecr 636789239], length 0
12:32:59.599888 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112210447:112213545, ack 892, win 252, options [nop,nop,TS val 636789239 ecr 2765744465], length 3098
12:32:59.600110 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112191975, win 13299, options [nop,nop,TS val 2765744465 ecr 636789239], length 0
12:32:59.600117 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112213545:112226525, ack 892, win 252, options [nop,nop,TS val 636789240 ecr 2765744465], length 12980
12:32:59.600120 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112194871, win 13299, options [nop,nop,TS val 2765744465 ecr 636789239], length 0
12:32:59.600122 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112197767, win 13315, options [nop,nop,TS val 2765744466 ecr 636789239], length 0
12:32:59.600124 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112200663, win 13315, options [nop,nop,TS val 2765744466 ecr 636789239], length 0
12:32:59.600126 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112203559, win 13315, options [nop,nop,TS val 2765744466 ecr 636789239], length 0
12:32:59.600128 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112226525:112231717, ack 892, win 252, options [nop,nop,TS val 636789240 ecr 2765744465], length 5192
12:32:59.600135 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112206455, win 13299, options [nop,nop,TS val 2765744466 ecr 636789239], length 0
12:32:59.600355 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112210447, win 13299, options [nop,nop,TS val 2765744466 ecr 636789239], length 0
12:32:59.600365 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112231717:112242655, ack 892, win 252, options [nop,nop,TS val 636789240 ecr 2765744466], length 10938
12:32:59.600368 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112213545, win 13315, options [nop,nop,TS val 2765744466 ecr 636789239], length 0
12:32:59.600372 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112216441, win 13315, options [nop,nop,TS val 2765744466 ecr 636789240], length 0
12:32:59.600378 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112219337, win 13315, options [nop,nop,TS val 2765744466 ecr 636789240], length 0
12:32:59.600381 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112222233, win 13315, options [nop,nop,TS val 2765744466 ecr 636789240], length 0
12:32:59.600383 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112225129, win 13299, options [nop,nop,TS val 2765744466 ecr 636789240], length 0
12:32:59.600606 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112227973, win 13299, options [nop,nop,TS val 2765744466 ecr 636789240], length 0
12:32:59.600613 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112242655:112245551, ack 892, win 252, options [nop,nop,TS val 636789240 ecr 2765744466], length 2896
12:32:59.600615 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112231717, win 13315, options [nop,nop,TS val 2765744466 ecr 636789240], length 0
12:32:59.600618 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112234613, win 13315, options [nop,nop,TS val 2765744466 ecr 636789240], length 0
12:32:59.600621 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112237509, win 13315, options [nop,nop,TS val 2765744466 ecr 636789240], length 0
12:32:59.600624 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112240405, win 13299, options [nop,nop,TS val 2765744466 ecr 636789240], length 0
12:32:59.600844 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112242655, win 13299, options [nop,nop,TS val 2765744466 ecr 636789240], length 0
12:32:59.600848 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112245551:112250743, ack 892, win 252, options [nop,nop,TS val 636789240 ecr 2765744466], length 5192
12:32:59.600858 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112245551, win 13315, options [nop,nop,TS val 2765744466 ecr 636789240], length 0
12:32:59.600949 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112250743:112257135, ack 892, win 252, options [nop,nop,TS val 636789240 ecr 2765744466], length 6392
12:32:59.601089 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112249895, win 13315, options [nop,nop,TS val 2765744467 ecr 636789240], length 0
12:32:59.601093 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112257135:112263625, ack 892, win 252, options [nop,nop,TS val 636789240 ecr 2765744467], length 6490
12:32:59.601095 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112250743, win 13315, options [nop,nop,TS val 2765744467 ecr 636789240], length 0
12:32:59.601097 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112255087, win 13315, options [nop,nop,TS val 2765744467 ecr 636789240], length 0
12:32:59.601102 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112263625:112267519, ack 892, win 252, options [nop,nop,TS val 636789240 ecr 2765744467], length 3894
12:32:59.601407 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112257135, win 13299, options [nop,nop,TS val 2765744467 ecr 636789240], length 0
12:32:59.601415 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112260031, win 13315, options [nop,nop,TS val 2765744467 ecr 636789240], length 0
12:32:59.601418 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112263625, win 13315, options [nop,nop,TS val 2765744467 ecr 636789240], length 0
12:32:59.601421 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112266521, win 13315, options [nop,nop,TS val 2765744467 ecr 636789240], length 0
12:32:59.601424 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112267519, win 13315, options [nop,nop,TS val 2765744467 ecr 636789240], length 0
12:32:59.601582 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112267519:112273911, ack 892, win 252, options [nop,nop,TS val 636789240 ecr 2765744467], length 6392
12:32:59.601826 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112273911:112285593, ack 892, win 252, options [nop,nop,TS val 636789240 ecr 2765744467], length 11682
12:32:59.602071 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112270415, win 13315, options [nop,nop,TS val 2765744468 ecr 636789240], length 0
12:32:59.602077 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112285593:112290423, ack 892, win 252, options [nop,nop,TS val 636789240 ecr 2765744468], length 4830
12:32:59.602315 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112273911, win 13299, options [nop,nop,TS val 2765744468 ecr 636789240], length 0
12:32:59.602549 IP [IPFIRE].800 > [LINUX].36088: Flags [P.], seq 112290423:112295615, ack 892, win 252, options [nop,nop,TS val 636789240 ecr 2765744468], length 5192
12:32:59.602561 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112276807, win 13315, options [nop,nop,TS val 2765744468 ecr 636789240], length 0
12:32:59.602565 IP [LINUX].36088 > [IPFIRE].800: Flags [.], ack 112279703, win 13315, options [nop,nop,TS val 2765744468 ecr 636789240], length 0
^C
30966 packets captured
66464 packets received by filter
35489 packets dropped by kernel
[ipfire ~]#

4
Hardware and Performance / Re: Massive performance problems floating rules vs interface rules
« on: September 21, 2021, 12:25:30 pm »
Sorry for not giving enough information. I am quite new to OPNsense and do not know what information is needed, as I had so many problems, I did not know where to start.
Neverthelesse, I found out the problem myself.

For now I use a WAN gateway in my LAN until I can fully switch to OPNsense.
I found out how to look at the raw FW rules and found the difference for the LAN interface in the rules with "reply-to"
So disabling that behaviour seems to make this problem go away.

Firewall -> Settings -> Advanced -> Disable reply-to -> (x) Disable Reply-to on WAN rules

5
Hardware and Performance / Re: Massive performance problems floating rules vs interface rules
« on: September 20, 2021, 05:31:09 pm »
Quote from: mimugmail on September 20, 2021, 05:16:14 pm
It's something in the virtualisation. Did you enable hardware offloading in OPNsense?
No and no.
It is the same problem on hardware. And no I did not enable hardware offloading.

6
Hardware and Performance / Re: Massive performance problems floating rules vs interface rules
« on: September 20, 2021, 05:08:41 pm »
@openfwb:
The problem is i have massive problems all over the place.
First I tried a cluster with two OPNsense instances on the ESX. But the moment I get little bit of load on the OPNsense (about 50 Mbit ) I got packet loss of about 30%.
So I thought maybe it is the virtualization, and so I put the Master on the Hardware I mention in my post.
Still the same problems.
During testing I found the problem with the floating vs interface rules, and thought maybe that is my problem.
As it is reproducible very easy I thought that might be solvable.
I do not want to buy a hardware appliance for our company if I can not be sure that those problems do not occur there.
Another problem: When trying to download through a web proxy behind an IPFire firewall I get 20Mbit from a Linux Server, but only 2Mbit from the OPNsense. When I use the IPFire as child web proxy I get the 20Mbit from the OPNsense as well as the Linux server.
So there are problems.
And you can not tell me, that there is no problem with the firewall, when the performance degrades 50 times by just moving a firewall rule from the floating rules to the interface rules.

7
Hardware and Performance / Re: Massive performance problems floating rules vs interface rules
« on: September 20, 2021, 01:17:52 pm »
But as you can see in my post iPerf works fine if using the floating rule or disabling the firewall.
So iperf is not the problem but OPNsense is.

8
Hardware and Performance / Re: Massive performance problems floating rules vs interface rules
« on: September 20, 2021, 12:55:58 pm »
So I should not use OPNsense? Well that is a bold answer in an OPNsense forum.

9
Hardware and Performance / [solved] Massive performance problems floating rules vs interface rules
« on: September 20, 2021, 12:16:14 pm »
Hi there I have many performance issues with OPNsense. Here is one of them:
I have GBit Ethernet connected with about 20 VLANs.
I can iPerf between two VLANs at about 930 MBits/sec
When running the iPerf3 server on the OPNsense I get the following behaviour:

Firewall disabled: about 930 MBits/sec

Firewall enabled, Port 5201 opened in floating rules (tried different rule positions, top, middle, bottom): about 930 MBits/sec

Firewall enabled, Port 5201 opened in interrface rules (first rule): about 25 MBits/sec

I got more, but maybe this one is an easier one :D

System information:
OPNsense 21.7.2
OPNsense Cluster:
Master: 24 core Xeon @ 1,9GHz 192GB RAM
Backup: VM on ESX, 8 cores @2,2 GHz 16GB RAM

Pages: [1]
OPNsense is an OSS project © Deciso B.V. 2015 - 2024 All rights reserved
  • SMF 2.0.19 | SMF © 2021, Simple Machines
    Privacy Policy     | XHTML | RSS | WAP2