[SOLVED] Squid cache seems to not be used

[bluepr0]

Hello!

I have the latest OPNSense version to date (OPNsense 15.7.22-amd64) and wanted to enable Proxy Server to cache web stuff and make it quicker. However it seems to not be working:

This is the Cache tab log

Code: [Select]

2015/12/16 15:50:04| Error sending to ICMPv6 packet to [2a00:1450:400c:c02::79]. ERR: (65) No route to host
2015/12/16 15:49:59 kid1| ipcacheParse: No Address records in response to 'e.monetate.net'
2015/12/16 15:49:58 kid1| ipcacheParse: No Address records in response to 'nexus.ensighten.com'
2015/12/16 15:48:13| Error sending to ICMPv6 packet to [2001:41c8:1000:21::21:35]. ERR: (65) No route to host
2015/12/16 15:48:13| Error sending to ICMPv6 packet to [2001:a78:5:1:216:35ff:fe7f:6ceb]. ERR: (65) No route to host
2015/12/16 15:47:53| Error sending to ICMPv6 packet to [2a00:1450:4004:800::200e]. ERR: (65) No route to host
This is the Access log. I have never seen a TCP HIT. I wish it could be a way to see "live" logs or maybe a website to have a better overview of the data (apart from the tail -f /var/squid/logs command)

Code: [Select]

1450277683.236 265 10.0.1.59 TCP_MISS/200 377 POST http://bridge.meethue.com/queue/getmessage? - ORIGINAL_DST/64.233.166.121 text/plain
1450277657.965 998 10.0.1.59 TCP_MISS_ABORTED/000 0 POST http://bridge.meethue.com/queue/getmessage? - ORIGINAL_DST/64.233.166.121 -
1450277631.960 246 10.0.1.59 TCP_MISS/200 376 POST http://bridge.meethue.com/queue/getmessage? - ORIGINAL_DST/64.233.166.121 text/plain
1450277606.705 233 10.0.1.59 TCP_MISS/200 376 POST http://bridge.meethue.com/queue/getmessage? - ORIGINAL_DST/64.233.166.121 text/plain
1450277581.466 241 10.0.1.59 TCP_MISS/200 375 POST http://bridge.meethue.com/queue/getmessage? - ORIGINAL_DST/64.233.166.121 text/plain
1450277576.692 947 10.0.1.14 TCP_MISS/200 204884 GET http://is2.mzstatic.com/image/thumb/Music/v4/8c/37/00/8c3700ab-3874-be8c-3cef-334a05486161/source/800x800bb.jpg - ORIGINAL_DST/77.67.29.203 image/jpeg
1450277556.214 245 10.0.1.59 TCP_MISS/200 376 POST http://bridge.meethue.com/queue/getmessage? - ORIGINAL_DST/64.233.166.121 text/plain
1450277536.453 1383 10.0.1.14 TCP_MISS/200 372830 GET http://a4.mzstatic.com/us/r30/Music/v4/8c/37/00/8c3700ab-3874-be8c-3cef-334a05486161/cover1400x1400.jpeg - ORIGINAL_DST/77.67.29.194 image/jpeg
1450277534.908 15987 10.0.1.14 TCP_MISS/200 7241694 GET http://aod.itunes.apple.com/apple-assets-us-std-000001/Music/v4/be/61/df/be61dfb6-375b-0b87-9c64-c70198df7f96/mzaf_7113937544685891047.m4a? - ORIGINAL_DST/17.253.39.207 audio/x-m4a
1450277530.964 242 10.0.1.59 TCP_MISS/200 376 POST http://bridge.meethue.com/queue/getmessage? - ORIGINAL_DST/64.233.166.121 text/plain
1450277518.424 161 10.0.1.59 TCP_MISS/200 941 POST http://dcp.cpp.philips.com/DcpRequestHandler/index.ashx - ORIGINAL_DST/5.79.62.93 application/CB-Encrypted
This is the Store log

Code: [Select]

File /var/log/squid/store.log doesn't exist.

I'm running OPNSense on a VM (VMWare ESXi 6). I haven't make any special configurations, only enabled Proxy server). I would also love to be able to cache SSL connections without the "middle in the man" technique

Thanks!

[Tikimotel]

Squid should work, however in the default mode it only allows for mem_cache, not disk cache.
That needs to be enabled via the "Enable local cache (requires service restart)" option in the General proxy settings -> Local cache settings pull/drop-down menu options. In advanced mode you can set the disk cache size.

However "https" sites do not cache without some MITM certificate configuration on both the squid and the clients. So it is getting less and less effective to do a squid cache with more and more sites becoming https.

I've tinkered with the template file to use more ram than the default of 256MB.
It would be great if you could set a value for the "cache_mem" setting in the GUI.

In (/usr/local/opnsense/service/templates/OPNsense/Proxy/squid.conf)
I added @ approx. line 271
after this line:

Code: [Select]

# Deny all other access to this proxy
http_access deny all
I added:

Code: [Select]

# Increase cache_mem to 8GB (I have 32GB available)
cache_mem 8192 MB

But I do see TCP_MEM_HIT, so it is caching.

Code: [Select]

1450281877.924 0 192.168.0.51 TCP_MEM_HIT/200 4375 GET http://m.bestofmedia.com/sfp/js/plugins/head.min.js? - HIER_NONE/- application/javascript
1450281877.924 0 192.168.0.51 TCP_MEM_HIT/200 1299 GET http://m.bestofmedia.com/sfp/css/socialPromote.css? - HIER_NONE/- text/css
1450281877.644 509 192.168.0.51 TCP_MISS/200 700 GET http://facebook.computing.net/cgi-bin/recent_json.pl? - HIER_DIRECT/69.167.142.128 application/json
1450281877.402 186 192.168.0.51 TCP_MISS/200 849 POST http://www.tomshardware.com/destilar-rtffvqutvwxzfb.js? - HIER_DIRECT/95.100.96.185 text/plain
1450281877.325 0 192.168.0.51 TAG_NONE/400 4357 NONE error:invalid-request - HIER_NONE/- text/html
1450281877.314 1 192.168.0.51 TCP_MEM_HIT/200 6627 GET http://img.tomshardware.com/F/K/262352/2/262352.gif - HIER_NONE/- image/gif
1450281877.313 1 192.168.0.51 TCP_MEM_HIT/200 7841 GET http://img.tomshardware.com/I/O/262464/2/262464.gif - HIER_NONE/- image/gif
1450281877.313 1 192.168.0.51 TCP_MEM_HIT/200 3170 GET http://img.tomshardware.com/G/V/331087/2/331087.jpg - HIER_NONE/- image/jpeg
1450281877.313 1 192.168.0.51 TCP_MEM_HIT/200 3334 GET http://img.tomshardware.com/U/C/358788/2/358788.jpg - HIER_NONE/- image/jpeg
1450281877.307 115 192.168.0.51 TCP_MEM_HIT/200 49455 GET http://img.tomshardware.com/1/J/359047/2/359047.png - HIER_NONE/- image/png
1450281877.307 115 192.168.0.51 TCP_MEM_HIT/200 38051 GET http://img.tomshardware.com/G/H/358289/2/358289.png - HIER_NONE/- image/png


[bluepr0]

Interesting! I will check it out. Thanks for your reply.

Do you know if there's other "new" method of caching that includes SSL? I would love to cache websites, but also things I downloads or files (for example when I download OS' updates – I'm a Mac and iOS user)

Will report back!

[smajor]

I was looking for roughly the same, there is a ticket in for some expanded Squid options via the GUI.  I hope the 16.1 milestone for them comes true! (I also wish I could code at all so I could help.)

https://github.com/opnsense/core/issues/417

[AdSchellevis]

Ticket 417 seems to be about caching options, which I guess are already in there (only size option for cache_mem  seems to be missing / defaulting to 256MB).

There is some additional info about peek and splice for squid at the bottom of https://github.com/opnsense/core/issues/460 . Not sure when this feature will enter OPNsense.

[Tikimotel]

I was looking for roughly the same, there is a ticket in for some expanded Squid options via the GUI.  I hope the 16.1 milestone for them comes true! (I also wish I could code at all so I could help.)

https://github.com/opnsense/core/issues/417

This seems to do the trick (I'm not good at diffs  )

The first step is to edit the template conf-file.
Add the "OPNsense.proxy.general.cache.memory" parts after line 270 like below.
"/usr/local/opnsense/service/templates/OPNsense/Proxy/squid.conf"

Code: [Select]

# Deny all other access to this proxy
http_access deny all

{% if helpers.exists('OPNsense.proxy.general.cache.memory')  %}
# Set cache_mem, (default is 256 MB)
cache_mem {{OPNsense.proxy.general.cache.memory.size}} MB
{% endif %}

{% if helpers.exists('OPNsense.proxy.general.cache.local')  %}
{%  if OPNsense.proxy.general.cache.local.enabled == '1' %}
# Uncomment and adjust the following to add a disk cache directory.
cache_dir ufs {{OPNsense.proxy.general.cache.local.directory}} {{OPNsense.proxy.general.cache.local.size}} {{OPNsense.proxy.general.cache.local.l1}} {{OPNsense.proxy.general.cache.local.l2}}
{%  endif %}
{% endif %}

Then edit the model XML to include the new memory section, just start it before the "<local>" section in the "<cache>".
@ around line 71
"/usr/local/opnsense/mvc/app/models/OPNsense/Proxy/Proxy.xml"

Code: [Select]

            <cache>
<memory>
                    <size type="IntegerField">
                        <default>256</default>
                        <MinimumValue>8</MinimumValue>
                        <ValidationMessage>Specify a positive memory cache size. (number of MB's)</ValidationMessage>
                        <Required>Y</Required>
                    </size>
</memory>
                <local>
                    <enabled type="BooleanField">
                        <default>0</default>
                        <Required>Y</Required>
                    </enabled>

Then create a new subtab field called "proxy-general-cache-memory" before the "proxy-general-cache-local" section and you should have a new pull down option in the proxy service menu this is done in the form XML.
(@ around line 93)
"/usr/local/opnsense/mvc/app/controllers/OPNsense/Proxy/forms/main.xml"

Code: [Select]

            <div class="text-info"><b>NOTE:</b> the current Squid implementation of encode and chop violates
                RFC2616 by not using a 301 redirect after altering the URL.</div>]]></help>
                <advanced>true</advanced>
            </field>
        </subtab>
<subtab id="proxy-general-cache-memory" description="Memory Cache Settings">
            <field>
                <id>proxy.general.cache.memory.size</id>
                <label>Memory Cache size in Megabytes</label>
                <type>text</type>
                <help><![CDATA[Enter the storage size for the memory cache (default is 256).]]></help>
                <advanced>true</advanced>
            </field>
        </subtab>
        <subtab id="proxy-general-cache-local" description="Local Cache Settings">


[franco]

The cache_mem setting addition will be part of 15.7.24 tomorrow.

[Deku2]

I don't see any hits in the logs and I don't want to enable local cache.  I want it all in memory - assigned 4Gig, but don't see that anything is hitting. 

Don't see any logs at all except for a repeated "Error sending to ICMPv6 packet" in Cache: https://forum.opnsense.org/index.php?topic=1254.0  Access is empty.