Caching system updates for the home lab
If you're like me, you've got a home lab with a dozen or so virtual machines doing all sorts of things - and each of them are pulling down updates from somewhere on the internet.
What if you could have a single endpoint for all VMs to reference? That way, updates that are common would be distributed to all systems at LAN speeds after the first download.
Introducing - mod_cache for Apache :)
Assuming you're already running Apache somewhere, you can start mapping part of the local path structure to a remote endpoint.
Drop the following into /etc/httpd/conf.d/mod_cache.conf
:
CacheEnable disk /fedora CacheRoot /var/cache/httpd/fedora CacheMaxFileSize 524288000 CacheDefaultExpire 14400 CacheDetailHeader on # common caching directives CacheQuickHandler off CacheLock on CacheLockPath /tmp/mod_cache-lock CacheLockMaxAge 5 CacheHeader On # cache control #CacheIgnoreNoLastMod On #CacheIgnoreCacheControl On # unset headers from upstream server Header unset Expires Header unset Cache-Control Header unset Pragma ProxyRequests Off ProxyPass /fedora http://dl.fedoraproject.org/pub/fedora ProxyPassReverse /fedora http://dl.fedoraproject.org/pub/fedora UseCanonicalName On
When in use, this will map http://my.apache.host/fedora
to the Fedora mirror, and cache all responses and downloaded files.
The cache won't automatically clean itself though - so we need a systemd service to clean things up over time. Create the file /etc/systemd/system/http-cache-clean.service
as follows:
[Unit] Description=Apache cache cleaner After=network-online.target [Service] Type=forking ExecStart=/usr/sbin/htcacheclean -d 60 -i -l 5G -p /var/cache/httpd/fedora/ [Install] WantedBy=multi-user.target
This will limit the cache size to 5Gb and remove the oldest files first.
There is one gotcha when using this with Fedoras updates and dnf
- zchunk. I believe this is because mod_cache
doesn't work
on partial content requests - which is how zchunk
functions.
To get around this, we can disable zchunk in the DNF configuration file /etc/dnf/dnf.conf
. I also disable deltarpm
- as its quicker to download the file from the LAN cache than it is
to rebuild a drpm update.
[main] gpgcheck=True installonly_limit=3 clean_requirements_on_remove=True best=False skip_if_unavailable=True max_parallel_downloads=10 fastestmirror=True zchunk=False deltarpm=0
We can then point the yum repo file to the local apache server - for example, part of /etc/yum.repos.d/fedora-updates.repo
:
[updates] name=Fedora $releasever - $basearch - Updates #baseurl=http://download.example/pub/fedora/linux/updates/$releasever/Everything/$basearch/ #metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch baseurl=http://my.apache.host/fedora/linux/updates/$releasever/Everything/$basearch/ enabled=1