Caching system updates for the home lab

Steven Haigh Comments

If you're like me, you've got a home lab with a dozen or so virtual machines doing all sorts of things - and each of them are pulling down updates from somewhere on the internet.

What if you could have a single endpoint for all VMs to reference? That way, updates that are common would be distributed to all systems at LAN speeds after the first download.

Introducing - mod_cache for Apache :)

Assuming you're already running Apache somewhere, you can start mapping part of the local path structure to a remote endpoint.

Drop the following into /etc/httpd/conf.d/mod_cache.conf:

CacheEnable                 disk /fedora
CacheRoot                   /var/cache/httpd/fedora
CacheMaxFileSize            524288000
CacheDefaultExpire          14400
CacheDetailHeader           on

# common caching directives
CacheQuickHandler           off
CacheLock                   on
CacheLockPath               /tmp/mod_cache-lock
CacheLockMaxAge             5
CacheHeader                 On

# cache control
#CacheIgnoreNoLastMod       On
#CacheIgnoreCacheControl    On

# unset headers from upstream server
Header unset Expires
Header unset Cache-Control
Header unset Pragma

ProxyRequests               Off
ProxyPass                   /fedora http://dl.fedoraproject.org/pub/fedora
ProxyPassReverse            /fedora http://dl.fedoraproject.org/pub/fedora

UseCanonicalName            On

When in use, this will map http://my.apache.host/fedora to the Fedora mirror, and cache all responses and downloaded files.

The cache won't automatically clean itself though - so we need a systemd service to clean things up over time. Create the file /etc/systemd/system/http-cache-clean.service as follows:

[Unit]
Description=Apache cache cleaner
After=network-online.target

[Service]
Type=forking
ExecStart=/usr/sbin/htcacheclean -d 60 -i -l 5G -p /var/cache/httpd/fedora/

[Install]
WantedBy=multi-user.target

This will limit the cache size to 5Gb and remove the oldest files first.

There is one gotcha when using this with Fedoras updates and dnf - zchunk. I believe this is because mod_cache doesn't work on partial content requests - which is how zchunk functions.

To get around this, we can disable zchunk in the DNF configuration file /etc/dnf/dnf.conf. I also disable deltarpm - as its quicker to download the file from the LAN cache than it is to rebuild a drpm update.

[main]
gpgcheck=True
installonly_limit=3
clean_requirements_on_remove=True
best=False
skip_if_unavailable=True
max_parallel_downloads=10
fastestmirror=True
zchunk=False
deltarpm=0

We can then point the yum repo file to the local apache server - for example, part of /etc/yum.repos.d/fedora-updates.repo:

[updates]
name=Fedora $releasever - $basearch - Updates
#baseurl=http://download.example/pub/fedora/linux/updates/$releasever/Everything/$basearch/
#metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch
baseurl=http://my.apache.host/fedora/linux/updates/$releasever/Everything/$basearch/
enabled=1

Comments

Comments powered by Disqus