apt-cacher-ng not working lately / 503 Errors

Posted by frashmanf@reddit | linuxadmin | View on Reddit | 19 comments

We have been using apt-cacher-ng for a couple of years now and have been having constant problems with it lately. The apt-cache host is an LXC container on our proxmox cluster and ran for years without problems. The errors are not persistent. I recently updated all of our clients at the same time via Ansible - some worked without error, some didn't, and the next time other PCs are affected. I can't see any pattern. Most of our clients are Debian 12, some are still on Debian 11. The error messages are mostly 503 errors like:

$ sudo apt update
Ign:1 http://security.debian.org/debian-security bookworm-security InRelease
Hit:2 http://ftp.debian.org/debian bookworm InRelease
Hit:3 http://ftp.debian.org/debian bookworm-updates InRelease
Hit:4 http://ftp.debian.org/debian bookworm-backports InRelease
Hit:5 http://ftp.debian.org/debian unstable InRelease
Ign:1 http://security.debian.org/debian-security bookworm-security InRelease
Ign:1 http://security.debian.org/debian-security bookworm-security InRelease
Err:1 http://security.debian.org/debian-security bookworm-security InRelease
  503  DNS error - nodename nor servname provided, or not known [IP: 10.0.1.147 3142]
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://security.debian.org/debian-security/dists/bookworm-security/InRelease  503  DNS error - nodename nor servname provided, or not known [IP: 10.0.1.147 3142]
W: Some index files failed to download. They have been ignored, or old ones used instead.

mostly security.debian.org is affected, but the other sources as well. We've often also seeing:

503 Resource temporarily unavailable [IP: 10.0.1.147 3142]
503 error: something with wrong permissions

this is the sources.list for all our clients:

$ cat /etc/apt/sources.list
deb http://ftp.debian.org/debian bookworm main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware
deb http://ftp.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
deb http://ftp.debian.org/debian bookworm-backports main contrib non-free non-free-firmware

this is the proxy config of the clients:

$ cat /etc/apt/apt.conf.d/00aptproxy 
# BEGIN ANSIBLE MANAGED BLOCK
Acquire::http::Proxy "http://10.0.1.147:3142";
Acquire::http::timeout "120";
Acquire::https::timeout "120";
# END ANSIBLE MANAGED BLOCK

things we tried/testet:

restart apt-cacher-ng
reset permissions (https://www.unix-ag.uni-kl.de/~bloch/acng/html/troublefaq.html#prob)
activate/deactivate HTTPS passthrough
delete/clear cache
ping/dig DNS (from different clients and apt-cache host)
reinstall
upgrade to Debian 12
new/fresh install of Debian 11 and 12
set SRV DNS record: "_http._tcp.deb.debian.org. 0 IN SRV 1 10 80 prod.debian.map.fastly.net."

Sun Jun 2 09:50:22 2024|Failed to move stale item debian/dists/bookworm/InRelease out of the way: No such file or directory Sun Jun 2 09:50:22 2024|Failed to move stale item debian/dists/bookworm-updates/InRelease out of the way: No such file or directory Sun Jun 2 09:50:22 2024|Failed to move stale item secure/dists/bookworm-security/InRelease out of the way: No such file or directory

# Limit requests for a particular URI to avoid identical requests hitting # apt-cacher-ng too quickly which tends to cause errors. limit_req_zone $request_uri zone=acng:10m rate=4r/s; # Define proxy cache proxy_cache_path /var/cache/nginx keys_zone=acng_cache:10m max_size=100m; server { listen 80; server_name apt.example.com; # Replace this with your appropriate domain name # Cache requests for .*InRelease to reduce calls to apt-cacher-ng location ~* ^.*InRelease$ { limit_req zone=acng burst=100; limit_req_log_level info; include proxy_params; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://127.0.0.1:3142; proxy_buffering on; proxy_pass_request_headers on; proxy_cache acng_cache; proxy_cache_key $scheme$proxy_host$request_uri; proxy_cache_valid any 1m; proxy_cache_lock on; } # Default location block location / { limit_req zone=acng burst=100; limit_req_log_level info; include proxy_params; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://127.0.0.1:3142; proxy_buffering off; proxy_pass_request_headers on; } }

aptly publish switch \ -batch \ -component=main,security,updates \ ${DISTRO_NAME} \ snapshot-jammy-main-20240517 \ snapshot-jammy-updates-20240517 \ snapshot-jammy-security-20240517 \ upstream

aptly publish switch \ -component="main,universe,multiverse" \ -batch="true" \ -gpg-provider="gpg" \ -passphrase="${PASSPHRASE}" \ ${DIST}-${POCKET} \ ubuntu \ snap-${PERIOD}-${DATE}-${DIST}-ubuntu-${POCKET}-main \ snap-${PERIOD}-${DATE}-${DIST}-ubuntu-${POCKET}-universe \ snap-${PERIOD}-${DATE}-${DIST}-ubuntu-${POCKET}-multiverse

CacheDir: /var/cache/apt-cacher-ng LogDir: /var/log/apt-cacher-ng SupportDir: /usr/lib/apt-cacher-ng Port:9696 BindAddress: 0.0.0.0 Remap-debrep: file:deb_mirror*.gz /debian ; file:backends_debian # Debian Archives Remap-uburep: file:ubuntu_mirrors /ubuntu ; file:backends_ubuntu # Ubuntu Archives Remap-klxrep: file:kali_mirrors /kali ; file:backends_kali # Kali Linux Archives Remap-cygwin: file:cygwin_mirrors /cygwin # ; file:backends_cygwin # incomplete, please create this file or specify preferred mirrors here Remap-sfnet: file:sfnet_mirrors # ; file:backends_sfnet # incomplete, please create this file or specify preferred mirrors here Remap-alxrep: file:archlx_mirrors /archlinux # ; file:backend_archlx # Arch Linux Remap-fedora: file:fedora_mirrors # Fedora Linux Remap-epel: file:epel_mirrors # Fedora EPEL Remap-slrep: file:sl_mirrors # Scientific Linux Remap-gentoo: file:gentoo_mirrors.gz /gentoo ; file:backends_gentoo # Gentoo Archives Remap-secdeb: security.debian.org security.debian.org/debian-security deb.debian.org/debian-security /debian-security cdn-fastly.deb.debian.org/debian-security ; deb.debian.org/debian-security security.debian.org cdn-fastly.deb.debian.org/debian-security ReportPage: index.html UnbufferLogs: 1 VerboseLog: 1 ForeGround: 0 ExSuppressAdminNotification: 1 MaxConThreads: -1 DisconnectTimeout: 30 DlMaxRetries: 10 Debug: 99 ExposeOrigin: 0 LogSubmittedOrigin: 1 NetworkTimeout: 30 LocalDirs: acng-doc /usr/share/doc/apt-cacher-ng ConnectProto: v4 KeepExtraVersions: 1 FreshIndexMaxAge: 7 RedirMax: 5 ReuseConnections: 1 PipelineDepth: 10

Failure to move file /var/cache/apt-cacher-ng/secdeb/dists/bookworm-security/InRelease out of the way or cannot create /var/cache/apt-cacher-ng/secdeb/dists/bookworm-security/InRelease.1698175991 - errno: File exists Error creating file item for secdeb/dists/bookworm-security/InRelease -- check file permissions!

[-]

cdubmkvi@reddit

I too have struggled with sporadic 50X errors in apt-cacher-ng in Debian 11 (v3.6.4-1) which worsened after moving to Debian 12 (v3.7.4-1). After trying many configuration changes (including the ones mentioned in this thread) I realized the issues that I was seeing were due to a many concurrent requests being made to the apt-cacher-ng service in a short amount of time. If I tell all 30 of my servers to apt update at the same time I am able to easily cause the issue. If I only tell 3 servers to look for updates at once, no errors are seen.

Digging into this further I found many of these of errors in /var/log/apt-cacher-ng/apt-cacher.err, always relating to InRelease files.

Building on the theory that too many concurrent requests were causing the issue, I added a NGINX reverse proxy server on standard TCP port 80 in front of apt-cacher-ng and ended up with the configuration below which has been rock solid for 3 weeks at this point. This rate limits all incoming requests on a per URI basis to 4 requests per second and also introduces a 1 minute cache on InRelease files. This essentially slows down identical requests to a point where apt-cacher-ng can handle them without falling over.

ClevelandV@reddit

Thanks! Looks great so far.

LW81@reddit

Hey, since we are facing the same issue i tried to implement your nginx solution.

unfortunately nginx won’t start for me with the config you pasted. The error is

„limit_req_zone“ directive is not allowed here in /etc/nginx/nginx.conf:3

Any idea what’s happening here?

Thx

Based on the line number indicated, it looks like you pasted the config at the top of the /etc/nginx/nginx.conf file. The "limit_req_zone" directive needs to be inside the http block of the NGINX config. (https://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone)

I have the config that I posted above in a separate file (/etc/nginx/sites-enabled/apt) which is included within the http block by the default nginx.conf file provided by Debian.

somecallmechief@reddit

I started getting the 503 error from my AptCacherNg instance during the upgrade from debian 12 to debian 13. I have half a dozen Proxmox hosts with a ton of Qemu/LXC instances that all needed to upgrade. The first few instances went smoothly (perhaps because none of the content was cached?), but then all apt update calls started failing with a Failed to fetch http://deb.debian.org/debian/dists/trixie/InRelease 503 Cannot allocate memory [IP: x.x.x.x 3142] error.

I've spent some time head banging trying to fix this, and I finally got to rtfm https://wiki.debian.org/AptCacherNg#Maintenance_tasks, and I discovered for the first time the GUI (I had no idea AptCacherNg had a GUI). So I went to my http://<your-local-ip-addres>:3142/acng-report.html?abortOnErrors=aOe&byPath=bP&byChecksum=bS&truncNow=tN&incomAsDamaged=iad&purgeNow=pN&doExpire=Start+Scan+and%2For+Expiration&calcSize=cs&asNeeded=an#bottom page and nuked all the assets showing 503 errors.

Now I'm getting different errors like The repository 'http://deb.debian.org/debian trixie-backports Release' no longer has a Release file., but it feels like progress at least. Hopefully there is a simple step to force the cache to download new assets and then I will be back in business.

js290@reddit

have you tried switching to https in your sources.list (or sources.list.d/ubuntu.sources)?

Even-Inspector9931@reddit

wow, how nice, every apt cache/mirror tool is unmaintained or de facto dead now. debmirror, apt-mirror, sapt-cacher-ng, quid-deb-proxy.........

reedacus25@reddit

I don't have a ton to offer in the way of troubleshooting apt-cacher-ng.

Only to say that I've not had any problems with aptly for deb based distros.

We use it for the purpose of local caching, but also for creating snapshots of the repos for the purpose of having consistent packages available, and having different channels for prod/staging/etc.

Might be overkill for your use case, but I've not had any issues with it.

Currently working through tricking Ubuntu's do-release-upgrade to pull from the aptly published repo(s), rather than upstream, which is less than trivial.

Glad_Hurry_7492@reddit

Hey there,

Regarding Aptly, do you have any advice for creating an automated snapshot configuration? I am trying to get this working at my organization but cannot for the life of me get it to work correctly. I guess I have the syntax wrong but I cannot figure it out.

After creating all the mirrors and publishing them, the idea is to update them regularly and publish, and switch out yesterdays snapshot with the new one.

aptly publish switch -batch -component=main,security,updates ${DISTRO_NAME} snapshot-jammy-sensu-20240517 snapshot-jammy-salt-20240517 upstream

With upstream being the final, published repo.

Any advice would be greatly appreciated, I've been banging my head against this for weeks.

I'll transform your one-liner to multi-liner just for ease of viewing.

Comparing it to my script below, I think the main thing is that your published repo name upstream should be before the snapshots, instead of after. ubuntu is my publish endpoint (more on this below), in my example.

Usage: aptly publish switch <distribution> [[<endpoint>:]<prefix>] <new-snapshot>

Also, I think that you may run into some issues with the differing named distributions with ubuntu's pocket naming, where the distribution is jammy-{updates,backports,security} from upstream. But maybe when you created your publish you rewrote that with -distribution="": distribution name to publish to collapse it all into one.

One thing I ran into issues with previously, was that when trying to use my aptly repos with do-release-upgrade, for the canonical ubuntu repos, it expects the ubuntu endpoint, and anything other than that will throw errors. I haven't tried that with a single collapsed distribution, would be curious if that works or not.

Anyways, hope this helps some. Aptly has a ton of rope to get hung up on.

Ahhhhh thank you so much, I got it working. I realllllly appreciate it!!

What is the {$POCKET} variable signify?

With Ubuntu the pocket is the different “channels” like release, updates, backports, security.

The release pocket doesn’t really change after the release occurs, so you don’t really need to update it. And while it is referred to as “release” it is effectively the absence of a pocket.

frashmanf@reddit (OP)

Thank you, I will look into it. But I think that shoots beyond our goal.

maybe I will try the older approx tool.

Cheeseblock27494356@reddit

Apt-cacher-ng is broken software with known bugs that never will be fixed.

https://www.reddit.com/r/debian/comments/klebea/psa_aptcacherng_is_a_buggy_pile_of_shit/

bctrainers@reddit

This issue has plagued me for many months since upgrading to the newest version of Debian (12), which probably upgraded apt-cacher-ng to something newer, and things became broken shortly thereafter. There was no rhyme or reason behind the errors, which happened randomly. Some VMs would get various 50x errors, while others were perfectly fine with the same repo being polled. I did nearly the same things that /u/frashmanf posted. After many days of head banging, I ended at this configuration that no longer outputs errors to my homelab Virtual Machines.

The debug and verbosity in the acng.conf code below can more or less be turned off or turned down.

With the permissions issue, I did notice this popping up in the logs a short while after getting the connection hangups and random 50x errors out of the way... (trimmed back some fluff on this log bit)

A quick chown -R apt-cacher-ng:apt-cacher-ng /var/cache/apt-cacher-ng/ solved that issue.

Apart from that, things feel like it's smooth sailing... so far.

this looks very promising. i will try it right away. thank you very much!

HelpImOutside@reddit

We are having the same issue with apt-cacher-ng.

We have also looked into aptly but also thought it seemed a bit overkill, but I am not sure if there is a better alternative.

You have the same problems? Can you remember when it started? Due to the inconsistency of the problem, it is difficult to pinpoint a date.

We only started using it to replace apt-mirror about six months ago and have had issues pretty much from the beginning