Previous | Contents | Next

Chapter 7: HOWTOs and FAQ

7.1 Package import

Already existing packages can be imported into apt-cacher-ng's cache pool instead of downloading them. There are some restrictions:

  1. Don't try to import incomplete files. They will be refused since their contents cannot be checked against the archive metadata.
  2. If possible, don't import symbolic links. Even if doing so, they should not point to other files inside of the cache and especially not to other files under the _import directory.

HOWTO:

  1. Make sure that apt-cacher-ng has valid index files in the cache. This is the tricky part. To get them right, a client needs to download them through apt-cacher-ng once. Therefore:
    1. Configure the server and one client before doing the import. See above for instructions.
    2. Run "apt-get update" on client(s) once to teach ACNG about remote locations of (volatile) index files. In some cases this is not sufficient. See the note on APT below for a workaround.
  2. Store copies of your .debs, .orig.tar.gz, ... somewhere in the "_import" subdirectory in the cache, ie. in /var/cache/apt-cacher/_import/. The files may be links or symlinks, does not matter. When done, apt-cacher will move those files to its own internal locations. Example:
    cd /var/cache
    mkdir apt-cacher-ng/_import
    cp -laf apt-proxy apt-cacher /var/cache/apt-cacher-ng/_import
    chown -R apt-cacher-ng apt-cacher-ng/_import
    
  3. Visit the report page and trigger the import action there. Check the results, look for (red) error messages.
  4. Check the _import directory again. All files that could be identified as referenced by archive metadata should no longer be there if they have been successfully moved. If some files have been left behind, check whether the client can use them ("apt-cache show" and checking checksums with md5sum/sha1sum tools). If yes, then repeat the procedure and be more careful in step 1. If that doesn't work, enable verbose output in the web interface and watch out for error messages.

NOTE: APT is pretty efficient on avoiding unneccessary downloads which can make a proxy blind to some relevant files. ACNG makes some attempts to guess the remote locations of missed (not downloaded) files but these heuristics may fail, especially on non-Debian systems. When some files are permanently ignored, check the process output for messages about the update of Packages/Sources files. When some relevant package sources are missing there, there is a brute-force method to force their download to the client (for clients with Debian only). To do that, run:

rm /var/cache/apt/*cache.bin
rm /var/lib/apt/lists/*Packages 
rm /var/lib/apt/lists/*Sources

on the client to purge APT's internal cache, and then rerun "apt-get update" there.

7.2 Cache overview

To get a basic overview of the cache contents, the distkill.pl script may be used. See section 6.3 for details and warnings.

# /usr/lib/apt-cacher-ng/distkill.pl
Scanning /var/cache/apt-cacher-ng, please wait...
Found distributions:
	1. testing (6 index files)
	2. sid (63 index files)
	3. etch-unikl (30 index files)
	4. etch (30 index files)
	5. experimental (505 index files)
	6. lenny (57 index files)
	7. unstable (918 index files)
	8. stable (10 index files)

WARNING: The removal action would wipe out whole directories containing
         index files. Select d to see detailed list.

Which distribution to remove? (Number, 0 to exit, d for details): d

Directories to remove:
 1. testing: 
  /var/cache/apt-cacher-ng/debrep/dists/testing 
 2. sid: 
  /var/cache/apt-cacher-ng/localstuff/dists/sid 
  /var/cache/apt-cacher-ng/debrep/dists/sid 
 4. etch: 
  /var/cache/apt-cacher-ng/ftp.debian-unofficial.org/debian/dists/etch 
 5. experimental: 
  /var/cache/apt-cacher-ng/debrep/dists/experimental 
 6. lenny: 
  /var/cache/apt-cacher-ng/security.debian.org/dists/lenny 
  /var/cache/apt-cacher-ng/debrep/dists/lenny 
 7. unstable: 
  /var/cache/apt-cacher-ng/debrep/dists/unstable 
  /var/cache/apt-cacher-ng/localstuff/debian/dists/unstable 
 8. stable: 
  /var/cache/apt-cacher-ng/debrep/dists/stable 
Found distributions:

WARNING: The removal action would wipe out whole directories containing
         index files. Select d to see detailed list.

7.3 Access control and inetd usage

Filtering by client IP or hostname is not supported directly. However, an inetd daemon is shipped with the package which makes the use of tcpd possible. Installation is done in following steps:

1. compile the inetd bridge tool "in.acng", if not already done (check /usr/lib/apt-cacher-ng).

2. Edit apt-cacher-ng's configuration (acng.conf, for example), and set a path for a new file in a writable directory, like this:

SocketPath:/var/run/apt-cacher-ng/socket

3. Edit /etc/inetd.conf and add following line with appropriate path names and TCP port:

3143  stream  tcp nowait  user /usr/sbin/tcpd 
        /usr/local/sbin/in.acng /var/run/apt-cacher-ng/socket

4. Edit hosts.allow and other files to configure ACLs for port 3143. See tcpd(8) and related manpages for further details.

5. Configure clients to use the alternative port (3143 in the example above).

7.4 JIGDO usage

It's possible to use apt-cacher-ng source with the jigdo-lite utility. There are some limitations, though:

But it's possible to feed jigdo-lite with the package contents from your mirror. To do that, first start jigdo-lite as usual, something like:

jigdo-lite http://cdimage.debian.org/.../...-DVD-1.jigdo

When asked about Debian mirror, enter something like:

http://proxy.host:3142/ftp.de.debian.org/debian/

i.e. construct the same URL as present in usual apt-cacher-ng's user's sources.list.

That's all, jigdo-lite will fetch the package files using apt-cacher-ng proxy.

7.5 Debugging

Preliminary meanings of Debug option settings are:

Getting HTTP headers from apt-get works like this: apt-get update -o Debug::Acquire::Http=true

7.6 Avoid use of apt-cacher-ng for certain hosts

Sometimes clients might need to access some remote side directly to do some non-file-transfer oriented work but still passing the data through configured apt-cacher-ng proxy. Such remote hosts can be marked for direct access in apt configuration, e.g. in /etc/apt/apt.conf:

Acquire::HTTP::Proxy::archive.example.org "DIRECT";
//or Acquire::HTTP::Proxy::archive.example.org  "other.proxy:port"

7.7 Avoid caching for certain domains or certain file types

Sometimes clients to download through apt-cacher-ng but the data shall not be stored on the harddisk of the server. To get it, use the DontCache directive (see examples for details) to define such files.

7.8 How to make big download series faster

Symptom: A common situation is a periodic download of hundreds of files through apt-cacher-ng where just a half is present in the cache. Although caching works fine, there are visible delays on some files during the download.

Possible cause and relief: the download from the real mirror gets interrupted while apt-cacher-ng delivers a set of files from the internal cache. While the connection is suspended, it times out and needs to be recreated when a miss occurs, i.e. apt-cacher-ng has to fetch more from the remote mirror. A workaround to this behaviour is simple, provided that the remote mirror can handle long request queues: set the pipelining depth to a very high value in apt.conf file or one of its replacement files in /etc/apt/apt.conf.d/. With something like:

Acquire::http { Pipeline-Depth "200"; }

there is a higher chance to get the server connection "preheated" before a stall occurs.


Comments to blade@debian.org
[Eduard Bloch, Tue, 20 Nov 2007 00:03:24 +0100]