How to mirror FreeBSD


Prev		Next

2.1 FTP

The FTP area is the largest amount of data, that needs to be mirrored. It includes the distributions sets, required for network installation, the branches, that are actually snapshots of checked-out source trees, the ISO Images to write CD-Roms with the installation distribution, a live filesystem, and lots of packages, the ports tree, distfiles and a huge amount of packages. All of course for various FreeBSD versions, and i386 and alpha architecture.

2.1.1 With FTP mirror

You can use a FTP mirror program, to get the files. There are a lot around, and widely used, like:

ftp/mirror was very popular, but seemed to have some drawbacks, as it is written in perl(1), and did have real problems on mirroring large directories like a FreeBSD site. There are rumors, that the current version has fixed this, by allowing to specify a different algorithm for comparing the directory structure.

In general FTP is not really good for mirroring, since it transfers each whole file, if it has changed, and does not create a single data stream, that will benefit from a large TCP congestion window.

2.1.2 With RSYNC

A better way, to mirror the FTP area is rsync. You can install the port net/rsync and then use rsync to sync with your upstream host. rsync is already mentioned in Section 1.4.2. Since rsync access is not required, your preferred upstream site may not allow it. Since it is quite common, though, chances are small, that you cannot use it. You can always consider using an upstream server, that offers it, just for the benefits of rsync.

Note: Since the number of rsync clients will have a significant impact on the server machine, most admins impose limitations on their server. For a mirror, you should ask the site maintainer you are syncing from, about their policy, and maybe an exception for your host (since you are a mirror).

A command line to mirror FreeBSD could look like that:

    % rsync -vaz --delete ftp4.de.freebsd.org::FreeBSD/ /pub/FreeBSD/

Consult the documentation for rsync, which is also available at http://rsync.samba.org/ about the various options to be used with rsync. Also you might want to set up a script framework, that calls such a command via cron(8).

2.1.3 With CVSup

A few sites, including the one-and-only ftp-master.freebsd.org even offer CVSup to mirror the contents of the FTP space. You need to install a cvsup client, preferably from the port: net/cvsup. (Also reread Section 1.4.4.) A sample supfile, suitable for ftp-master.freebsd.org looks like this:

              #
              # FreeBSD archive supfile from master server
              #
              *default host=ftp-master.FreeBSD.org
              *default base=/usr
              *default prefix=/pub
              #*default release=all
              *default delete use-rel-suffix
              *default umask=002
    
              # If your network link is a T1 or faster, comment out the following line.
              #*default compress
    
              FreeBSD-archive release=all preserve

It seems CVSup would be the best way to mirror the archive, in terms of efficiency, but it is only available from few sites. In fact I just know ftp-master.freebsd.org for sure.

Note: Please have look at the CVSup documentation like cvsup(1) and consider using the -s option, as it can reduce the amount of work to be done a lot.

2.2 Mirroring the CVS repository

Again you have various possibilities, but the most recommended one, is to use CVSup.

2.2.1 Using CVSup

CVSup was already described to some detail in Section 1.4.4 and Section 2.1.3.

Here we just describe an example to set up the supfile:

              #
              # FreeBSD CVS supfile from master server
              #
              *default host=cvsup-master.FreeBSD.org
              *default base=/usr
              *default prefix=/pub/FreeBSD/development/FreeBSD-CVS
              *default release=cvs
              *default delete use-rel-suffix
              *default umask=002
    
              # If your network link is a T1 or faster, comment out the following line.
              #*default compress
    
              cvs-all

You should also have a look at /usr/share/examples/cvsup

Note: Please don't forget to consider the hint, mentioned in this note above.

2.2.2 Using other methods

Using other methods than CVSup is generally not recommended. We describe them in short here anyway. Since most sites offer the CVS repository as part of the FTP fileset under the path /pub/FreeBSD/development/FreeBSD-CVS, the following methods could be used.

FTP
RSYNC
maybe even HTTP

If you find a site, that supports it, you could use net/sup, but it is inferior to CVSup and it's deficiencies caused John Polstra to develop CVSup in the first place, so it is clearly not recommended.

Important: You can NOT use AnonCVS to mirror the CVS repository, since CVS does not allow you to access the repository itself, but only checked out versions of the modules.

2.3 Mirroring the WWW pages

The best way is, to check out the www distribution from CVS. If you have a local mirror of the CVS repository, it is probably as easy as:

    % cvs -d /home/ncvs co www

and a cronjob, that calls cvs up -d -P on a regular basis, maybe just after your repository was updated. Of course, the files need to remain in a directory, available for public WWW access. The installation and configuration of a webserver is not discussed here.

If you don't have a local repository, you can use CVSup to maintain an ``up to date copy'' of the www pages. A sample supfile can be found in /usr/share/examples/cvsup/www-supfile and could look like this:

            #
            # WWW module supfile for FreeBSD
            #
            *default host=cvsup3.de.FreeBSD.org
            *default base=/usr
            *default prefix=/usr/local
            *default release=cvs tag=.
            *default delete use-rel-suffix
    
            # If your network link is a T1 or faster, comment out the following line.
            *default compress
    
            # This collection retrieves the www/ tree of the FreeBSD repository
            www

Using ftp/wget or other web-mirror tools, is probably not recommended.

2.3.1 Mirroring the FreeBSD documentation

As the documentation is referenced a lot from the webpages, it is recommended, that you mirror the FreeBSD documentation as well. However, this is not so trivial as the www-pages alone.

First of all, you should get the doc sources, again preferably via CVSup. Here is a corresponding sample supfile:

             #
             # FreeBSD documentation supfile
             #
             *default host=cvsup3.de.FreeBSD.org
             *default base=/usr
             *default prefix=/usr/share
             *default release=cvs tag=.
             *default delete use-rel-suffix
    
             # If your network link is a T1 or faster, comment out the following line.
             #*default compress
    
             # This will retrieve the entire doc branch of the FreeBSD repository.
             # This includes the handbook, FAQ, and translations thereof.
             doc-all

Then you need to install a couple of ports. You are lucky, that there is a meta-port: textproc/docproj to do the work for you. You need to setup some environment variables, like SGML_CATALOG_FILES, also have a look at your /etc/make.conf (copy /etc/defaults/make.conf if you don't have one), and look at the DOC_LANG variable. Now you are probably ready to run make in you doc directory (/usr/share/doc by default) and build the documentation. Again you need to make it accessible for your webserver and make sure, the links point to the right location.

Important: The building of the documentation, as well as lots of side issues, is documented itself in: fdp-primer. Please read this piece of documentation, especially if you have problems, building the documentation.

Note: XXX MAYBE THIS CAN BE LINKED FROM WITHIN - NOT USING AN ABSOLUTE URL XXX

2.4 How often should I mirror?

Every mirror should be updated on a regular basis. You will certainly need some script framework for it, that will be called by cron(8). Since nearly every admin does this his own way, we cannot give specific instructions. It could work like this:

Put the command to run your mirroring application in a script. Use of a plain /bin/sh script is recommended.
Add some output redirections, so diagnostic messages are logged to a file.
Test, if your script works. Check the logs.
Use crontab(1) to add the script the appropriate user's crontab(5).

Here are some recommended schedules:

FTP fileset: daily
CVS repository: daily to hourly
WWW pages: daily