Blog of Sylvain Le Gall

This blog post is a little recipe to do a Debian migration for a node using Puppet and some other good practices.

We do all the following commands as root, which on of the exceptional situation where you should have a real root session (login through console, or su -).

I tend to avoid using the X server while doing an upgrade, so my 'best' setup is to have a laptop to take note and check things on the internet and a session to the computer to upgrade (ssh + su - or login as root into console). In both case, I use screen during the upgrade so that I can handle disconnection.

Create or update your admin-syscheck script

First of all, a good practice is to have a script that runs various test on the system that checks everything is ok. This is not only for upgrade but in general. But in the case of upgrade, it can be particularly useful. I call this script admin-syscheck. This is a simple bash script.

This script check various aspect of the system and serve me as my external worker to check the most advanced knowledge I have gathered about setting up a service. For example, I know that having *.bak or *.dpkg-dist in /etc/ means that something needs to be merged and a file should be deleted. Another example is about setting up the right alias for 127.0.0.1 and ::1 (which you can differentiate using getent ahostsv4 and getent ahostsv6).

I have packaged this script and distributed it using a specific apt-get repository. You can just distribute it using puppet. I recommend to run it daily to track changes (e.g. after an apt-get dist-upgrade) and to check that my setup is aligned with my most up-to-date knowledge about setting up a service (i.e. this is my external worker).

In our case we are interested in checking presence of old and new configuration files, before and after upgrading. Here is this specific section of my script:

if ! $IS_ROOT; then
  warning "Not trying to detect dpkg leftover file in /etc/."
else
  LEFTOVER_FILES=( $(find /etc/ \
      -name "*.dpkg-dist" -o \
      -name "*.dpkg-old" -o \
      -name "*.ucf-old" -o \
      -name "*.ucf-dist" -o \
      -name "*.bak") )
  for i in "${LEFTOVER_FILES[@]}"; do
    if [ "$i" = "/etc/hosts.deny.purge.bak" ]; then
      continue
    fi
    if $fix; then
      BASE=${i%.*}
      cond_exec vim -d $BASE $i
      read -p "Delete $i (y/N)? " ans
      if [ "$ans" = "y" ]; then
        cond_exec rm $i
      fi
    else
       report_error "dpkg leftover file: '$i'."
    fi
  done
fi

(cond_exec allows to do a dry run, you can just remove it).

Setting $fix to true will spawn a vim -d old new command where you can edit and then delete the leftover file. This is extremly handy.

Upgrading to Wheezy

I strongly recommend to read first the upgrade chapter of the release notes. This gives you a more complete overview of the upgrade procedure. I just go through the basic step here.

1. Update everything on the system:

$> apt-get update 
$> apt-get dist-upgrade

2. Check that the current configuration apply cleanly:

$> puppet agent --test

3. Run admin-syscheck:

$> admin-syscheck

And fix all the problems.

4. Disable puppet:

I use a cronjob to run puppet, so I just comment the line for the job (/etc/cron.d/puppet-custom). You should disable puppet by stopping the daemon and preventing it to run by editing /etc/default/puppet and set START=no.

4. Fix your sources and pinning:

Change squeeze to wheezy in /etc/apt/sources.list and remove useless files in /etc/apt/sources.lists.d/. (You may keep certain sources that refers to stable, like google-chrome.list).

$> rm /etc/apt/sources.list.d/* # Check if this ok to do this with your system.

Although, I tend to fully purge /etc/apt/sources.list expect for the main line (removing backports and security is fine for a short time). The first run of puppet after upgrade will anyway reset this file.

$> rm /etc/apt/preferences.d/* # (at least the ones that do pin some version)

You can although remove all pinning from /etc/apt/preferences.

5. Now you start the real upgrade:

$> apt-get update 
$> apt-get dist-upgrade

6. During the upgrade you will be asked if you want to keep old configuration files or install the newer one from the maintainer.

I have always wondered what to answer to this. But here is the answer after a few major upgrade: always install configuration files from maintainer if the service has no ultra-specific settings that could break during the upgrade.

The only file, that I should not upgrade on my system, is /etc/sudoers. In this very specific case, you need to make sure before the upgrade that the old and new configuration can coexist. In the squeeze to wheezy case, I have just setup a few extra augeas rules to set the secure_path before the upgrade and it was fine. This is typically the kind of situation where you are thankful to have a real root session.

7. The upgrade can be long and require various fixing to remove/re-add packages to circumvent problems. At the end you will have a set of file *.dpkg-old and *.ucf-old (and some *.dpkg-dist and *.ucf-dist). The *-old files are your old version of the file, while the corresponding files match the maintainer version of it. The *-dist files are the maintainer version of the file and the corresponding files match your old version of it.

Starting from here you have 2 options:

• This is one of the first computer you upgrade, go to 'first upgrade'.
• Your puppet configuration for wheezy is already bullet proof, go to 'further upgrade'.

First upgrade

This is the tricky part, you'll have to spend a little time on it:

1. Go over all *.{ucf,dpkg}-{old,dist} and merge them with the corresponding configuration file. Use admin-syscheck with fix=true

2. Make a copy of your /etc directory into /etc.new:

$> cp -rp /etc /etc.new

3. Run puppet again:

$> puppet agent --test

4. Disable again the automatic run of puppet, if the previous command has re-enabled it.

5. Make a diff between /etc and /etc.new. Since you have a run of puppet, you know what has changed and should not have changed.

$> diff -Nurd /etc.new /etc

Everytime you find some files that doesn't match your expectation for the upgrade with puppet, change the corresponding puppet manifest to have what you expect.

For example, if this is a file:

if ($lsbdistcodename == 'wheezy') {
  file {
    "/etc/foo":
        source => "puppet:///files/foo.wheezy"
   }
} else {
  file {
    "/etc/foo":
        source => "puppet:///files/foo.squeeze"
   }
}

People working with augeas and puppet, will appreciate the fact that they probably have 0 changes to make for this to work (since it only does a few replacement in configuration files).

6. Once you are happy with the changes, copy back /etc.new to /etc and go to step 3 until the difference is almost 0.

7. Re-enable automatic run of puppet.

Do this procedure for a least each computer category you have (e.g. Desktop and Server nodes). Once you are fully confident your new puppet setup works, you will be able to use 'further upgrade' for the other nodes.

Further upgrade.

This one is super easy compare to a first upgrade:

1. Re-enable puppet and have it run at least once:

$> puppet agent --test

2. Merge *.{dpkg,ucf}-{dist,old} with corresponding files (you can run admin-syscheck with fix=true). This is mostly a sanity check since you should have already solved most problem with the 'first upgrade' procedure.

That's it.

Enjoy your upgrade to Wheezy with puppet.