Converting from mbox to Maildir

Introduction

This article describes how to convert from mbox to Maildir format. Maildirs have quite a few advantages over mbox format, especially when you have a lot of users with large mailboxes. Another big advantage is that with dovecot and Maildirs, users can create sub-folders.

These notes are specific to my situation. I wasn't able to find a complete guide to doing this anywhere, so hopefully it'll serve as a good guide to others. My system uses:

Before the conversion

Before the conversion, everyone used the mbox mail format. Everyone's mail spool was /var/mail/$USER. In addition, dovecot was setup with default_mail_env = mbox:%h/Mail/:INBOX=/var/mail/%u. So people also had mail in $HOME/Mail/, such as $HOME/Mail/Spam, $HOME/Mail/Sent etc.

Not every user on my system has a valid shell. Some users are mail only, so have a shell of /bin/false. Others are webspace only so have a shell of /usr/sbin/scponlyc. This makes the conversion process a little more complicated because I wasn't able to su to these users to convert their mailboxes.

Preparation

Before doing on work on converting everything, I stopped dovecot and exim from running and then changed my firewall so only I had access to these services. I didn't want any mail coming in or users checking their mail during the conversion. I also made sure all mail queues were cleared down.

It goes without saying that a full backup should be done and if possible, run a test restore. Just in case something goes horribly wrong.

Configuring exim

The first task was to reconfigure exim. By default it delivers mail to /var/mail/$USER, in mbox format. To tell exim that it should now deliver email to a Maildir you should change the local_delivery transport so that it looks like:

local_delivery:
{{{
 driver = appendfile
 create_directory = true
 directory = ${home}/Maildir/
 mode = 0600
 directory_mode = 0700
 maildir_format
 delivery_date_add
 envelope_to_add
 return_path_add

}}}

Also check the rest of your exim configuration, to ensure nothing is configured to look at /var/mail/$USER or $HOME/Mail. One pitfall I had was an overcomplicated and poorly written procmail transport.

Remember to restart exim.

Configuring dovecot

dovecot is very easy to reconfigure. Simply change the default_mail_env line so it looks like:

default_mail_env = maildir:%h/Maildir

Remember to restart dovecot.

Check the global procmailrc file

This one caused me a headache for quite a while. If you have a global procmailrc file (/etc/procmailrc) then you should to add the following to the beginning:

DEFAULT=$HOME/Maildir/

Run a test conversion

To do this you need to install mb2md which is a Perl script that will convert any mail in an mbox format to in to a Maildir format.

Before attempting a "mass conversion", I'd strongly suggest you sacrifice your own account as a test. Backup your /var/mail/$USER spool and your $HOME/Mail somewhere (just in case). Now run:

$ makemaildir.dovecot $HOME/Maildir
$ mb2md -s /var/mail/$USER -d $HOME/Maildir
$ mb2md -s $HOME/Mail -R -d $HOME/Maildir
$ cp $HOME/Mail/.subscriptions $HOME/Maildir/
$ mv /var/mail/$USER /var/mail/$USER.preMaildir
$ mv $HOME/Mail $HOME/Mail.preMaildir

Now you should have your mail in Maildir format. Give it a test by loading your mail client and making sure everything still works (all your mail is there, IMAP subscriptions are there and you can recieve a test email from yourself).

If something isn't quite right, check:

Once everything is working, undo the work you just did (because next we'll be converting everyone, at once):

$ rm -rf $HOME/Maildir
$ mv /var/mail/$USER.preMaildir /var/mail/$USER
$ mv $HOME/Mail.preMaildir $HOME/Mail

Converting everyone at once

This is the scary bit. Ensure you do have a working backup. As I mentioned previously, not all my users had a valid shell so it made automating the conversion a bit more tricky. So I decided to run the conversions as root and then use chown afterwards. All of this was scripted and the script I wrote and used is as follows:


{{{#!/bin/sh #

  1. *** README *** README *** README *** README *** README *** README ***

#

  1. This conversion script worked fine for me (tm). YMMV.

#

  1. Things to look out for:
  2. 1. On my system, mail in mbox format was also stored in ~/Mail. Your
  3. system might be different (~/mail/ or just ~/mbox).
  4. 2. The maildirmake.dovecot may be called something else on your system.

#

  1. Things you should do before executing this:
  2. 1. Add an "echo" before any of the commands, so they don't execute
  3. which will allow you to make sure everything looks OK.
  4. 2. Change line 29 so that it only processes a test account, not
  5. everyone (i.e. change "-ge 500" to "-eq 500" to only process the
  6. user account with UID 500).

#

  1. *** README *** README *** README *** README *** README *** README ***

IFS=$'\n'

  1. Cycle through users.

for line in "$(getent passwd)"; do

        # Only attempt to process a user with a UID > 500.
        if [[ $(echo|"$line" | awk -F: '{print $3;}') -ge 500 ]]; then

                # Get some information about the user.
                username=$(echo $line | awk -F: '{print $1}')
                uid=$(echo "$line" | awk -F: '{print $3}')
                gid=$(echo "$line" | awk -F: '{print $4}')
                homedir=$(echo "$line" | awk -F: '{print $6}')

                # Do they have a mail spool or mbox directory?
                if [[ -f|/var/mail/$username ]] || [[ -d|$homedir/Mail ]]; then

                        # For whatever strange reason, they may already have a
                        # ~/Maildir, when they probably shouldn't.
                        if [[ -d|$homedir/Maildir ]]; then
                                mv $homedir/Maildir $homedir/Maildir.preMaildir
                        fi

                        # Setup the Maildir structure.
                        maildirmake.dovecot $homedir/Maildir

                        # Convert their mail spool.
                        if [[ -f|/var/mail/$username ]]; then
                                mb2md -s /var/mail/$username -d $homedir/Maildir
                                mv /var/mail/$username /var/mail/$username.preMaildir
                        fi

                        # Convert their old mbox directory.
                        if [[ -d|$homedir/Mail ]]; then
                                mb2md -s $homedir/Mail/ -R -d $homedir/Maildir
                                cp -p $homedir/Mail/.subscriptions $homedir/Maildir/
                                mv $homedir/Mail $homedir/Mail.preMaildir
                        fi

                        # Uncomment the following to attempt to generate a
                        # subscriptions file (probably a bad hack).
                        #find $homedir/Maildir -type d -iname ".*" > $homedir/Maildir/.subscriptions

                        # Give the user back their rights.
                        chown -R $uid:$gid $homedir/Maildir
                        chmod 600 $homedir/Maildir/.subscriptions

                        if [[ -f|$homedir/.procmailrc ]] || [[ -f|$homedir/.forward ]]; then
                                echo "*** $username has a .procmail or .forward!"
                        fi
                        echo "*** $username has been converted."

                fi

                echo
        fi
done


}}}

This script worked perfectly and converted all the users in one go without a problem. Everyone's mail spool in /var/mail will be mv'd out of the way and have ".preMaildir" appended to it. The same applies for $HOME/Mail. Remember to investigate people's .procmailrc or .forward files to make sure they're compatible.

Changing the shell environment

The final part of the process is to change the shell environment. Without doing this, pam_mail (the PAM module that tells you if you've got new mail when you login) won't work and neither will mutt.

First edit /etc/pam.d/login and /etc/pam.d/ssh and add dir=~/Maildir to the pam_mail.so line:

session    optional   pam_mail.so standard noenv dir=~/Maildir

Now edit /etc/login.defs and uncomment the QMAIL_DIR line.

I'd also suggest running makemaildir.dovecot /etc/skel/Maildir

Lastly, if you use mutt, edit /etc/Muttrc and add/edit the following:

set mbox_type=Maildir
set folder="~/Maildir/"
set mask="."

Open the flood gates

Do a final check. Make sure everything looks correct and then unfirewall the exim and dovecot services. Finally keep your fingers crossed that no one notices any changes.


Page written by DavidRamsden

LinuxHints/ConvertFromMboxToMaildir (last edited 2010-10-22 13:19:20 by 41)