Before migrating to 10.6 Server we were running our entire mailstore on the 10.5.8 platform. Say what you will about Leopard, once we had it set up correctly it ran fine. We migrated because of the greater benefits of Dovecot over Cyrus. Many of the other features intriqued us as well such as the improved Wiki server and iCal server. However we were really excited about Dovecots ability to repair itself basically doing away with the need to ever have to rebuild a users mailbox.

The one great thing about 10.5.8 was the development of Mailbfr this was an amazing script that would help you backup your entire mailstore, recover email accounts, repair quotas, and of course rebuild or repair the entire mail-store. This was an invaluable tool, however since switching we have had to come up with our own solution.

We tried at first to get RSYNC running however this did not work because we did not have the permission to copy the mailstore with the permissions in tact. The reason is that while the root account does have access to look at the mailstore the secondary user on the mailstore folder is the mail user itself. Without running the script as each user then its near impossible to use RSYNC to move the mailstore or to even perform a simple backup.

Our solution was to create a backup script that use SCP with SCP we were able to move our mail-store to another drive on the server. It copies the store over and it resets the permissions to the administrator account. This solution works very well however after some time with a large mailstore you will run out of space on the target volume. Unlike RSYNC which uses hard links to save space SCP creates a new copy of the mailstore each time it is run.

In order to save space on the backup volume the oldest 2 weeks worth of backed up email gets dumped to DVD and removed from the drive on a monthly basis. This is fine but not optimal. Here is a copy of the script that we use on our server.

1
2
3
4
5
6

#!/bin/bash
echo backup started daily backup `date` >> /Volumes/EMAIL\ BACKUP/Backup/Logs/Backup_log.txt
 
scp -r /Volumes/Mailstore-Location/spool /Volumes/EMAIL\ BACKUP/Email-Backups/$(date +%d)-$(date +%m)-$(date +%Y)backup 
 
echo backup daily backup completed `date` >> /Volumes/EMAIL\ BACKUP/Backup/Logs/Backup_log.txt

This will backup the mailstore and then log each time that it does so. To recover an email to the original mailstore is not as hard as it seems. Navigate to the backed up mailstore destination and match up the name of the folder to the users UID of which you want to recover. For example if the users UID is 7458-58713-952554-544226 then you would look for a folder with the same name. Once in the folder you can copy or look at individual email files. Find the ones or one that you need and copy it to the folder of the original mailstore. In order to do this you will have to use sudo. For example this is how you would restore the entire folder

1

sudo scp -r "Volumes/EMAIL\ BACKUP/Email-Backups/7458-58713-952554-544226/.*" "/Volumes/Mailstore-Location/spool/7458-58713-952554-544226/.*"

Until Mailbfr comes back for Dovecot this is how we are protecting ourself against the accidental loss of email. I am not saying that this is the best method it is simply the one we are using. If you have another solution that works bette than please let me know and share your own experience!

*** UPDATE 12/8/2011 ***
Here is the script that I am using to date for your use. It logs the backups and emails me when they are complete.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#!/bin/bash
 
# This adds a record to our backup log
 
echo backup started daily backup `date` >> /Scripts/Logs/Backup_log.txt
 
# This backs up the files
 
rsync -avu /location/of/mail/spool /Volumes/backup-volume/
 
# This adds another record to our backup log
 
echo backup daily backup completed `date` >> /Scripts/Logs/Backup_log.txt
 
# This emails the Sys Admin
 
NOW=$(date +"%m-%d-%Y")
 
SUBJECT="Email Backed Up - "$NOW""
 
EMAIL="email@user.com"
 
EMAILMESSAGE="/Scripts/Logs/Backup_log.txt"
 
mail -s "$SUBJECT" "$EMAIL" < "$EMAILMESSAGE"

The easiest way to start / stop this is to setup a LaunchD or Cronjob, I prefer Cron since its easier in my opinion to setup. Setup the rsync script that you have below and put the code in a bash script. I use a GUI tool called Cronix http://code.google.com/p/cronnix/ once you download this launch it and put the full path to the bash script at the bottom the cron job code for every ten minutes is

0/10 * * * * * /path/to/bash/script.sh

[...] 10.6.2 Dovecot Mail Backups « Musings of an Apple Systems … [...]

[...] 10.6.2 Dovecot Mail Backups « Musings of an Apple Systems … [...]

Hi Jon!

Was googling around and found your post on the mail backups. I’m playing with rsync right now a bit and am using:

rsync -avu /Volumes/m1data/mailstores /Volumes/m1data/mail_backup/

to move my mailstores into another backup directory. When I’ve been doing my checking on the permissions on the mailstores it looks ok with those switches.

(Oh and very much agree with the Leopard being happy once the performance was figured out. We had 850 connections on it regularly, it wasn’t until after Thanksgiving this year that our connection load shot to 1250 and the server hit a process limit, which updating to Snow Leopard took care of nicely.)

I’m trying to backup using:
rsync -avu /var/spool/imap/dovecot/mail /Volumes/External

But there seems to be a problem with the permissions.
The folders named by user-id’s inside /imap/dovecot/mail have these permissions:
Username: Read&Write
everyone: no access

Should I change the permissions in any way to make the rsync command work?

The server manual states that mailservices have to be shut down before backing up otherwise the activemail storage is not backed up correctly. Mailbfr would only shutdown cyrus mail during backup, since other services should basically keep running and making a mailcue during backup.
On an apple dicussion is also mentioned:
launchctl unload /System/Library/LaunchDaemons/org.dovecot.dovecotd.plist
do your backup
launchctl load /System/Library/LaunchDaemons/org.dovecot.dovecotd.plist
(and not with the -w option of course, that would make it permanent and also seems to cause some problems)

What are your suggestions on this?

(they use rsync)
http://discussions.apple.com/thread.jspa?threadID=2204308

My name is Stephen Hutchinson Mac engineer, and I’m hoping you can help me regarding backing up Dovecot Mail. I’m pretty new to this. I’ve read your discussion (below) and presently testing backing up Dovecot with the rsync method that works a treat however, how do I schedule this to backup every 10 minutes to the same destination incrementally ?

By Jon Brown

The easiest way to accomplish this is to setup a LaunchD or Cronjob, I prefer Cron since its easier in my opinion to setup. Setup the rsync script that you have below and put the code in a bash script. I use a GUI tool called Cronix http://code.google.com/p/cronnix/ once you download this launch it and put the full path to the bash script at the bottom the cron job code for every ten minutes is

0/10 * * * * * /path/to/bash/script.sh

I hope that helps you out here is a screenshot as well.

Reply

Jon, these are great suggestions. However, I have one dilemma with rsync. I notice that when the script is run, it dates the destination folder created with the same date as the original folder. The problem with that is that I have a purge script set to remove folders older than seven days, and if it uses the date of the original store it would be past seven days. Do you know of any way to get rsync to date the top level folder with the current date? scp would do this, but I would rather use rsync. Thank you.

Jon, it appears that your latter suggestion worked. Thank you so much. Now I can have mail stores older than seven days purged.

Jon. Apologies for being a little off topic.

I have a client running 10.6.7 server. We have 20 or so mailboxes on it which all have quite a lot of email in them 10 GB is not unusual.

My problem is that one user is complaining of emails arriving and then disappearing. I have checked his Mail app for rules which might explain this, but found nothing.

Do you have any tips on how to troubleshoot odd behavior like this? I have started to create a new account for him, and was thinking I would move all of his email into the new account and set the old one to forward mail to the new.

Before I do this, I was wondering if there is some utility or technique to rebuild an existing mailbox like we had with mailbfr. Any suggestions you have would be very appreciated.

Thanks,

Rusty

Hmm, early onset file corruption sounds scary…

While I cannot be 100% sure he didn’t delete the mail, there are more than a few messages in question, and I could not find them via web client.

Can you elaborate a bit on the file corruption possibility?

Thanks so much.

Ok, I have decided to try your suggestion to back up the user’s mail store, delete the account, then restore the mail store for that user and update permissions.

Unfortunately I don’t know how to do this. Do you know of any online instructions for this task? I am guessing I could modify your backup script above to make the copy and copy back. But, I don’t know how to identify which folder inside of spool/imap is the correct one for that user.

Again thank you for taking the time to offer help.

1

sudo rsync -avp /path/of/user/store /target/drive

1

defaults write com.apple.Finder AppleShowAllFiles TRUE

1

killall Finder

OK one more question. I get the following error trying to run the script for backups.

line 2: syntax error near unexpected token `&’

This is your line 2. (I subbed in my path in my version).
echo backup started daily backup `date` >> /Volumes/EMAIL\ BACKUP/Backup/Logs/Backup_log.txt

I looked up $gt; and see that it translates to >. Should I just put the greater than sign in where $gt; appears?

Thanks!

Well answered my own question when i posted it, since the form translated the items into greater than signs… and they worked…

Next question, do I have to log in as root to set up the cron job?

Thanks!

I was able to do the backup, delete, restore. Thank you!!

The user had a ton of .Saved folders which I didn’t notice at first until I saw your note about making invisible files visible… It would have caused a panic if I had forgotten those…

So my last question is, does Cronnix still work with 10.6 server? I set up a job to run at hour 22 every day. But it has not run on its own. I’m using your tip for the backup script, which I was able to get working just fine if I simply reference the .sh file via terminal. And, I logged in as root to set up the cron job.

Thanks again!!

By Jon Brown

Yes, I am glad you caught that. First off yes, Cronnix does work in 10.6 all of my machines are 10.6 however in order to run the script you need to put /bin/bash /path/to/script.sh instead of just /path/to/script.sh in the command section. You can see an actual screenshot from my server below.

Reply

[...] In Apples guide for administering the Mail Service they strongly recommends you to stop the mail service before running a backup of it. My previous scripts has done this and contained more code for stopping/checking status/starting than for actually fulfilling it’s purpose. That was until I read the comments on “10.6.2 Dovecot Mail Backups” on Musings of an Apple System Administrator. [...]

So could we have the updated script, with time/date fix and how you stop/start it as well?

Thanks

By Jon Brown

Here is the updated code as requested.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

#!/bin/bash   # This adds a record to our backup log   echo backup started daily backup `date` >> /Scripts/Logs/Backup_log.txt   # This backs up the files   rsync -avu /location/of/mail/spool /Volumes/backup-volume/   # This adds another record to our backup log   echo backup daily backup completed `date` >> /Scripts/Logs/Backup_log.txt   # This emails the Sys Admin   NOW=$(date +"%m-%d-%Y")   SUBJECT="Email Backed Up - "$NOW""   EMAIL="email@user.com"   EMAILMESSAGE="/Scripts/Logs/Backup_log.txt"   mail -s "$SUBJECT" "$EMAIL" < "$EMAILMESSAGE"

The easiest way to start / stop this is to setup a LaunchD or Cronjob, I prefer Cron since its easier in my opinion to setup. Setup the rsync script that you have below and put the code in a bash script. I use a GUI tool called Cronix http://code.google.com/p/cronnix/ once you download this launch it and put the full path to the bash script at the bottom the cron job code for every ten minutes is

0/10 * * * * * /path/to/bash/script.sh

Here is a screenshot as well using Cronix

Reply