Ubuntu as an Apple Time machine capsule with Netatalk

I use my Ubuntu server for nearly everything, it'd be great if my Macs could back up to it too using the great built in Apple Time Machine functionality. TimeMachine relies on the AFP as the transfer protocol for backing up and transmitting data over the network. Netatalk is the Open Source implementation of AFP. Mac OS X requires encryption to work properly but the standard package of netatalk provided in the Ubuntu repositories doesn’t include this feature. So we have to build our own netatalk package from the sources with the encryption feature enabled. Read on...
This post is heavily based upon and updates Matthias Kretschmann's excellent original TimeMachine/Ubuntu guide, all credit to Matthias and his respective posters. This post reflects my experience, difficulties and solutions in setting up TimeMachine between Leopard 10.5.8 and Ubuntu 10.04LTS using Netatalk and Avahi. Updated in 2014 to also work with Mavericks & Yosemite and Ubuntu 14.04.

We'll walk through setting up TimeMachine  using an Ubuntu server as a Time Capsule from step one:
  
1. Enable source code
First you have to enable the Source Code repositories via System > Administration > Software Sources under the Ubuntu Software tab. Check the Source Code Box, click Close and choose Reload in the next dialogue.

Now fire up your Terminal under Applications > Accessories and execute the following lines (separately). You have to type Y for yes when Terminal asks you if it should continue.
cd ~
sudo apt-get build-dep netatalk
sudo apt-get install cracklib2-dev fakeroot libssl-dev
sudo apt-get source netatalk
cd netatalk-2*
  
2. Build and Install Netatalk 
Next you have to build the Netatalk package with the encryption option enabled:  

sudo DEB_BUILD_OPTIONS=ssl dpkg-buildpackage -rfakeroot

This may take a while and you will see the build output in the terminal. If Netatalk built without errors the created package can now be installed:

sudo dpkg -i ~/netatalk_2*.deb 

To stop Ubuntu from overwriting your custom Netatalk package you should set its state to hold. This will cause the Netatalk package being grayed out in the syanptic software update:

echo "netatalk hold" | sudo dpkg --set-selections 

Now you have successfully build and installed your custom Netatalk package which now has support for encrypted logins.

3. Configure Netatalk 
First you should deactivate services provided by Netatalk which are not needed if you just want to use your Ubuntu box for file sharing. This will speed up the response and startup time of Netatalk dramatically. For instance Netatalk starts the old AppleTalk protocol by default which is just needed for pre OS X systems. To use edit the config to stop the unneeded services from a terminal run:

sudo gedit /etc/default/netatalk 

gedit should pop up with the netatlak config file loaded as superuser. Find the “#Set which daemons to run” part and replace the default values with the following to enable just AFP and disable all unneeded services.

ATALKD_RUN=no
PAPD_RUN=no
CNID_METAD_RUN=yes
AFPD_RUN=yes
TIMELORD_RUN=no
A2BOOT_RUN=no

Then Save and Close.

Here it’s very important to run the cnid_meta daemon because this service will handle all the metadata for us (namely the resource fork) which would get lost due to the fact that your Linux box isn’t formatted as Apple’s HFS+. If you’re interested what the other services could do: 

PAPD_RUN is the pap daemon used to share Linux connected printers with Macs   
ATALKD_RUN is the AppleTalk daemon (pre-OSX file sharing, old printing)
TIMELORD_RUN can allows the Linux box to act as a network time server
A2BOOT_RUN suggests it’s a netboot server for client Macs 

Next we have to edit the main config file for AFP sharing called afpd.conf:

sudo gedit /etc/netatalk/afpd.conf
 
Scroll to the bottom of the config and add this to the bottom (replace the whole line in case there’s already one). This is one line so be sure that there’s no line break in your afpd.conf file: 

- -transall -uamlist uams_randnum.so,uams_dhx2.so -nosavepassword -advertise_ssh
 
Then Save and Close.
 
4. Configure shared Volumes
Now we have to tell the afpd daemon what Volumes to share. This is defined in the AppleVolumes.default file inside /etc/netatalk/. 

Create the folder that will be where Time Machine backs up to this is the netatalk volume = Time Machine Disk.

sudo mkdir /pathtotimemachinebackupfolder

The following line will open this file in the gedit editor with superuser privileges (required for saving) where we can define our shared volumes:

sudo gedit /etc/netatalk/AppleVolumes.default

Scroll to the bottom of the document and define your Volume shares. By adding the following line you will share each users home directory with the user name as the Volume name. To make things more secure you can define all users who are allowed to connect to your Ubuntu box via AFP. The simple syntax of the volume share is:

SharePath ShareName AllowedUsers Cnidscheme Options

Which in practice would equate to something like the following:

~/ "username" allow:backupwriter cnidscheme:cdb 

This will:

• share the home directory of the user that logs on to the afpd
• shares the directory under the username, used to log on afpd
• allows the user backupwriter only 

5. Configure the TimeMachine shared volume
Because we want to use the Ubuntu machine as a backup server for Time Machine you should define a volumes just for Time Machine. 


Using the folder you created earlier i.e. /pathtotimemachinebackupfolder add the following line to your AppleVolumes.default. This is one line so be sure that there’s no line break in your AppleVolumes.default file:

/pathtotimemachinebackupfolder "TimeMachineUsername" allow:YourUser cnidscheme:cdb options:tm,usedots

2014 UPDATE - For Mavericks and Yosemite the share looks like:

/pathtotimemachinebackupfolder/ "TimeMachineUsername" allow:YourUser cnidscheme:dbd options:upriv,tm,usedots


tm option: This enables Time Machine suport for the share.

usedots option: This is required if you want to use invisible files and folders (those starting with a dot in the name). Otherwise afpd would encode them as :2e which is bad if you have to use invisible files (like .htaccess). Personally I cannot get this to work, I can't figure out why at the moment and I'm still looking into it. I've omitted this and my time machine backups are eventually working but dots are converted to hex.

upriv option: If you’re on Leopard and have no Tiger installed Macs in your network or mixed OS X versions in your network you should use the upriv option which adds support for AFP3 unix privileges. If you have Macs with Tiger installed just use options:usedots to avoid unexpected behavior. I have both Tiger and Leopard on my network so I don't use upriv.

My configs for pre Maverick Time Machines look like:

  /DATA/Backup/TimeMachineUserName/ "TimeMachineUsername" allow:backupwriter  cnidscheme:cdb options:tm

Finally if you want more stability and can accept slower file transfers you can use the dbd cnidscheme (cnidscheme:dbd).

Press Ctrl + S to save the document or choose File > Save.

Important - Your back up folder i.e. /pathtotimemachinebackupfolder must be owned by the same user you specified in the netatalk config i.e. YourUser  this will be the user you use in Mac OSX Time Machine preferences to access the Time Machine Disk.

To do this:

sudo chown YourUser:YourUser /pathtotimemachinebackupfolder 

Finally restart Netatalk to activate the changes:

sudo /etc/init.d/netatalk restart 

Although we now have a fully configured AFP file server it will not show up in the Finder sidebar on Mac OS X Leopard (but it’s reachable via Go > Connect to Server… in the Finder). Macs use a service called Bonjour for the sidebar thing (and for a lot of other cool stuff) and on the Linux side we can have this functionality with the Open Source implementation of Bonjour, called Avahi.
 
6. Install Avahi
The Avahi daemon will advertise all defined services across your network just like Bonjour do. So let’s install the avahi daemon and the mDNS library used for imitating the Bonjour service. When fully configured this will cause all Macs in your network to discover your Ubuntu box automatically:

sudo apt-get install avahi-daemon
sudo apt-get install libnss-mdns
 
To make everything work properly you have to edit the nsswitch.conf file:

sudo gedit /etc/nsswitch.conf
 
Just add “mdns” at the end of the line that starts with “hosts:”. Now the line should look like this:

hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4 mdns

Press Ctrl + S to save the document or choose File > Save.
 
7. Configure Avahi and advertise services
Next we have to tell Avahi which services it should advertise across the network. In our case we just want to advertise AFP sharing. This is done by creating a xml-file for each service inside /etc/avahi/services/ following a special syntax. Let’s create a xml-file for the afpd service with the following line:

sudo gedit /etc/avahi/services/afpd.service
 
A blank document should open in gedit. Now paste the following into the document and save the file by pressing Ctrl + S or by choosing File > Save:

The last part from _device-info._tcp down is used to assign a specific (Apple) hardware model to your Linux box. In this example your server will be advertised as an XServe and will be shown with this icon in the Finder sidebar. This will come in handy when you want to use your own icon for it or the one’s made by me provided within this article. Thanks to Simon Wheatley for figuring this out. Additionally you can use these models in this file: 

• RackMac (same as Xserve)
• PowerBook
• PowerMac
• Macmini
• iMac
• MacBook
• MacBookPro
• MacBookAir
• MacPro
• AppleTV1,2
• AirPort

Finally restart the avahi daemon to activate all changes:

sudo /etc/init.d/avahi-daemon restart
 
Now you have configured the Avahi daemon to advertise AFP sharing across your network which will cause your Ubuntu box to show up in Finder’s sidebar in Mac OS X Leopard. In Mac OS X 10.4 Tiger your Ubuntu server should now be visible under Network.

Another side effect of using AFP is that your Ubuntu box will show up as a nice Apple icon. If you’ve followed this version of this article your Linux box should now be represented by a Xserve icon in Finder.

Now you’re done with setting up AFP file sharing on your Ubuntu box and advertising it across the network for Finder’s sidebar. Check if everything works as intended by clicking on your Ubuntu server in Finder and clicking on the “Connect As” button. Enter your username and password (the ones you’re using on your Ubuntu machine) and you should see the Volumes we defined earlier with the AppleVolumes.default file:

INSERT IMAGE 

In practice I had a lot off issues at this stage, although I could see the afpd server under Finder in OSX, I couldn't log on to it successfully or see any shares. I was receiving the following errors and my log files were full of non descript Netatalk 5002 errors:

"Connection Failed - There was an error connection to the server. Check the server name or IP address and try again" 

and 

"There was an error connecting to the server. Check the server name or IP address and try again. If you are unable to resolve the problem contact your system administrator." 

8. My issues 
My issues were, and were resolved as follows:

Disable your firewall until you get everything working
I had plenty of issues at this stage, turning the firewall off on the ubuntu  box helped to work through them and eliminate them one at a time. See the section later that talks about re enabling your firewall and configuring it successfully. To turn off your firewall using ufw:

sudo ufw disable

The owner of the Ubuntu directory shared must be the same as the user used to log into apfd 
This caused me a lot of pain. Despite recursively chmod'ing all of my directories that were to be shared over afpd to 777 to allow full read, write and execute access; I was still unable to see my shares from OSX (despite being successfully logged into the server). 

My afpd / TimeMachine backup shares reside on an external disk where the ultimate parent directory of the disk is owned by a different user to my backup user. My backup subdirectory however, is owned by my backupuser. This alone wasn't sufficient to allow the TimeMachine shares to be seen. After a lot of trial and error I found owning the root directory above my backup directory by the user root, group root, solved the problem.

sudo chown username:username /path_to_the_parent_directory_of_the_time_machine_share 

Where username matches the username you set to allow access to the netatalk volume in apple volumes, which is also the username you use in the Time Machine preferences.

Log into the afpd / Netatalk server using a valid Ubuntu user (not your Mac username) 
From OSX to log on to afpd / Netatalk you have to be using valid credentials for the Ubuntu machine. i.e. if you are logged on to your Mac as UserX, and UserX doesn't exist on the Ubuntu server with the same username and password then you will not be able to log in to Netatalk using your Mac user's credentials.

Instead you will need to use the 'Connect As' option in OSX and use a valid user from the Ubuntu machine. Alternatively you could try and set up a user to mimic your OSX user, with the same username and password as your OSX user but your success may vary. I use a dedicated backup user on my Ubuntu box so I log into apfd/Netatalk using the 'Connect AS' option under OSX. Once I had figured this, I was now able to successfully log on to the server.

9. Key Lessons Learnt 
1. Log onto afpd with a valid Ubuntu user, not the Mac user credentials;
2. Ensure the parent directory of the shared directory is owned by root;
3. Ensure the shared directory is owned by the user used to log onto afpd with;
4. Ensure the shared directory is readable, writeable and executable by the user  used to log onto afpd with.

10. Configure Time Machine on the Mac
On the Mac side you have to enable the option to use network volumes as Time Machine drives first. Without it your freshly shared and advertised network volume won’t show up in the disk selection dialogue in Time Machine. This is a hidden option not accessible via the graphical user interface so you have to copy & paste this in Terminal (it’s one line):

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1
 
Now just mount the “TimeMachine” Volume first and choose it as a backup disk in the Time Machine system preferences. Time Machine will create a sparsebundle disk image inside the volume and mounts that too. This is needed because the hard drive on your Ubuntu box is formatted as ext3 or Reiserfs and not the native Mac format HFS+. Because Time Machine backups everything to the disk image all metadata (like creation dates etc.) will be preserved.
When your first Time Machine backup is done you can remove all Volumes and the next time Time Machine starts it will automatically mount the disk image from your TimeMachine volume on your Ubuntu box without mounting the whole TimeMachine volume:

11. Firewall settings
In short I found I had to allow communications over ports

• 548 tcp
• 5353 tcp
• 49152 udp
• 52883  tcp

If with your firewall enabled you get the following errors: 

"Connection Failed - There was an error connection to the server. Check the server name or IP address and try again"

or 

"There was an error connecting to the server. Check the server name or IP address and try again. If you are unable to resolve the problem contact your system administrator."

Try adding the above rules. With ufw this is rather simple:

sudo ufw allow from [INSERTIPRANGE] to [SERVERIPADDRESS] ports [PORT#]
sudo ufw enable  

Remember that these errors can be caused by a myriad of problems and just a lot of other configurations on your side. So you should try a minimal config and build up to start eliminating issues.
 
12. Matthias Kretschmann's Conclusion, Problems and more information
I cannot really add to Matthias' conclusion here so all credit to Matthias:

You see that Linux and Ubuntu can be configured to behave like Macs in your network. But it’s sad, that you have to fire up the Terminal to achieve this. I hope especially the Ubuntu team will simplify this in future versions. Another sad thing is that Ubuntu is missing a useful avahi/Bonjour and AFP implementation for Nautilus, the file manager. So there’s no way to access your Mac from Ubuntu via AFP in the file manager. In my network I use SSH to access the Macs from Ubuntu or Linux by allowing Remote Login on the Macs in the Sharing preferences. But there’s a command line based AFP client available called afps-ng which uses the FUSE system.

I cannot guarantee this works without errors or backup data loss on your system too. So I encourage you to check the system logs on Ubuntu AND on Mac OS X with the Console app in your Utilities folder. 

Finally the only problem remaining is that your Ubuntu or Linux box isn’t formatted as journaled HFS+ so some scenarios would fail or make problems. This can include having your iTunes, iPhoto or Aperture library on your Ubuntu server. But the cnid_meta daemon will always try handle that for us. This will cause some hidden folders to show up in Ubuntu which are used to store all the metadata required to almost reproduce a HFS+ file system. But if you ever run into problems you can easily resolve these by creating sparse disk images on your server with the Disk Utility built into Mac OS X. Just create a new sparse disk image on your desktop, copy it to your mounted Ubuntu volume and mount the copied disk image by double clicking it. Now you can put all the files and libraries in it which depends on HFS+

13. Problems & Errors (WORK IN PROGRESS)

Problems with creating the backup disk image
If time Machine says “The backup disk image could not be created” during the first backup attempt you can do the following to avoid this problem and some others (backup fail due to permissions):

In short, you have to create the backup disk image on your Desktop and copy it to your mounted Time Machine volume. But Time Machine creates a unique filename for the disk image and we can find out this name with a little trick:

First open up the Console from your Applications > Utilities folder and open the Time Machine preferences. In Time Machine preferences set your backup volume back to none. After that reselect your mounted Time Machine volume. The counter should start and Time Machine’s big button will change to on. When the backup tries to start and fail have a look at your Console (Click All Messages in the sidepane). There should be a line telling you the name of the disk image:

Creating disk image /Volumes/TimeMachine/computername_0014e3856bd0.sparsebundle  The computername should be the name you have assigned to your Mac. Now just click on that line and hit command + C to copy the message.

Now Open Disk Utility from your Applications > Utilities folder and click on New Image in the toolbar. Now just hit Command + V in the Save As field and remove everything before the name of the sparsebundle.

Update: The secret number Time Machine adds to your computer name is nothing more than the MAC address of the network device that your Mac uses to talk to your server, as Simulacrum and Todd found out in the comments. As he points out you can open up Network Utility and under the info tab you’ll find a field called “Hardware Address”. Just remove the colons and you have the number Time Machine adds to the disk image name.

In the field volume name write Backup of computername. Now FIRST chose sparse bundle disk image as the image format and THEN adjust the volume size to the size of your internal harddrive (minimum, choose more if you like) afterwards. Remember that Disk Utility won’t let you make a bigger image file as you have physical hard drive space available if you don’t chose sparse bundle disk image as the image format first.

Select your Desktop as destination and click create. After the creation is finished drag the created disk image to your mounted Time Machine volume (you can delete the disk image on your desktop when copy is finished). Finally go to Time Machine preferences and start your backup again and everything should work as expected now. After the first backup (this can take a long time depending on your harddrive) you can unmount your Time Machine volume and the next time Time Machine starts it will grab and mount the sparse bundle disk image automatically (with “Backup of computername” as the volume name).

For people still having problems with error messages 
On Mac OS X have a look in the Console app inside your Utilities folder. Click on “All Messages” in the left sidebar, try to connect to your Ubuntu box in Finder and see the messages in Console. These are the “real” error messages which normally will lead you to a solution. 
  
-5014 error 
Some people have problems when connecting to an AFP share and get a -5014 error. As J5 pointed out in the comments you have to delete the hidden .AppleDB folders on your Ubuntu box and restart netatalk afterwards:

sudo /etc/init.d/netatalk restart
 
Time Machine Full System Restore
In case of a full system restore you would have to boot your Mac from the Mac OS X installation DVD (the one delivered with your Mac) by pressing the c key during boot. Your Mac will start with a minimal UI where you have a Utilities section in the top menu bar. There you’ll find “Restore from a Time Machine Backup” but it won’t find your network share with your Time Machine backup. Luckily Dmitry Nedospasov found a way to manage this by simply mounting your Time Machine network share with the Terminal (which you can find under Utilities in the menu bar too) by utilizing the following syntax (shamelessly copied from Dmitry):

mount -t afp afp://username:password@hostname/ShareName /Volumes/ShareMount
 
Replace everything instead of /Volumes with your matching names. You can test if your network share was properly mounted by doing

ls /Volumes
 
which outputs the content of the Volumes folder and you should see your network share.
Now you can close the Terminal and select “Restore from Time Machine Backup” from the Utilities entry in the menu bar and select your mounted Time Machine backup and thats it. Oh, needless to say: a gigabit ethernet connection will speed things up dramatically even compared to (draft)n-WLAN. 

Netatalk backup disk reaching maximum capacity
As Seron pointed out in the comments, there is some discussion on the net regarding problems with Netatalk and TimeMachine when the backup disk reaches maximum capacity. This is due to missing support for the AFP commands FPSyncDir aka commands 78 and 78 in Netatalk. As a commenter in an ArsTechnica forum says: “As soon as your backup volume will reach max capacity, it will self destruct because of it.”

In the forum you’ll also find some links to various patches to avoid problems with that. If you have tested such patch please post your experiences in the comments for this article.

14. Translations Of The Original Article
 The following articles are direct translations of Matthias' original article are below, but some of them are slightly modified or simplified. Remember that the authors/translators are responsible for the content.

German: Ubuntu + Apple Fileserver + TimeMachine on kde4.de
French: Tuto: Comment créer votre serveur Time Capsule sous Debian on the Blog of Pierre de la Celle

 Fantastic original article, credit to Matthias Kretschmann's:

http://www.kremalicious.com/2008/06/ubuntu-as-mac-file-server-and-time-machine-volume/