Arq: Amazon S3 Backups Made Easy for Mac

I have tried a number of online backup solutions – among them Mozy, Carbonite, JungleDisk, MobileMe’s iDisk, and CrashPlan. This article is not about those products, but I can tell you that all of them let me down in one way or another.

I hit problems with one of them when disaster struck and I found that I couldn’t actually use my backup files; another was terribly, unusably slow and gave little control over when backups were run; another kept my MacBook’s fan’s running all the time, because the backup app was leaking memory all over the place.

Despite all these disappointments, I do feel that I need an off-site backup (along with my Time Machine and SuperDuper! backups) – it’s an extra line of protection that helps me feel more secure – so I’ve kept looking for a solution that’ll do the best possible job.

For the last few months, I’ve been using Dropbox and, although it’s not expressly designed for the purpose of backup, it just works, and I’ve been very happy with the service. My only complaint is that Dropbox is a little expensive for my needs – after all, I’m only currently using 12% of the 50GB my $10/month buys.

I recently came across Haystack Software’s Arq, and I’m thinking this may be a very good option for keeping online backups running smoothly and seamlessly. Join me after the break for a walkthrough of its features.

Getting Going

Once you’ve downloaded Arq and moved it into your Applications folder, double-click to run the app, and you’ll see this window:

Setting Up

Setting Up

Because Arq uses Amazon’s S3 data storage (as does Dropbox), you’ll need to enter your account details if you already have one, or go through the process of creating one. Clicking on the ‘Sign Up for Amazon S3′ button will open your browser and you’ll need to log into or create an Amazon account to proceed. If you already have an S3 account, then you can enter your details, or click on ‘Retrieve Amazon S3 Keys’ if you don’t have a handy note of them.

Once you have your keys, enter these in the boxes and select your Bucket Region – this is quite important, because costs differ in different places in the world. It also controls where your backups are stored, which might impact backup times – as well as perhaps stirring some xenophobic fears or nationalistic pride.

Click ‘Continue’, and Arq will quickly verify your account details, and then moves you on to a particularly welcome stage in the setup process: setting your budget.

Set Your Budget & Backup Type

Your Budget

Your Budget

That’s a really nice feature: you get to specify what you spend on storing your backup each month, rather than being constrained by a service’s limitations, or surprised by a huge bill at the end of the month. Bear in mind, though, that Amazon charges transfer costs separately, so your bills are likely to be higher than the figure you set here – especially in the first month, when you’re running the initial upload of all your files.

Set your own budget and click through, and you now get to set your encryption password. Your password is stored in your Mac’s Keychain and isn’t transmitted to anyone else, so your data’s good and secure in transit and on Amazon’s servers. But do heed the warning screen that shows next:

Lose at Your Own Risk

Lose at Your Own Risk

And, finally, you can select either to back up your entire Home folder, or to manually select which folders and files are backed up. If you choose to select folders, you’re then presented with this window:

Add a Folder

Add a Folder

Click on the + at bottom-left (or click the ‘Add a Folder for Backup’ button on the right) to select folders or files from a familiar Finder window.

Arq in Use

Once you’ve added things, they’ll show in a list in the left panel, and as you select each one, you’ll see its current backup status:

Backup Progress

Backup Progress

You can use a filter system to exclude any items within folders – another nice feature that gives you greater control over your backup.

Arq Agent

Arq Agent

Once you’ve set things up, you can close Arq, and the Arq Agent will continue to run in the background and look after your backup with no user intervention needed. You can opt to have it display an icon in the menubar:

The light in the tiny Ark’s windows flashes from left to right to let you know when data is being transferred. If you prefer to save on menubar space, though, you can choose to hide the icon (in the Preferences menu, under the ‘Advanced’ tab), though I find it useful to be able to see when there’s something happening, as well as being able to open the app from the icon.

How Arq Works

Arq uses the same backup schedule as Time Machine: it saves hourly backups for the past 24 hours, daily backups for the past month, and weekly backups until the amount of data stored in S3 reaches the budget you’ve set in the Preferences. Backups run automatically a few times in an hour – whenever a file is changed, Arq receives a notification from OS X, but it won’t actually upload the changed file until 20 minutes after the last backup. Because this is incremental backup, after your initial upload, which can take some time, subsequent updates are quick. I have found Arq’s uploads to be significantly faster than my experience of other online backups.

If you ever want to restore a file from your backup, you need simply click on the disclosure icon to the left of the folder name, and you’ll see a list of all the versions of the folder held in your backup. Then select the file you want restored from the right-hand panel’s list, and click ‘Restore’:

Backed Up Files

Backed Up Files

The file will be downloaded and filed in a folder titled ‘Restored by Arq’ in the original location – usefully, once that process is done, you’re presented with a dialogue that lets you click to go to the restored file:

Your Restored File

Your Restored File

(Some implementation of Quick Look within the app window itself might have been nice here, though it’s really not necessary – and perhaps it’s not even technically possible.)

Conclusion

That’s all you need to know: once Arq is up and running, you really can forget about it. It’s been stable in my experience, and is lightweight and unobtrusive.

Arq doesn’t have the instant synchronisation/backup of any changed files that DropBox or Mozy offers, but things are updated a few times in each hour, which might in general be enough for most users. It would be easy to come up with a workflow that combined a free Dropbox account for sharing files between machines and taking advantage of that instant backup magic (say a ‘Current’ folder for whatever documents you’re working on at the moment), while using Arq for your main backup.

I love Dropbox for its elegance and simplicity, but actually those are exactly the qualities I like in Arq too. Having paid out for the last couple of months for a Dropbox Pro account, I think I’m going to downgrade at the end of the month and give Arq a real test as my main online backup option. It’s looking good enough to become the app I rely on to be the final, crucial part of my backup strategy.


Summary

Arq is the only online Amazon S3 backup product built for the Mac and only the Mac. So it has the look and feel you expect from a Mac application.

9
  • Jack

    Nice Review. Is there any free apps could provide those kind of backup function in the cloud?

    I have mobileme, but the upload speed is really really slow for such a data backup.

  • todd miller

    I’ve always debated on using an online service mainly because of the speed / time it takes to do the actual backup. What was your take on that?

    • Ian Tromp

      Of all the online backup options I’ve tried, I think Arq has been the quickest in uploading. YMMV of course.

    • http://www.webmaster-source.com redwall_hp

      The biggest problem is the initial upload. I’ve been using Dropbox for a subset of my files, and it’s very fast. Text files and images upload in seconds, right as I change them.

      I imagine that the Time Machine-type schedule of Arq would make for reasonably-sized upload batches, unless you’re backing up video or something.

  • Ayush saran

    Why is ur entire backup just 12% of 50 GB i.e. 6GB

    That seems quite small. Mine are huge. What are u backing up , assuming ur selectively avoiding all the system files

  • http://www.robertgordonphotography.com Robert

    Can you go into detail about which other solutions you’ve used that let you down? I am currently using Jungledisk and wondering what you experienced with them that you disliked. I like having the ability to access the files remotely if need be where this service doesn’t appear to have that option.

    • http://www.haystacksoftware.com/arq/ Stefan Reitshamer

      Hi Robert,
      (I’m the author of Arq).
      You can access the files from another computer by installing Arq on that other computer.
      There’s also an open-source command-line utility for restoring/downloading from your Arq backups: http://sreitshamer.github.com/arq_restore/

      Also, I’m planning an iPhone/iPad app for viewing your Arq backups, emailing files from the backups, and making a public-readable, decrypted copy of files/folders in your backups so you can email a link to friends.

      - Stefan

  • Bichuf

    I think all these solutions, except Backup3 (Apple), have what to me is a huge problem and that is that the naming scheme in the cloud is nothing like what your filenames are on your Mac so you will always need the original app to know what is what. On Backup to MobileMe the names and dir structure are the same so no problem plucking the file you need or searching for it fron iDisk. All these apps take ages to backup the first time. From there on it is just a question of backing up only those files that change so that is much faster unless you continually add huge new files.

    • http://www.haystacksoftware.com/arq/ Stefan Reitshamer

      Hi Bichuf,
      (I’m the author of Arq.)
      There are 2 reasons Arq doesn’t store the files in S3 with the same names as the files on your disk:

      1. Arq de-duplicates the data. It stores each file using the SHA-1 hash (a unique ID) of its contents as the filename as part of its system of avoiding duplication of data in S3.
      2. The overhead of S3 PUT requests becomes a big performance problem when uploading many small files, so Arq combines small files together into “pack files” to dramatically improve network performance.

      - Stefan

      • http://davidappleyard.net David Appleyard

        That’s smart. Explains why the app works so quickly!

  • http://vacuumstoragebags.net Francis

    Still looking around for that perfect backup solution and it seems that Arq might be it. Was considering between Jungledisk and Arq. Jungledisk can backup to rackspace cloud as well though.

    • heebs

      I have tried both Arq and Jungledisk. Jungledisk reports a faster upload bit rate but it may only be due to bad math. Terminal shows both have equal throughput.

  • Bob Smith

    Could you post a more direct comparison of this to other S3 backup apps like Jungle Disk? You don’t say why you thought Jungle Disk inadequate.

    • aaron

      inadequate cos junkledisk is no bring moneys in exshange fo review. junkldisk is no lion. lion is no junkledik

      • Whaaaa?

        You off your meds bro?

  • Tom

    I’ll stick with Dropbox.
    If you get near 50GB JungleDisk and Arq are more expensive and not nearly as simple.

    • http://www.haystacksoftware.com/arq/ Stefan Reitshamer

      Hi Tom,

      With Arq, your data go into your own Amazon S3 account. S3′s pricing is unfortunately complicated, but it’s $.10/GB per month plus data transfer charges ($.10/GB upload after June 30). So if you back up 50GB, that’s $5 storage + $5 transfer =$10 for the first month. If you change, say, 10% of your data every month, that’s $5 storage + $.50 transfer = $5.50/month for subsequent months. If you only back up, say, 40GB, ongoing cost is $4.40/month.

      Dropbox is $9.99/month for up to 50GB.

      Also, I tried to make Arq extremely simple. Just drag a folder onto Arq and it gets backed up. Here’s a video demo: http://www.haystacksoftware.com/arq/intro_movie.php

      - Stefan

      • http://www.haystacksoftware.com/arq/ Stefan Reitshamer

        Actually Amazon just extended their free upload pricing until November 1, 2010! (It was to expire June 30):

        http://aws.amazon.com/s3/pricing/

      • Aaron Lynch

        I think I’m sold Stefan. Thanks for your attention to this tread!

      • aaron

        suck its goig to when free app to do this come out. no cash for u. people mov one and u nobody. thanks for solution but problems when ask for moneys u now obligate to keep working on same app so stupid to ask money just do people favor why not u want want want want gimme gimme gimme will blow up face back fire u just should ask donation not require cash flow

  • http://www.suki.co.il Suki

    I’ve been working with Arq for like a week now. and up until now it looks awesome!
    really really easy to use and definitely fast.

    The only thing i’m missing is a way to track all the latest files activities.
    It will give me much more secure feeling to know what are the exact files thats been backed up.

    • http://www.haystacksoftware.com/arq/ Stefan Reitshamer

      I agree, it would be comforting to see that info. Highlighting which files/folders were added/modified/deleted in each backup version is definitely on the product roadmap. Thanks

      • aaron

        u have dlc like ps3?

  • http://www.twitter.com/j_holtslander Jay

    Canadians… Worth noting….
    Ontario’s Freedom of Information and Protection of Privacy Acts (FIPPA) doesn’t allow you to store sensitive information in jurisdictions that permit secret warrants, like those mandated by the US Patriot Act.
    Because many backup tools like Arq use Amazon’s servers it falls under the jurisdiction of the Patriot act. If you need your data to be protected by CANADA’s privacy laws instead of the USA’s, you need to use a backup service that doesn’t use Amazon’s S3. I use: http://www.nerdsbackup.com/jasonh1234 which keeps my data in Canada (Which is a work mandated requirement for me.)

    I told my accountant about this same thing and her face went white. LOL

    • http://www.haystacksoftware.com/arq/ Stefan Reitshamer

      I’m not sure what any government agency would do with your Arq backups if they got the data from Amazon, since it’s encrypted with AES-256. As long as you used a reasonably long encryption password, cracking the encryption is “far, far beyond our capabilities of computation” according to Bruce Schneier: http://www.schneier.com/blog/archives/2009/07/new_attack_on_a.html

      Is it still considered sensitive information if it’s encrypted in that way? Also, who’s bound by FIPPA?

  • Pingback: Online backups « Hyperlands.co.uk

  • http://liquidsoapdispenser.com/ Rob

    Stefan, I’m trying to understand security issues as they relate to online backup. Seems there’s security during file transfer and then security of stored files. My understanding is that the stored files are as secure as the S3 service (and the strength of the password one is using), but I’m not clear on security of data during file transfers. I see other services use 256-bit AES encryption for data transfer (which appears to be very secure), can you tell me what security is used by Arq for data transfer? And if it’s less than 256-bit AES, can you explain why you think that is safe enough?

    I really am intrigued by your software (and using S3 to host my offsite backup), but am also a little nervous to try a solution offered by a small publisher, when it comes to all the data on my computer. Hard to find any reviews on the product (other than those by one or two users), can you point me to any?
    Thanks!

  • David C.

    Nerds Backup? Are you serious? $2/GB? Your entry sounds borderline trolling mate.

  • Matthew Nicholas

    Arq is outstanding in it’s attention to detail. The author has clearly spent a great deal of time and very hard work dotting every ‘I’ and crossing every ‘T’; with that level of commitment and enthusiasm you only get the very best of results.

    I’ll be sticking to Arq for the long haul, that’s for sure! One more very happy customer here…!

    • jim

      Is that why if I leave my MBP on overnight (8+ hours), with InsomniaX and Caffeine running, I get 4% progress on a 20 GB backup? Then as soon as I restart the Arq agent I then see at least some kind of progress?

  • Matt M.

    I had an awful experience with Backblaze. It doubly hurt because I was so enthusiastic about their service. Long technical story of my process, but in short I was doing my best to maintain the same backup state while replacing a defective hard drive in my Mac. Total blunder. I was left having to re-upload my entire set of data again. Never again do I want my backup to be tied to a single machine. Perhaps Backblaze technical support could have helped me, but at that point I knew I needed to move away from the $5/mo all-you-can-eat-but-only-a-single-machine style services.

    I fortunately found Arq and it’s now busy pushing my 165 gig backup to S3.

    • jacob

      Wait til you have to restore that chunk of data! Adventure!

  • Robert

    Is there an easy way in Arq to exclude a folder under the users/ folder from being backed up? I know its easy to include folders, but I want to exclude just one folder with a bunch of large files I don’t need backed up.

  • Richard Testani

    I just found this and am super impressed with the simplicity, and quick upload speed Arq provides. I have been using CrashPlan which is nice because it’s unlimited storage but it’s dog slow and seems to disconnect often from the servers.

    Good job Stefan!

  • http://www.giving-to-charity.info Elissa Heimerdinger

    I really do like the design this blog did you make it yourself?

    • aaron

      so glad i found this wanted to backup wifes files found arq now its great thank u so much i love it yes dog slow but errors well

  • Pingback: 7 Alternatives for Your Off-Site Backup | Mac.AppStorm

  • Pingback: Freelance Mac App Bundle | SSTechBlog

  • Chris

    Is it possible to use arq with any other type of online storage? I have 2 web servers at a datacenter and I would like to be able to use them to store my online backups.

    • aaron

      is it possible yes to do this i can tell u this does not work well if u try to ucan do nothing

  • http://artesanosdelparquecentenario.blogspot.com Artesanos

    the development, time to market and staff cost savings from re-using open source code

    • aaron

      my own site for this too. its appstorm. u can promote crap to sell here. liek cragslit

  • http://shawnkoppenhoefer.com/ Shawn Koppenhoefer (@DrKdev on Twitter)

    I’ve been using CrashPlan for a year or so and am happy. Should I switch?… and why?… I store The raw files from photo-shoots on an external Drobo (4-8GB per shoot) so the unlimited storage (of CP) is nice. But I want ‘the best’ and often council other photogs on their IT needs; So what might be the args. in favour of switching over to Arq? (@Stefan:r u still reading?).

    cheers.
    /shawn

theatre-aglow
theatre-aglow
theatre-aglow
theatre-aglow