Backups. A technical and practical breakdown for photographers.

Backups are central to our whole philosophy of capturing and preserving memories.  One of the core elements of what we do.

This article is aimed at photographers, who are wanting to understand how to effectively backup and look after wedding photographs.  If you are a Bride or Groom, refer to our page about backing up your wedding photographs

Whatever you do, it must be a tried and tested part of your work routine. You need to weave your backup routine into your regular workflow. 

If you want to have bulletproof backups, this is what you need to think about  

  • Avoid compromising your memory cards
  • Assess your pre-backup weak and danger points
  • Don’t risk data until you have backups
  • Back up to at least three devices
  • Back up to different types of device
  • Back up to at least three locations
  • Verify each backup
  • Routinely move backups on to new media and devices
  • Back up your software too
  • Back up the ancillary files generated by your software
  • Do a disaster test
  • Consider (near) real-time backup for work in progress

Before you shoot

Backing up, and preserving data starts before you shoot.  You need to think about this carefully.  The things we do routinely make a difference in data security.  You want to avoid conflicts, corrupt data, and losing data.

Part of our pre-shoot routine, when we are cleaning cameras, resetting menus, synchronising clocks, is to format the memory cards in-camera.  We then avoid potential future file name conflicts by setting our cameras to prefix the filenames for each shoot with a unique ID and camera ID. 

Formatting memory cards
Using your camera to format a memory card before a job, ensures you only have files from that job on it, which avoids confusion about what is or is not on the memory card later.

We shoot with multiple cameras.  We have no assurance that the file system on our D5’s is exactly the same as the file system our D850’s or D500’s, so we don’t swap memory cards between cameras.  Firmware updates change things too, so don’t assume for example that two Nikon D850, are exactly the same. 

By using the camera, as opposed to your computer, to format a memory card, you are assured that the file system on the memory card is optimal, just as the camera that formatted the memory card wants it to be.

Unique file names
The file name conflict problem is real. 

What will you do when your computer asks you if you want to replace 1234_DSC.nef with 1234_DSC.nef.  Most of the time you will be smart and avoid the trap of overwriting a file from one shoot with a file from another.

However to avoid this totally, adopt file naming that ensures total uniqueness. 

For example, 1234_AF1.nef, and 1234_FD2.nef. 

In our office, the above file names would indicate that they are from the shoot with job number: AF, and FD, and also that they were from cameras: 1 and 2

In this way, you don’t end up with two cameras on the same shoot giving you the same file name, neither do you ever get the same file names between different shoots. 

If you adopt this approach to naming, you have a much lower chance of accidentally overwriting a file with a different one of the same name.  You also get the bonus that your file names are referencing a job number, which makes finding things a whole lot easier too.

The shoot

If you are shooting with one camera, on to one memory card, you are at a lot more risk to losing data, compared to the shooter that is shooting with multiple cameras, with dual memory card slots.

On the day of the shoot, you are at high risk of losing data. 

Threats are both physical and from corruption.  Let’s look at these.

Physical risk
Physical risk includes theft, misplacing your camera, damage, and loss of your camera and damage to memory cards. Weddings take us to all sorts of places – beaches, the sea, uneven ground, cliffs, boats, and busses.  We all use lots of gear, and it’s all shiny and expensive looking.  When we drive home, we are tired, often eating late at night in districts we are unsure of.

One remedy to all of this is to shoot to two memory cards and separate them once they are out of your camera. 

Ideally one memory card is with you, and the other memory card is kept in your car or with your assistant.  When you enter a new area of risk (i.e. you get on a boat), swap a new set of memory cards in.  Remember to keep the pair of memory cards that came out separated. 

Have a gear list, and count it out and count it into your car.  Pack your car so it doesn’t scream photographer.  When you go for your late-night meal on the way home, one set of memory cards will be on you, and the other in the car.

Next time you are on a shoot, have a hard think about security and where you are most at risk of losing data.

Avoiding corruption
Corrupting data is a whole different issue.  Shooting to two memory cards will mitigate against a single memory card failure.  Shooting to two memory cards won’t mitigate against the camera not writing data properly.

What often will corrupt your memory cards or files is when you delete images on the day.  The file system on a camera is a bit more primitive than the one found on a PC, and it doesn’t cope as well as a PC at reorganising files after deletions. 

When you delete and reshoot to a memory card, you are at a very high risk of corrupting a whole batch of images.  DO NOT delete images on your camera.

Only remove and insert memory cards when the camera is turned off.  And always do this in pairs if you have dual memory card slots. 

Do not swap memory cards between cameras on a shoot.

Do not turn your camera off, or interrupt it whilst it is saving images to memory cards.  Be assured that turning off your camera or swapping out a battery whilst it is writing a burst of images to the memory card will certainly increase the chances of corrupting a few files if not the whole file system on your memory cards.

Back at the office

Your first priority is to safely back up your memory cards.  Work with only one whole set (if you use cameras that shoot to two memory cards). 

Have a routine.  We literally lay them out on the right side of the desk, and move processed memory cards to the left side.  Same routine, every time.  When we are done, we do a sanity check.  The advantage of our naming system, we can ID the camera in the file name, so omissions are easy to spot.

Using a good high-quality memory card reader, back up the images from the whole shoot to your workstation, and then make a second backup to a different device.  At this point, you have at least three backups (the memory card + two devices).

Only at this stage ought you even start to think about viewing, ingesting or doing anything else with the image files.  This is why we don’t entertain same-day edits.

Backing up – the nuts and bolts

The rules

  • Backups must be verified
  • Backups will Ideally in three different locations
  • Backups will ideally on three different types of media
  • Backups are revisited periodically and maintained and moved to new media

Other considerations

  • What to back up
  • Realtime work in progress backups

Verifying backups
OK this is an interesting one.  A few years ago, we noticed that on one of our devices, a Synology NAS, was occasionally causing corruption to a small percentage of files that were over about 80mb.  We investigated this, and found out that the cause was the default MTU network setting on the NAS network adapter.  We then altered the setting and tested the fix, problem solved.

This posed us a quandary.  Since we only rarely go back to our NAS to grab the odd file back from, how would we actually know, until it was too late, that 1 out of 5000 files, were corrupted?  what about future corruption of files on the way out? 

The good news was that we didn’t lose any data, as we had original files and three other good backups elsewhere.  However, this issue threw up a number of questions.  We did need to do something.

Bit perfect file backups
This is why we implemented a 100% bit-perfect copy/verification routine.

To achieve this, we now use the same software that forensic teams use to accurately copy files: TeraCopy.  We use TeraCopy for every single file movement, including copying from memory cards, into our editing software, and out to our final backups.

TeraCopy, used properly, copies the file, and then compares the copied file to the original.  It flags issues, and gives you a report. 

Yes, this doubles the time to do a copy, but you have the peace of mind of knowing that the copied file is 100% correct and uncorrupted. 

TeraCopy integrates into your PC operating system.  If you do timed batch backups, then you can be sure that these backed up files are verified too.

So, how many files get corrupted? 
The great thing about verifying every file movement is that you know straight away if a file gets corrupted.  Because we know about it immediately, we can copy it again and re-verify it again.
 
Within the same PC, we get about one file a year that is corrupted during copy. 

From memory card to PC, we never had had a corrupted file flagged.  The only exception was from one faulty memory card reader, which we binned. 

Across our network, we get about 1 file out of 5000 that corrupt. 

Into the cloud, we have so far have had no corrupt files.

Here is the thing, occasionally files absolutely do get corrupted when you move them, not many, not enough to notice immediately.  But using software that checks, and lets you sort them out means, you can resolve these issues before they become a problem.

Verifying to mitigate against long term corruption
Hard drives fail, small packets of data are lost.  Using TeraCopy allows you, in the future to re-verify that a file is not corrupted too. TeraCopy can save data (a hash file) alongside the file that can be used to re-check a file at a later date, without looking at the original file, after the event. 

Backups to at least three locations
You are ideally aiming to have backups in your office on several devices, and additionally backups offsite too.  The plan here is to mitigate against failure, theft, fire, flood, earthquake, etc.

That’s why you want both on and offsite backup, and preferably backup out of the country too i.e. the cloud.

This is what we do: In our office, we have two identical workstations that mirror each other, each with 36TB of storage only drives. In addition, out of the building we have one Synology NAS also with 36TB of storage.  If the building burnt down or flooded, the NAS would be intact.

Additionally, we store RAW files to two different cloud storage systems.  Finished files go down the same route, and additionally to Google Drive and MS One Drive too.

Backups are both triggered manually, and using timed windows batch files, and of course every file movement is verified. 

Backups to three different types of media
Can you currently read an image file from a CD?  Unlikely. 

Mix your backup strategy to include different types media.  This lower the risk of simultaneous failure, for example, the same make and age hard drives in multiple machines failing on similar dates  You also mitigate against the progress of new media obsoleting old media.

Ideally you back up to three different types of media i.e. SSD drive, Hard Drive and Cloud.

Long term considerations
Remember the CD issue – the files that are unreadable due to scratches, or just that we don’t have CD drives in PC’s anymore.  Well you need to mitigate against this too.  This is a commitment to routinely revisit archives, and move them to whatever the new media that is available in the future.

Pop a note in your diary, and in your down season, routinely migrate your archive.

Back-up software and catalogues
Back in the day, we used Word Perfect for word-processing.  Can we open one of these files (incidentally on a floppy disk) now – No, and Yes.

No, because we don’t use WordPerfect anymore, neither can we read from floppy disks on modern computers. 

Yes, because we migrated the files to Word, and saved them on a CD, then DVD, then hard drive.

But the real answer is yes, and yes, because there is another route.  Save both the software with the archive.  We do this for old versions of Photoshop, Lightroom and Capture One.  If you want to adopt this strategy, you must save everything you need to boot the software back up again. 

Additionally, when you decommission a PC, keep the software on it, and slip in a paper document with passwords, usernames for software and notes about anything you need to know, into the case. 

This approach does work, when we have had to go back 15 years, to retrieve something that was not work-related, we managed it, although we hated remembering how to work Windows 3.1!

When you decommission a PC, think about how you would boot it back up again, extract something from it.  How are you physically going to store it and look after it? How are you going to remember the password etc.

Watch out for file transitions
One day JPG’s, Tiff’s, PSD’s, CR2 and NEF files won’t be current.  Keep an eye on the trends and transform files to new formats before you are struggling to read them.

Ancillary file backup
Do you back up just your RAW files, just the finished files or something else?

We mainly work with CaptureOne sessions and sidecar files generated by PhotoMechanic, so for us, it is an easy choice, we back up the whole folder – and we have the lot.  Database, finished files, RAW files, and all the little files that make things work. 

If you work with Lightroom, or some other RAW processor, or DAM, you might need to back up quite a lot more.

You will need to test this.  In isolation, you need to be able to go to the backup, and reopen the catalogue, file, session, application, and edit images and export images etc..

Disaster testing
Imagine what would happen if a set of data disappeared.  Could you still operate and recover?  Take the time to test this, this is where you find your points of weakness, and, lets you resolve them before you have a real problem.

You need to do this.  A backup is not a backup unless you have tested it.  The whole point is to know that when your main workstation, or NAS, or other backup falls over, you can be back up and running quickly, with no loss of data.  This is easy to test; you just have to do it.

Realtime backup for work in progress
What would you feel like if just as you are exporting images from a wedding, your computer crashed, never to be revived again.  Pretty awful, your last few days of editing have just been lost.  This is where a smart routine of backing up work in progress is helpful.

We have three variations of nearly realtime work in progress backup. Timed backups manually triggered backups, and automatically backing up when specific software closes.

Once an hour, each of our workstations, does a timed back up of its whole work in progress drive,  over to the other workstation in the office, and vice versa.  Once a day this is mirrored off to our work in progress drive on our NAS.  Since the TeraCopy software can be set to only move new or changed files, this pretty much is done as a process in the background, and has little practical impact on working.  In this way, we have more than an hour of edits that are not backed up for the wedding we are currently working on.  We use the task scheduler on the PC to trigger a windows batch file, that kicks off the process. 

Secondly, we can manually kick off a backup, if we just did something significant (imported a load of files for example)

Thirdly, we use the Computer Management tool to spot that we have closed Capture One, which triggers a batch file that does a similar backup.

Implementation
You need to plan this.  Draw a map of which files are being moved from and to.  We did a map and a spreadsheet.  This is a totally logical task, and as such, you need to mentally test that you are doing the right thing, making the right movements. 

When you implement it, especially if you are making batch files, you need to do dry runs with a handful of files first.  Make sure that what you wanted to happen, did happen, before you mass move folders worth of customers files

If you go down an automated backup route, make it a once a month task to doublecheck that things are running as expected.  Operating systems update, firewalls interfere, things change, computers get turned off, other people in your office change things etc.

Summary

Backups are something you really need to think about.  From shoot to delivery, you’re in charge of ensuring your client’s memories are looked after.  This is simply a systems and workflow thing, something you need to sort out, do, test and periodically reassess.

Our circumstances, and the way we work drive the choices that work well for our studio.  Your circumstances will be different.  However, the principles are the same.  Mitigate against loss and corruption, and have multiple backups. We all loose data, and encounter corruption, but with a good plan in place, you can mitigate the impact.

 

 

 

Leave a Reply

Close Menu