Description: Challenges of Backup and Recovery can expand beyond just the size of a Data Store.

When trying to maintain a Backup and Recovery solution for any large environment, you can encounter many unexpected challenges. One of the least expected problems to run into by most Administrators is file count. A frequent story we see at DS3 DataVaulting is Administrators that have backups taking an exceedingly large amount of time to complete for not a large amount of data. Often times, we discover the problem ends up boiling down to file count.

As directories become wider and deeper, the time to enumerate increases until you reach a point that the enumeration saturates the backup process. Administrators who are attempting to backup millions of small files will find themselves faced with a daunting task of simply trying to determine where to divide the groups of files to achieve the most efficient backups.

Thanks to advances in modern processing power, the ability to run multiple threads at a solution makes the handling of high-file count scenarios easier. Running a process with upwards of 50 or 60 threads against a directory structure that is both wide and deep provides impressive results without sacrificing the CPU to the utilization gods.

The challenges of backing up and restoring millions of small files is still an issue for many legacy, single-threaded, applications. For those applications, the Administrator must remember that throwing horsepower (CPU, Memory) at the problem will only create severely underutilized systems. There are better ways out there!