cat brain.log | less

Getting it down on `paper`

Periodic Tarballs: Find and Awk

I have a gang of directories. There’s a new directory each day! Each directory stores about 20k files. Over time, the performance of the directory hierarchy degrades, and it’s best for me to tar up old stuff.

In linux-land, there’s always more than one way of doing things… even if it’s not the right way. Today, we’ll explore find piped to awk piped to the shell.

touch . ; find . -type d -maxdepth 1 -mtime +60 | sort |
awk -F"/" '{print "tar czf " $2 ".tar.gz --remove-files ./" $2 "; rmdir " $2 }' | sh

Basically, make sure the current direction has been recently modified, so it’s not returned by the find command.  Then, find all sub-directories in the current directory — don’t go recursive. Sort for a predictable sequence of operations. Extract the file name portion (lots of ways to do this). Generate a shell command tar czf {{dirname}}.tar.gz --remove-files ./{{dirname}}; rmdir {{dirname}} to build the tarball and remove the empty directory when done. Finally, we just tell sh to execute it all… one directory at a time. Bam!

Update: Further experiments with awk and several “oops” events later, this variation of the code has helped prevent instances where an empty string leads to the entire directory being tar.gzipped and removed.

find . -maxdepth 1 -type d | sort | 
awk -F"/" '{ if ($2 !~ /^$/) print "tar czf " $2 ".tar.gz ./" $2 }' | sh


No comments so far.

(comments are closed)