Re: [Hampshire] Grouping Files for backup

Top Page

Reply to this message
Author: Simon Reap via Hampshire
To: hampshire
CC: Simon Reap
Subject: Re: [Hampshire] Grouping Files for backup
To use multi-volume, do something like:

    cd to directory with files
    tar cvf /dir/out1.tar -M -L 52428800 *
(where /dir/ is a place to hold the 50GB tar files, and 52428800 is
50*1024*1024, the number of 1k blocks in 50GB)

It will write the first 50GB to /dir/out1.tar, then prompt with:
  Prepare volume #2 for '/dir/out1.tar' and hit return:
Type in
    n /dir/out2.tar
(that's 'n space /dir' etc) and press return. The next 50GB goes to
/dir/out2.tar and it prompts with
    Prepare volume #3 for '/dir/out2.tar' and hit return:
Type in
    n /dir/out3.tar
and press return. Continue for 37 more files!

(When I started writing this, I didn't realise how much effort it was!).

Another solution could be:

    tar cvf - * | split -b 52428800 -d - /dir/out

This will write /dir/out00, /dir/out01, /dir/out02 and so on to
/dir/out40. The tar writes to stdout and pipes to split. The '-b' splits
the file into that number of bytes per file, and the '-d' uses numeric
suffices on the filenames (otherwise it would be aa, ab, ac etc to b0).
The '-' reads from stdin, and the '/dir/out' is the prefix for the
output filenames.

This splits the files at a given number of bytes, so all parts need to
be connected together to extract data. The benefit of the 'tar -M'
version is that each part is a valid tar file (though some files may be
split across two tar files).


On 29/05/2019 16:12, James Courtier-Dutton via Hampshire wrote:
> Hi,
>  Tar already has such a feature. A long time ago, it would have needed
> to spread data across backup tapes.
> look at the man page for tag for something like --multi-volume.
> Kind Regards
> James
> On Wed, 29 May 2019 at 12:12, Rob Malpass via Hampshire
> <hampshire@??? <>>
> wrote:
>     Hi all

>     I’ve never been a bash script expert (to say the least!) but I
>     need a clever script (I assume that’s the easiest way but please
>     say if not) that can turn 120 files (amounting to 2Tb of data)
>     into 40 files of <=50Gb for an archiving project I’m about to
>     embark on.   Can anyone help / give me some pointers?

>     I can guarantee none of the files are over 50Gb in size. So it’s a
>     question of something like:

>     Create an empty archive (call it n.tar.gz)

>     Repeat

>     Find next (uncompressed) file

>     Compress it to tmp.tar.gz

>     If size of tmp.tar.gz < (50Gb – current size of n.tar.gz) then

>                 Append tmp.tar.gz to n.tar.gz

>     Else

>                 Close n.tar.gz

>     Alert me so I can write n.tar.gz to MD

>                 Create next empty archive ((n+1).tar.gz)

>     Add tmp.tar.gz to (n+1).tar.gz

>     End If

>     Until all files have been compressed

>     The end product should be 40 files called n.tar.gz (where 0<n<41).

>     Any (constructive!) ideas very welcome.

>     Cheers

>     -- 
>     Please post to: Hampshire@???
>     <>
>     Web Interface:
>     LUG URL:
>     --------------------------------------------------------------

> <> 
>     Virus-free. 
> <> 

> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Please post to: Hampshire@???
Web Interface: