Archiving, compression & transfer

tar, gzip, zip, scp, rsync, curl, wget.

Bundling files together, compressing them, and moving them between machines are everyday tasks — backups, deployments, log collection. The core tools are tar for bundling, gzip for compressing, and scp/rsync for transferring, plus curl/wget for downloading. They combine cleanly, and a few flag combinations cover almost everything.

tar — bundle files into one archive

tar packs many files and directories into a single archive file (a "tarball"), preserving their structure and permissions. The flags read like a sentence: c = create, x = extract, t = list, v = verbose (show the files), f = the archive filename, and z = also gzip-compress. So tar czf backup.tar.gz mydir/ creates a compressed archive, and tar xzf backup.tar.gz extracts it. Remember "czf to create, xzf to extract."

$ tar czf logs-backup.tar.gz /var/log/app/    # Create Zipped, to File
tar: Removing leading `/' from member names
$ ls -lh logs-backup.tar.gz
-rw-r--r-- 1 deploy deploy 2.4M Jul  3 10:30 logs-backup.tar.gz
$ tar tzf logs-backup.tar.gz | head -2         # Table (list) contents without extracting
var/log/app/app.log
var/log/app/error.log
$ tar xzf logs-backup.tar.gz -C /tmp/restore   # eXtract into a chosen directory (-C)

gzip, zip — compression

gzip compresses a single file in place (file.log becomes file.log.gz and the original is replaced); gunzip or gzip -d reverses it. It is what tar uses under the z flag. zip creates the cross-platform .zip archives Windows and macOS understand, bundling and compressing in one step — useful when sending files to someone not on Linux.

$ gzip big.log            # big.log -> big.log.gz (original replaced)
$ ls
big.log.gz
$ gunzip big.log.gz       # back to big.log
$ zip -r site.zip public/ # a cross-platform .zip of a directory (recursive)

scp and rsync — copy between machines

scp copies files over SSH between hosts — scp file.txt user@host:/path/ pushes a file to a server, and reversing the arguments pulls one down. rsync does the same but smarter: it only transfers the differences, so re-syncing a large directory after a small change is fast, and it can mirror directories exactly. rsync -avz (archive, verbose, compressed) is the everyday form for syncing a folder to a server.

$ scp config.yml deploy@web-01:/opt/app/    # push a file to a server over SSH
config.yml                          100%  512    1.2MB/s   00:00
$ scp deploy@web-01:/var/log/app.log .      # pull a file down (note the reversed order)
$ rsync -avz ./site/ deploy@web-01:/var/www/   # sync a directory (only changed files transfer)
sending incremental file list
index.html
sent 1,204 bytes  received 35 bytes  ...

curl and wget — download from the web

Both fetch things over HTTP(S). wget is the simple downloader — wget URL saves the file. curl is the swiss-army tool for talking to web services: by default it prints the response, -O saves to a file, -I fetches just the headers, and it can send any HTTP method with data, which makes it the go-to for testing an API from the command line.

$ wget https://example.com/app.tar.gz        # download and save the file
$ curl -s https://api.internal/health        # fetch and print a response
{"status":"ok"}
$ curl -I https://example.com                 # just the headers
HTTP/2 200
content-type: text/html
$ curl -sS -X POST -d '{"n":1}' -H 'Content-Type: application/json' https://api.internal/items