echo6-docs/runbooks/ia-download-mirror.md

Download & Mirror from Internet Archive

Procedures for downloading items, filtering by format/pattern, bulk downloading from collections, and mirroring entire collections via the ia CLI on pi-nas.

---

Prerequisites

• ia CLI installed on pi-nas (192.168.1.245) — v5.7.2
• Authenticated if downloading restricted items: ia configure
• Sufficient storage on pi-nas (check with df -h)
• Reference: ia-cli-reference.md for search/query syntax

---

1. Download a Single Item

An "item" is a logical unit on archive.org identified by its identifier (visible in the URL: archive.org/details/<identifier>).

# Download all files in an item to ./<identifier>/
ia download <identifier>

# Example
ia download prelinger_films

The default creates a directory named after the identifier containing all files (originals + derivatives).

Gate

Verify the download directory exists and has files:

ls -la <identifier>/

---

2. Filtered Downloads

By glob pattern

Download only files matching a shell glob pattern.

# Only PDFs
ia download <identifier> --glob="*.pdf"

# Only MP4 video files
ia download <identifier> --glob="*.mp4"

# Multiple patterns (pipe-separated)
ia download <identifier> --glob="*.pdf|*.epub"

With exclusions

Exclude patterns require --glob to also be set.

# All MP4s except low-quality variants
ia download <identifier> --glob="*.mp4" --exclude="*512kb*"

# All files except metadata/review XMLs
ia download <identifier> --glob="*" --exclude="*_meta.xml|*_reviews.xml|*_files.xml"

# Multiple exclusions
ia download <identifier> --glob="*.mp4" --exclude="*512kb*|*_thumb*"

By format name

Download files of a specific archive.org format (as shown by ia metadata --formats).

# Check available formats first
ia metadata <identifier> --formats

# Download only a specific format
ia download <identifier> --format="512Kb MPEG4"
ia download <identifier> --format="PDF"
ia download <identifier> --format="EPUB"

Note: --format is incompatible with --glob and --exclude. Use one approach or the other.

On-the-fly formats

Some formats (EPUB, MOBI, DAISY, MARCXML) are generated on demand.

ia download <identifier> --on-the-fly --format="EPUB"

---

3. Download Options

Control output location

# Download to a specific directory
ia download <identifier> --destdir=/mnt/archive/downloads/

# Flatten directory structure (no subdirectory per item)
ia download <identifier> --no-directories

Resume interrupted downloads

# Resume — skips files that already exist and match checksum
ia download <identifier> --checksum

# Checksum mode compares MD5 hashes — safe to re-run

Preserve timestamps

# Keep original timestamps from archive.org
ia download <identifier> --no-change-timestamp

Dry run

# See what would be downloaded without actually downloading
ia download <identifier> --dry-run

---

4. Bulk Download from Search Results

Pipe search results directly into download. This is the primary method for downloading multiple items.

Basic pattern

# Search → itemlist → download
ia search 'collection:prelinger mediatype:movies' --itemlist | \
  ia download --itemlist -

# The - tells ia download to read identifiers from stdin

With filters

# Download only PDFs from all items in a collection
ia search 'collection:arrl_qst' --itemlist | \
  ia download --itemlist - --glob="*.pdf"

# Download only MP3s from an audio collection
ia search 'collection:librivoxaudio' --itemlist | \
  ia download --itemlist - --glob="*.mp3"

With destination directory

# Download to a specific location
ia search 'collection:prelinger' --itemlist | \
  ia download --itemlist - --destdir=/mnt/archive/prelinger/

Save itemlist for reuse

When a search is large, save the itemlist first so you can resume without re-searching.

# Step 1: Save itemlist
ia search 'collection:prelinger mediatype:movies' --itemlist > prelinger-items.txt

# Step 2: Check count
wc -l prelinger-items.txt

# Step 3: Download from file
ia download --itemlist prelinger-items.txt --glob="*.mp4"

# Step 4: Resume if interrupted (just re-run with --checksum)
ia download --itemlist prelinger-items.txt --glob="*.mp4" --checksum

---

5. Bulk Download with GNU Parallel

For faster bulk downloads, use GNU Parallel for concurrent item downloads.

# Install parallel if not present
sudo apt install -y parallel

# Download 5 items concurrently
ia search 'collection:prelinger' --itemlist | \
  parallel -j5 'ia download {} --glob="*.mp4"'

# With destination directory
ia search 'collection:prelinger' --itemlist | \
  parallel -j5 'ia download {} --glob="*.mp4" --destdir=/mnt/archive/prelinger/'

# From saved itemlist
parallel -j5 'ia download {} --glob="*.pdf"' < items.txt

Caution: Be respectful of archive.org bandwidth. 3-5 concurrent downloads is reasonable. Higher parallelism may trigger rate limiting.

---

6. Mirror an Entire Collection

Mirroring means downloading everything and being able to re-run to pick up new additions.

Initial mirror

# Step 1: Create working directory
mkdir -p /mnt/archive/<collection-name>
cd /mnt/archive/<collection-name>

# Step 2: Generate itemlist
ia search 'collection:<collection-name>' --itemlist > itemlist.txt
echo "Found $(wc -l < itemlist.txt) items"

# Step 3: Download all items (adjust --glob as needed)
ia download --itemlist itemlist.txt --destdir=/mnt/archive/<collection-name>/

# Or with format filter
ia download --itemlist itemlist.txt --glob="*.pdf" --destdir=/mnt/archive/<collection-name>/

Update an existing mirror

Re-run the same commands. Use --checksum to skip already-downloaded files.

cd /mnt/archive/<collection-name>

# Refresh itemlist (new items since last run)
ia search 'collection:<collection-name>' --itemlist > itemlist-new.txt

# Download only new/changed files
ia download --itemlist itemlist-new.txt --checksum --destdir=/mnt/archive/<collection-name>/

Mirror with a script

For recurring mirrors, create a simple script:

#!/bin/bash
# mirror-collection.sh <collection-name> [glob-pattern]
COLLECTION="$1"
GLOB="${2:-*}"
DEST="/mnt/archive/$COLLECTION"

mkdir -p "$DEST"

echo "Refreshing itemlist for $COLLECTION..."
ia search "collection:$COLLECTION" --itemlist > "$DEST/itemlist.txt"
COUNT=$(wc -l < "$DEST/itemlist.txt")
echo "Found $COUNT items"

echo "Downloading (glob: $GLOB)..."
ia download --itemlist "$DEST/itemlist.txt" --glob="$GLOB" --checksum --destdir="$DEST/"

echo "Mirror complete: $DEST"

Usage:

chmod +x mirror-collection.sh
./mirror-collection.sh arrl_qst "*.pdf"
./mirror-collection.sh prelinger "*.mp4"

---

7. Practical Patterns

Download all PDFs from a collection

ia search 'collection:arrl_qst' --itemlist | \
  ia download --itemlist - --glob="*.pdf" --destdir=/mnt/archive/arrl-qst/

Download specific media types from a collection

# High-quality video only
ia search 'collection:prelinger' --itemlist | \
  ia download --itemlist - --format="MPEG4" --destdir=/mnt/archive/prelinger-video/

# Audio in MP3 format
ia search 'collection:librivoxaudio creator:"Mark Twain"' --itemlist | \
  ia download --itemlist - --glob="*64kb*.mp3" --destdir=/mnt/archive/twain-audio/

Download items matching a date range

ia search 'collection:arrl_qst date:[1950-01-01 TO 1959-12-31]' --itemlist | \
  ia download --itemlist - --glob="*.pdf" --destdir=/mnt/archive/arrl-1950s/

Download a single specific file from an item

# List files first
ia list <identifier>

# Download just one file
ia download <identifier> specific-file.pdf

Download and preserve directory structure

# Default behavior — each item gets its own subdirectory
ia download --itemlist items.txt --destdir=/mnt/archive/output/
# Result: /mnt/archive/output/<identifier1>/files...
#         /mnt/archive/output/<identifier2>/files...

Pipe a single file to stdout

# Stream a file without saving to disk
ia download <identifier> specific-file.pdf --stdout | less
ia download <identifier> data.json --stdout | jq .

---

8. Storage Planning

Before large downloads, estimate storage requirements.

# Count items in collection
ia search 'collection:<name>' --num-found

# Check a sample item's size
ia metadata <sample-identifier> | jq '[.files[].size | tonumber] | add / 1048576 | floor'
# Output in MB

# Check available storage on pi-nas
df -h /mnt/

Rule of thumb

• Text collections (PDFs, EPUBs): ~10-100 MB per item
• Audio collections: ~100 MB - 1 GB per item
• Video collections: ~1-10 GB per item
• Software archives: highly variable

---

Troubleshooting

Download hangs or stalls

# Kill and resume with checksum verification
# Ctrl+C to stop, then re-run with --checksum
ia download --itemlist items.txt --glob="*.pdf" --checksum

"Item not found" errors in bulk download

Some items in a collection may be restricted or taken down. These will fail individually but the batch continues. Check errors in output.

Disk full during bulk download

# Check what's using space
du -sh /mnt/archive/*/ | sort -rh | head -20

# Resume after freeing space — checksum mode skips completed files
ia download --itemlist items.txt --checksum

Rate limiting / 429 errors

Archive.org may throttle aggressive downloads.

• Reduce parallel jobs (if using GNU Parallel)
• Add delays between items: parallel -j2 --delay 5 'ia download {}' < items.txt
• Wait and retry later

Corrupt downloads

# Re-download with checksum verification — replaces corrupt files
ia download <identifier> --checksum

Permission denied on destination

# Ensure the download user owns the target directory
sudo chown -R $(whoami):$(whoami) /mnt/archive/

---

Checklist: Collection Mirror

[ ] Identify collection identifier on archive.org
[ ] Check available storage (df -h)
[ ] Estimate collection size (--num-found + sample item size)
[ ] Generate itemlist (ia search --itemlist > itemlist.txt)
[ ] Review itemlist count (wc -l itemlist.txt)
[ ] Start download with appropriate filters (--glob, --format)
[ ] Verify downloaded files exist and are non-zero
[ ] If interrupted, resume with --checksum
[ ] Record collection details in project notes

---

Last updated: 2026-02-14