When One AI Was Not Enough for My Family-Tree Research

Researching a family tree eventually changes character.

At the beginning, progress can be surprisingly fast. A name appears on a certificate, that certificate identifies two parents, a marriage record adds another generation, and each discovery produces several new directions to investigate.

For some time, much of my work followed this pattern. I would find a promising person in an online archive, open the relevant record, download the image and give it to an artificial-intelligence tool. The AI would examine the old handwriting, produce a verbatim—or ipsis litteris—transcription of the page, translate it into English and help me identify the information that belonged in my family-tree database.

This is already an extraordinary capability. Many of these documents were written in the eighteenth or nineteenth century, in handwriting that is difficult for a modern reader to decipher. They may be in Italian, German, Portuguese, Latin or another language, and the vocabulary and administrative conventions may be as unfamiliar as the script itself.

What might once have required access to a specialist can now be attempted at home, one document at a time.

But eventually I reached a different stage of the research.

The problem was no longer reading the document

Once the easily identifiable people and records had been found, the bottleneck moved.

The problem was no longer: How can I read this document?

It became: Which document should I read?

I knew that relevant information might still exist, but I no longer knew precisely where it was. A record could be somewhere within a book containing hundreds of photographed pages. The spelling of a surname might have changed. A person might appear as a witness, parent, spouse or declarant rather than as the principal subject of the record. An index might be incomplete, damaged, difficult to interpret or entirely absent.

Italy has made an enormous quantity of historical civil-registration material available through the Ministry of Culture’s Portale Antenati. Depending on the archive and municipality, its collections include records from the Napoleonic period, the Restoration and the later Italian civil-registration system. Coverage is not identical everywhere, but the portal provides access to vast numbers of digitised registers that previously required physical archival research.

Having the books online, however, does not mean that every person mentioned within them can be found through a normal text search. Many pages are essentially photographs of handwritten documents. Until somebody—or something—reads them, their contents remain largely invisible to conventional search tools.

Digitisation makes an archive accessible, but it does not automatically make every handwritten fact within that archive searchable.

An idea in the morning

One morning, I began thinking about the problem from a programmer’s perspective.

If I could not identify the correct page in advance, why should I continue opening pages manually?

The archive already contained the images. Its search results identified the available registers, and each register had an internal structure that the web viewer used to navigate between pages. A computer should therefore be able to identify the books returned by a search, retrieve every page from those books and store the images locally with predictable filenames.

Once the material existed on my computer, an AI agent with access to local files could work through it systematically. It could transcribe the handwriting, translate the contents and search the resulting text for people who might be connected to my family.

The process I imagined was approximately this:

Search expression
       │
       ▼
Registry search results
       │
       ▼
Extract every book ARK
       │
       ▼
Read each IIIF manifest
       │
       ▼
Download every page image
       │
       ▼
Transcribe and translate
       │
       ▼
Identify possible relatives
       │
       ▼
Prepare family-tree fields

I asked Grok to investigate the archive website, understand how it worked and write the necessary code.

I expected this to be the beginning of a programming exercise. Normally, even a relatively small scraper or download utility requires investigation. One has to inspect requests, understand identifiers, handle pagination, choose filenames, manage errors and test the result. Websites are not always designed to make automated retrieval straightforward, and small assumptions often fail when the program encounters an unexpected page.

Instead, Grok produced the first script, I ran it—and it worked.

No corrections. No debugging. No repeated conversation about an error message. No manual adjustment of selectors or URL formats.

It worked on the first attempt.

The solution then developed into two scripts with separate responsibilities.

The first, antenati_list_registry_arks.sh, accepts an Antenati registry-search URL. It moves through the search-result pages, identifies every book returned by the expression, removes duplicate entries and writes both a simple URL list and a tab-separated catalogue containing the available labels and identifiers.

The second, antenati_download_books.sh, accepts a single register URL, several URLs or the complete list produced by the first script. It resolves the International Image Interoperability Framework—or IIIF—manifest for each book, obtains the pages in their correct viewer order and downloads the images into a local directory.

Within roughly ten minutes, I had moved from an idea to a practical research pipeline. Had I written it myself, the same small project could easily have consumed two, three or four days, especially when accounting for investigation, testing and the inevitable distractions of ordinary life.

The remarkable part was not simply that AI wrote some code. It was that a passing idea became a working research tool before the idea had time to disappear.

Using different AI agents for different jobs

This experiment reinforced something I have increasingly noticed: there is no reason to expect one AI system to be the best at everything.

At the moment, in my own practical use, Grok is producing code of a quality and directness that ChatGPT does not consistently match. That is not a universal benchmark or a declaration that one system is always superior. It is my observation from the work I am doing today. In this case, Grok understood the technical objective and generated functioning scripts immediately.

ChatGPT, however, remains substantially better for another part of my workflow.

Once the documents have been downloaded, I need more than code. I need the contents of the pages to be understood in context. Names must be compared with people already present in my family tree. Dates, places, occupations, witnesses and relationships must be extracted. Alternative spellings need to be recognised. Apparent contradictions must be highlighted rather than silently resolved. Evidence from different documents must be connected and then presented in a form that I can enter into MacFamilyTree.

That work requires information processing, organisation and explanation. In my experience, ChatGPT handles it much better than Grok.

So I am not choosing one product and declaring it the winner. I am treating them as specialised agents:

• Grok investigates the archive and produces the acquisition tools.
• ChatGPT transcribes, translates, compares and organises the historical information.
• I define the research question, judge the evidence and decide what belongs in the family tree.

The best AI workflow may not involve finding one perfect assistant. It may involve combining several imperfect but differently capable assistants under human direction.

From individual pages to entire collections

The addition of the registry-listing script changes the scale of the idea considerably.

The original objective was to download every image from one known book. That was already useful, but I still had to identify the relevant register manually.

Now the first script can begin with an Antenati search expression and identify all the books returned by it. Its default example searches the Cairo Montenotte series and finds hundreds of registers. The resulting URL file can be given directly to the downloader, which can then process the books one after another.

This means that the pipeline no longer begins with a page or even with a particular book. It can begin with a place, archival series or other search expression.

The second script also does more than repeatedly press the viewer’s next-page button. The visible viewer URLs do not contain a simple sequential parameter such as ?page=2. Individual leaves have their own identifiers. The script therefore finds the book’s IIIF manifest, reads the ordered list of canvases and derives the appropriate image-service URL for each page.

It supports both IIIF Presentation 2 and Presentation 3 structures, configurable image sizes, selected page ranges, delays between requests and the resumption of partially completed downloads. Existing images above a minimum size are not downloaded again.

These details matter. A script that works for one page is a demonstration. A script that can catalogue books, preserve page order, tolerate interruptions and resume work is the beginning of a usable research process.

The two scripts

These are the two scripts generated for this workflow. They are shown exactly as supplied. The scripts are intended to run under Bash—such as within WSL on Windows—and require Python 3.9 or later. Their own comments explain the available arguments, environment variables and the possibility of receiving an HTTP 403 response when they are executed from a datacentre address rather than a normal home connection.

1. Listing every book returned by an Antenati search

antenati_list_registry_arks.sh reads the registry search, extracts every unit ARK and creates the URL list consumed by the downloader.

#!/usr/bin/env bash
# Read an Antenati registry-search page and list EVERY book ARK.
#
# The index does not include the short leaf token (/wXxJre6). That token is
# page 1 inside the viewer. The unit ARK is enough for antenati_download_books.sh:
#   https://antenati.cultura.gov.it/ark:/12657/an_ua1434603
#
# Default search: Cairo Montenotte, serie 1434193 (316 registers).
#
# Run in WSL on your home network. Datacentre IPs get 403.
#
# Usage:
#   chmod +x antenati_list_registry_arks.sh
#   ./antenati_list_registry_arks.sh
#   ./antenati_list_registry_arks.sh 'https://antenati.cultura.gov.it/search-registry/?serie=1434193&descrizione=Cairo%20Montenotte&s_size=100'
#   ./antenati_list_registry_arks.sh --out ~/FamilyDocuments/Antenati_Downloads
#   ./antenati_list_registry_arks.sh --resolve   # also open each book for page-1 URL
#
# Writes in --out (default: current directory):
#   antenati_registry_urls.txt    one unit ARK per line  ← feed this to the downloader
#   antenati_registry_books.tsv   unit, label, url
#
# Then download every book:
#   ./antenati_download_books.sh antenati_registry_urls.txt

set -euo pipefail

SEARCH_DEFAULT='https://antenati.cultura.gov.it/search-registry/?serie=1434193&descrizione=Cairo%20Montenotte&s_size=100'
OUT_DIR="${ANTENATI_OUT:-$PWD}"
SLEEP_S="${ANTENATI_SLEEP:-0.35}"
RESOLVE=0
SEARCH_URL="$SEARCH_DEFAULT"

while [[ $# -gt 0 ]]; do
  case "$1" in
    --resolve) RESOLVE=1; shift ;;
    --no-resolve) RESOLVE=0; shift ;;
    --out) OUT_DIR="$2"; shift 2 ;;
    -h|--help)
      sed -n '2,32p' "$0"
      exit 0
      ;;
    http*) SEARCH_URL="$1"; shift ;;
    *) echo "Unknown argument: $1" >&2; exit 1 ;;
  esac
done

mkdir -p "$OUT_DIR"

python3 - "$SEARCH_URL" "$OUT_DIR" "$SLEEP_S" "$RESOLVE" <<'PY'
import re, sys, time, urllib.error, urllib.parse, urllib.request
from html.parser import HTMLParser
from pathlib import Path

search_url, out_dir, sleep_s, resolve = sys.argv[1], Path(sys.argv[2]), float(sys.argv[3]), sys.argv[4] == "1"

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "it-IT,it;q=0.9,en;q=0.8",
    "Referer": "https://antenati.cultura.gov.it/",
}

UNIT_RE = re.compile(r"/ark:/12657/(an_ua\d+)\b")
VIEWER_RE = re.compile(r"https?://antenati\.cultura\.gov\.it/ark:/12657/(an_ua\d+)/([A-Za-z0-9_-]{4,12})")
COUNT_RE = re.compile(r"(\d[\d.]*)\s+risultat", re.I)


def fetch(url, timeout=60):
    req = urllib.request.Request(url, headers=HEADERS)
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return resp.read(), resp.geturl()


def page_url(url, page, size=100):
    parts = urllib.parse.urlsplit(url)
    q = urllib.parse.parse_qs(parts.query, keep_blank_values=True)
    q["s_page"] = [str(page)]
    q.setdefault("s_size", [str(size)])
    query = urllib.parse.urlencode({k: v[-1] for k, v in q.items()})
    return urllib.parse.urlunsplit((parts.scheme, parts.netloc, parts.path, query, parts.fragment))


class CardParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.links = []
        self._href = None
        self._buf = []
        self._depth = 0

    def handle_starttag(self, tag, attrs):
        attrs = dict(attrs)
        href = attrs.get("href", "")
        if tag == "a" and "/ark:/12657/an_ua" in href:
            self._href = href
            self._buf = []
            self._depth = 1
        elif self._depth:
            self._depth += 1

    def handle_endtag(self, tag):
        if not self._depth:
            return
        self._depth -= 1
        if self._depth == 0 and self._href:
            text = re.sub(r"\s+", " ", "".join(self._buf)).strip()
            self.links.append((self._href, text))
            self._href = None

    def handle_data(self, data):
        if self._depth:
            self._buf.append(data)


def add_unit(seen, href, text=""):
    m = UNIT_RE.search(href)
    if not m:
        return False
    uid = m.group(1)
    if uid in seen:
        if text and len(text) > len(seen[uid]["label"]):
            seen[uid]["label"] = text
        return False
    seen[uid] = {
        "unit": uid,
        "unit_url": f"https://antenati.cultura.gov.it/ark:/12657/{uid}",
        "label": text or uid,
        "viewer_url": "",
        "leaf": "",
    }
    return True


def harvest():
    seen = {}
    announced = None
    page = 1
    stale = 0
    while page <= 50:
        url = page_url(search_url, page)
        print(f"Index page {page}: {url}", flush=True)
        try:
            raw, _ = fetch(url)
        except urllib.error.HTTPError as e:
            raise SystemExit(
                f"HTTP {e.code} on the search page.\n"
                "Run this script in WSL on your home connection.\n"
                f"{url}"
            ) from e
        html = raw.decode("utf-8", "replace")
        if "403" in html[:800] and "Forbidden" in html[:800]:
            raise SystemExit("403 Forbidden. Run this in WSL at home, not from a server.")

        if announced is None:
            cm = COUNT_RE.search(html)
            if cm:
                announced = int(cm.group(1).replace(".", ""))
                print(f"  site says {announced} results", flush=True)

        parser = CardParser()
        parser.feed(html)
        new = 0
        for href, text in parser.links:
            if add_unit(seen, href, text):
                new += 1
        # catch any ark the card parser missed
        for uid in UNIT_RE.findall(html):
            if add_unit(seen, f"/ark:/12657/{uid}"):
                new += 1

        print(f"  new books this page: {new}; running total: {len(seen)}", flush=True)
        if announced and len(seen) >= announced:
            break
        if new == 0:
            stale += 1
            if stale >= 2:
                break
        else:
            stale = 0
        page += 1
        time.sleep(sleep_s)
    return list(seen.values()), announced


def resolve_viewer(row):
    try:
        raw, final = fetch(row["unit_url"])
    except urllib.error.HTTPError as e:
        row["viewer_url"] = f"ERROR_{e.code}"
        return
    html = raw.decode("utf-8", "replace")
    uid = row["unit"]
    m = VIEWER_RE.search(final) or VIEWER_RE.search(html)
    if m and m.group(1) == uid:
        row["leaf"] = m.group(2)
        row["viewer_url"] = f"https://antenati.cultura.gov.it/ark:/12657/{uid}/{m.group(2)}"
    else:
        row["viewer_url"] = row["unit_url"]


rows, announced = harvest()
rows.sort(key=lambda r: (r["label"], r["unit"]))

if resolve:
    print(f"Opening {len(rows)} unit pages for the page-1 token…", flush=True)
    for i, row in enumerate(rows, 1):
        resolve_viewer(row)
        print(f"  [{i}/{len(rows)}] {row['unit']} {row['viewer_url']}", flush=True)
        time.sleep(sleep_s)
else:
    for row in rows:
        row["viewer_url"] = row["unit_url"]

tsv = out_dir / "antenati_registry_books.tsv"
urls = out_dir / "antenati_registry_urls.txt"

with tsv.open("w", encoding="utf-8") as fh:
    fh.write("unit\tlabel\tunit_url\tleaf\tviewer_url\n")
    for row in rows:
        fh.write(
            f"{row['unit']}\t{row['label'].replace(chr(9), ' ')}\t"
            f"{row['unit_url']}\t{row['leaf']}\t{row['viewer_url']}\n"
        )

with urls.open("w", encoding="utf-8") as fh:
    fh.write("# Cairo Montenotte unit ARKs from the registry search index\n")
    fh.write(f"# source: {search_url}\n")
    for row in rows:
        fh.write(row["unit_url"] + "\n")

print()
print(f"Books listed: {len(rows)}" + (f" (site announced {announced})" if announced else ""))
print(f"URL list:     {urls}")
print(f"Table:        {tsv}")
print("Download with:")
print(f"  ./antenati_download_books.sh {urls}")
PY

2. Downloading every image from the identified books

antenati_download_books.sh resolves the IIIF manifest and downloads every page in the order presented by the archive’s viewer.

#!/usr/bin/env bash
# Download every page of a Portale Antenati register, in viewer order.
#
# The on-screen controls "1 di 226 • pag. 1" plus the next-page arrow do NOT
# expose a URL such as ?page=2. Each leaf has a random id (LpGavnM, …).
# This script therefore:
#   1. opens the entry-point URL you can see in the browser
#   2. reads the IIIF manifest for that book
#   3. downloads page 1, then page 2, … until the last page
#
# Run in WSL on your home connection. Datacentre IPs often get 403 / WAF.
#
# Usage:
#   chmod +x antenati_download_books.sh
#   ./antenati_download_books.sh
#   ./antenati_download_books.sh 'https://antenati.cultura.gov.it/ark:/12657/an_ua1434283/LpGavnM'
#   ./antenati_download_books.sh urls.txt
#
# If the gallery page is WAF-blocked, open the book in your browser, copy the
# IIIF manifest link from the left panel
#   https://dam-antenati.cultura.gov.it/antenati/containers/.../manifest
# and pass that URL instead.
#
# Environment:
#   ANTENATI_OUT      output root   (default: ~/FamilyDocuments/Antenati_Downloads)
#   ANTENATI_SIZE     max pixels    (default: 2500; 0 = full resolution)
#   ANTENATI_SLEEP    seconds       (default: 0.6 between pages)
#   ANTENATI_FIRST    first page    (default: 1)
#   ANTENATI_LAST     last page     (default: all)

set -euo pipefail

OUT_ROOT="${ANTENATI_OUT:-$HOME/FamilyDocuments/Antenati_Downloads}"
MAX_PX="${ANTENATI_SIZE:-2500}"
SLEEP_S="${ANTENATI_SLEEP:-0.6}"
FIRST="${ANTENATI_FIRST:-1}"
LAST="${ANTENATI_LAST:-}"

need() {
  command -v "$1" >/dev/null 2>&1 || {
    echo "Missing command: $1" >&2
    exit 1
  }
}

ensure_python() {
  need python3
  python3 - <<'PY'
import sys
if sys.version_info < (3, 9):
    sys.exit("Python 3.9+ required")
import urllib.request
PY
}

collect_urls() {
  if [[ $# -eq 0 ]]; then
    printf '%s\n' 'https://antenati.cultura.gov.it/ark:/12657/an_ua1434283/LpGavnM'
    return
  fi
  if [[ $# -eq 1 && -f "$1" ]]; then
    grep -vE '^\s*(#|$)' "$1"
    return
  fi
  printf '%s\n' "$@"
}

download_book() {
  local url="$1"
  echo
  echo "=== entry: $url"
  FIRST="$FIRST" LAST="$LAST" MAX_PX="$MAX_PX" SLEEP_S="$SLEEP_S" OUT_ROOT="$OUT_ROOT" \
    python3 - "$url" <<'PY'
import json, os, re, sys, time, urllib.error, urllib.request
from pathlib import Path

start_url = sys.argv[1].strip()
out_root = Path(os.environ["OUT_ROOT"]).expanduser()
max_px = int(os.environ.get("MAX_PX", "2500"))
sleep_s = float(os.environ.get("SLEEP_S", "0.6"))
first = max(1, int(os.environ.get("FIRST", "1") or 1))
last_env = os.environ.get("LAST", "").strip()
last = int(last_env) if last_env else None

UA = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
)
HEADERS = {
    "User-Agent": UA,
    "Accept": "text/html,application/json;q=0.9,*/*;q=0.8",
    "Accept-Language": "it-IT,it;q=0.9,en;q=0.8",
    "Referer": "https://antenati.cultura.gov.it/",
}


def fetch(url, timeout=60):
    req = urllib.request.Request(url, headers=HEADERS)
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return resp.read(), resp.headers.get_content_type(), resp.geturl()


def looks_like_manifest(url: str) -> bool:
    path = url.rstrip("/")
    return path.endswith("/manifest") or path.endswith(".json")


def extract_manifest_url(html: str) -> str | None:
    m = re.search(r"manifestId\s*[:=]\s*['\"](https?://[^'\"]+)['\"]", html)
    if m:
        return m.group(1)
    m = re.search(r"https?://dam-antenati\.cultura\.gov\.it/[^'\"\s]+/manifest", html)
    if m:
        return m.group(0)
    m = re.search(r"https?://[^'\"\s]+/iiif/[^'\"\s]+/manifest", html)
    if m:
        return m.group(0)
    return None


def load_manifest(url: str) -> tuple[dict, str]:
    if looks_like_manifest(url):
        raw, _, final = fetch(url)
        return json.loads(raw.decode("utf-8", "replace")), final
    try:
        raw, ctype, final = fetch(url)
    except urllib.error.HTTPError as e:
        raise SystemExit(
            f"Gallery page blocked ({e.code}). Open the book in your browser, "
            "copy the IIIF manifest URL from the left panel, and pass that instead."
        ) from e
    text = raw.decode("utf-8", "replace")
    if "403" in text and "Forbidden" in text:
        raise SystemExit(
            "Gallery page returned 403. Use the IIIF manifest URL from the left panel."
        )
    manifest_url = extract_manifest_url(text)
    if not manifest_url:
        raise SystemExit(
            "Could not find a IIIF manifest on that page. "
            "Copy the manifest link from the Antenati left panel and retry."
        )
    print(f"Manifest: {manifest_url}")
    raw, _, final = fetch(manifest_url)
    return json.loads(raw.decode("utf-8", "replace")), final


def meta_value(manifest: dict, label: str) -> str:
    for item in manifest.get("metadata") or []:
        if str(item.get("label", "")).strip().lower() == label.lower():
            val = item.get("value", "")
            if isinstance(val, list):
                val = " ".join(str(v) for v in val)
            return re.sub(r"\s+", " ", str(val)).strip()
    return ""


def canvases_of(manifest: dict) -> list:
    if "items" in manifest and manifest.get("type") == "Manifest":
        return manifest.get("items") or []
    seqs = manifest.get("sequences") or []
    if seqs:
        return seqs[0].get("canvases") or []
    return []


def image_service_id(canvas: dict) -> str | None:
    # IIIF Presentation 2
    try:
        res = canvas["images"][0]["resource"]
        if isinstance(res.get("service"), dict) and res["service"].get("@id"):
            return res["service"]["@id"]
        rid = res.get("@id") or ""
        if "/full/" in rid:
            return rid.split("/full/")[0]
        if rid:
            return rid
    except (KeyError, IndexError, TypeError):
        pass
    # IIIF Presentation 3
    try:
        body = canvas["items"][0]["items"][0]["body"]
        if isinstance(body, list):
            body = body[0]
        service = body.get("service")
        if isinstance(service, list):
            service = service[0]
        if isinstance(service, dict):
            return service.get("id") or service.get("@id")
        return body.get("id") or body.get("@id")
    except (KeyError, IndexError, TypeError):
        return None


def rewrite_size(image_base: str) -> str:
    base = image_base.rstrip("/")
    size = "pct:100" if max_px <= 0 else f"!{max_px},{max_px}"
    if "/full/" in base:
        prefix = base.split("/full/", 1)[0]
        return f"{prefix}/full/{size}/0/default.jpg"
    if base.endswith((".jpg", ".jpeg", ".png")):
        return base
    return f"{base}/full/{size}/0/default.jpg"


def slug(text: str, fallback: str) -> str:
    text = re.sub(r"<[^>]+>", " ", text or "")
    text = re.sub(r"[^A-Za-z0-9]+", "-", text).strip("-").lower()
    return text[:80] or fallback


manifest, manifest_url = load_manifest(start_url)
canvases = canvases_of(manifest)
total = len(canvases)
if total == 0:
    raise SystemExit("Manifest contains no pages.")

end = total if last is None else min(last, total)
if first > end:
    raise SystemExit(f"Nothing to download: first={first} last={end} total={total}")

context = meta_value(manifest, "Contesto archivistico") or meta_value(manifest, "Title")
title = meta_value(manifest, "Titolo") or manifest.get("label") or ""
typology = meta_value(manifest, "Tipologia")
if isinstance(title, list):
    title = " ".join(str(x) for x in title)
ark = ""
m = re.search(r"an_ua(\d+)", start_url) or re.search(r"an_ua(\d+)", manifest_url)
if m:
    ark = m.group(1)

book_name = "-".join(
    part
    for part in (
        slug(context, "antenati"),
        slug(str(title), "registro"),
        slug(typology, "pagine"),
        ark or "book",
    )
    if part
)
book_dir = out_root / book_name
book_dir.mkdir(parents=True, exist_ok=True)

print(f"Book:    {context} | {title} | {typology}".strip(" |"))
print(f"Pages:   {total} (viewer: 1 di {total})")
print(f"Range:   {first} .. {end}")
print(f"Output:  {book_dir}")

ok = 0
failed = []
for i in range(first, end + 1):
    canvas = canvases[i - 1]
    service = image_service_id(canvas)
    if not service:
        failed.append(i)
        print(f"[{i}/{total}] no image URL — skipped")
        continue
    img_url = rewrite_size(service)
    dest = book_dir / f"page_{i:04d}.jpg"
    if dest.exists() and dest.stat().st_size > 2000:
        print(f"[{i}/{total}] exists {dest.name}")
        ok += 1
        continue
    try:
        data, _, _ = fetch(img_url, timeout=90)
        if len(data) < 500 or data[:15].lstrip().startswith(b"<") or data[:4] == b"403 ":
            # retry with pct:100 if sized request was blocked
            retry = rewrite_size(service.split("/full/")[0] if "/full/" in service else service)
            if max_px > 0:
                retry = service.rstrip("/")
                if not retry.endswith(".jpg"):
                    retry = f"{retry}/full/pct:100/0/default.jpg"
                data, _, _ = fetch(retry, timeout=90)
        dest.write_bytes(data)
        print(f"[{i}/{total}] saved {dest.name} ({len(data)} bytes)")
        ok += 1
    except Exception as e:
        failed.append(i)
        print(f"[{i}/{total}] FAIL {e}")
    time.sleep(sleep_s)

print(f"Done. {ok} pages saved, {len(failed)} failed, folder={book_dir}")
if failed:
    print("Failed pages: " + ",".join(str(n) for n in failed[:40]))
    sys.exit(2)
PY
}

main() {
  need curl
  ensure_python
  mkdir -p "$OUT_ROOT"

  echo "Output root: $OUT_ROOT"
  echo "Page size:   ${MAX_PX}px (0 = full)"
  echo "Pause:       ${SLEEP_S}s between pages"
  echo "Range:       ${FIRST} .. ${LAST:-end}"

  local n=0
  while IFS= read -r url; do
    url="${url//$'\r'/}"
    [[ -z "$url" ]] && continue
    download_book "$url"
    n=$((n + 1))
    sleep 2
  done < <(collect_urls "$@")

  echo
  echo "Finished $n book(s)."
  echo "Copy $OUT_ROOT into Google Drive and tell Grok the path."
}

main "$@"

Automation does not turn evidence into fact

This workflow changes the scale of what I can investigate, but it does not make the results infallible.

Historical handwriting recognition remains difficult. Ink fades, pages tear, writers abbreviate words, names acquire regional spellings and an AI can confidently misread a letter. A transcription that looks convincing is not automatically correct.

There is also a difference between ordinary OCR and what this work actually requires. OCR is often used as a broad term for converting an image into text, but historical cursive material is more accurately a handwriting-text-recognition problem. Even then, recognising the text is only the first stage. Establishing that the person in one document is the same person found elsewhere is genealogical analysis, not merely character recognition.

For that reason, the original image must remain attached to every conclusion. Page numbers, book identifiers and filenames need to be preserved. Uncertain readings must be marked as uncertain, and important discoveries should be checked against additional records wherever possible.

AI can accelerate the search and expose possible connections, but it does not remove the need for provenance, verification and human judgement.

A new scale of personal research

What impressed me most was not the novelty of any individual component. Scripts have downloaded image sequences for decades. OCR and handwriting recognition are established fields. Genealogists have always searched registers page by page, and archivists have spent entire careers reading historical handwriting.

The important change is the accessibility and speed with which these capabilities can now be assembled by one person.

I did not create a formal software project, recruit a team or spend several days learning the internal structure of an archival website. I described the desired outcome in ordinary language. One AI agent produced the acquisition tools. Another can help process the resulting local files and transform hundreds or thousands of difficult images into structured research material.

The role of the human has not disappeared. In fact, it becomes more important as the quantity of generated material increases. I still need to decide which archival collections are relevant, recognise implausible conclusions, compare independent evidence and avoid turning an attractive possibility into an invented ancestor.

But the balance of effort has changed dramatically.

Instead of spending days constructing the mechanism that allows me to begin the research, I can spend that time evaluating the historical evidence itself. The computer performs the repetitive traversal. The AI systems assist with acquisition, recognition, translation and organisation. I concentrate on meaning.

For a personal project such as a family tree, that is not a small improvement. It changes which questions are realistically possible to ask.

A collection containing hundreds of registers and many thousands of handwritten pages is no longer simply an archive I may someday have time to inspect manually. It can become a searchable body of evidence—imperfectly transcribed, certainly, and still requiring careful verification, but finally within practical reach.

And all of this began with an idea in the morning and code that, to my genuine surprise, worked on the first try.

AI Assistance Disclosure: This article is based on my original ideas, experience, analysis and conclusions. Artificial intelligence tools were used directly in the work described: Grok generated the archive-discovery and register-download scripts, while ChatGPT assisted with document-processing concepts and the editorial preparation of this article. AI was also used to help organise, expand and refine the text, clarify technical distinctions, and verify general information about the Italian archival portal.

Where relevant, factual references were checked against the official Portale Antenati and Italian Ministry of Culture resources. Historical transcriptions and AI-generated interpretations can contain errors and must be checked against the original archival images. The final selection, interpretation, opinions and conclusions presented here remain my own.