Load your own corpus

Archetype ships as an empty platform: the code knows how to search, view, and annotate manuscripts, but it holds no manuscripts of its own. This guide walks you through loading a corpus into a running instance — modelling the descriptive records, ingesting page images for IIIF, importing transcriptions (including HTR output), and finally building the Meilisearch indexes that make it all searchable. It assumes you already have the stack up (see the Run it locally) and can run just recipes against the backend container.

1. Model the corpus

The domain lives in api/apps/manuscripts, scribes, symbols_structure, annotations, and publications. The manuscript spine is a chain of records, each a Django model:

• HistoricalItem — the intellectual object (a charter, a codex), with CatalogueNumber, HistoricalItemDescription, and a date assessment.
• CurrentItem — the physical shelfmark, held by a Repository.
• ItemPart — a part of a historical item as it survives in a current item (the join between the two, and what most search results key on).
• ItemImage — one photographed surface (folio/page) of an item part; its image is an IIIFField.
• ImageText — a Transcription or Translation attached to an ItemImage, stored as TEI in content, with an editorial status (Draft/Review/Live/Reviewed). One row per (item_image, type).

Palaeographic layers sit alongside: Scribe → Hand (bound to an ItemPart), the Character/Allograph/Component/Feature taxonomy in symbols_structure, and Graph in annotations — a region on an image (annotation_type of image, text, or editorial).

Create the descriptive records through the backoffice UI, the Django admin, or just shell. Run migrations first if this is a fresh database:

just migrate

See Data model for the full field reference.

2. Ingest images for IIIF

ItemImage.image is an IIIFField holding a path relative to the media root (storage/media/; field default upload_to="historical_items"). The media root is bind-mounted into the SIPI image server, which serves IIIF Image API tiles: at /sipi/images in the dev stack (api/compose.yaml), and from MEDIA_HOST_PATH (default ./storage/media) at /sipi/images/sipi in infrastructure/compose.yaml, where nginx also serves it at /media/. Point the backend at SIPI with the IIIF_HOST env var (default http://localhost:8182/); the frontend reads NEXT_PUBLIC_IIIF_UPSTREAM — see Configuration.

Superusers can upload images from the backoffice. Uploads go up in chunks to /api/v1/uploads/sessions/; a Celery task then assembles the file, converts it to JP2 under uploads/<folder>/, checks that SIPI can serve a tile, and creates the ItemImage. In production, run just setup-upload-storage once first (see Production deployment), and reap abandoned sessions with python manage.py cleanup_stale_uploads. Alternatively, drop files into the media directory and set each ItemImage.image to the stored path; SIPI derives tiles on demand, with no pre-tiling step.

3. Import transcriptions

Transcriptions are TEI XML in ImageText.content. To ingest HTR output, POST PAGE-XML or ALTO as a superuser to the import endpoint on ImageTextManagementViewSet:

POST /api/v1/manuscripts/management/image-texts/import-htr/
Content-Type: application/json

{ "item_image": 42, "format": "page", "type": "Transcription",
  "language": "la", "xml": "<PcGts>…</PcGts>" }

format is "page" or "alto". The parser (apps/manuscripts/services/htr) turns each text line into a <seg type="line">, materialises a text Graph for every line that carries a polygon, and embeds a corresp="#gid-N" back-reference. It returns the new draft image-text id; a 409 means that (image, type) already exists.

If you are migrating a legacy DigiPal-style data-dpt corpus, the manuscripts app ships round-trip commands (run --dry-run first, then --apply). Migration 0027 embeds the graph-side text↔region links while migrate runs, so the by-hand steps start after it:

just bash   # then, inside the api container:
python manage.py migrate_imagetext_to_tei --apply   # data-dpt HTML → TEI
python manage.py verify_tei                          # well-formedness gate
python manage.py check_text_links                    # audit text→Graph refs

4. Build the search indexes

Nothing is searchable until the Meilisearch indexes exist and are populated. Create them once, then reindex from the database:

just setup-search-indexes        # create every index
just sync-all-search-indexes     # populate all of them
just sync-search-index item-parts  # or reindex just one

The registry in apps/search/registry.py is the single source of truth; valid index segments are item-parts, item-images, scribes, hands, graphs, texts, clauses, people, and places. Re-run the relevant sync-search-index after any bulk data change; for a large import, set SEARCH_AUTO_SYNC=false while it runs, then reindex. If searches 500 with index_not_found, you skipped setup-search-indexes. These recipes exist in both api/ (dev stack) and infrastructure/ (production, where just reindex runs both steps); run them from the stack you are loading.

Your corpus is now browsable, viewable, and searchable. For the query surface, see The API; to extend the model, see For contributors.