{% load i18n %}

{% trans "Import Arches JSON" %}

{% blocktrans %}Bulk import of Arches JSON business data, built for large loads. Upload a .json Arches business-data export, a .jsonl file with one resource object per line, or a .zip of .json files (upload a .jsonl file on its own). Files are not imported: file values must point at files already in storage, and anything else in a zip is ignored.{% endblocktrans %}

{% blocktrans %}Prefer .jsonl for very large loads: it streams, whereas a .json file is parsed whole. To convert an existing export:{% endblocktrans %} 

jq -c '.business_data.resources[]' export.json > export.jsonl

{% blocktrans %}The file is read twice: once to validate every resource, once to write. If validation finds any error nothing is written, and the problems appear in the standard error report. If writing fails, everything the load wrote is removed again and its message says where it failed. A load replacing existing resources keeps what it replaced before the failure; run the same file again to finish it.{% endblocktrans %}

{% blocktrans %}Tick Overwrite existing resources to replace resources whose ids are already in the database. Each resource in the file completely replaces the existing one: any of its tiles not in the file are deleted, while the resource itself and links to it from other resources are kept. To update only part of a resource, use the Tile Excel importer. Left unticked, importing a resource id that is already present will fail the load.{% endblocktrans %}

{% blocktrans %}Undo import removes the resources a load created. Stored files are kept, since the import never stored them. A load that replaced existing resources cannot be undone, and neither can a load whose resources have been changed since or are now linked to from other resources.{% endblocktrans %}

{% blocktrans %}Untick Index the data after import to write the tiles without indexing them. The data is saved, but it will not appear in search, and the resources will have no display names, until you run python manage.py es index_database --recalculate-descriptors. Useful when loading several files in a row: skip indexing on each and index once at the end.{% endblocktrans %}

{% trans "Importing from the command line" %}

{% blocktrans %}The same importer runs from the command line, which is the better route for very large loads: it reports progress as it validates, and it is the only path that can index across several processes.{% endblocktrans %} 

python manage.py etl arches-json-importer -s /path/to/export.jsonl

{% trans "Flags:" %}

  • -s, --source — {% trans "the .json, .jsonl or .zip file to load. Required." %}
  • -ow, --overwrite — {% trans "replace resources that already exist instead of failing on them." %}
  • --no-index — {% trans "write the data but skip indexing; reindex later with index_database." %}
  • -mp, --use_multiprocessing — {% trans "index across several processes after loading." %}
  • -mxp N, --max_subprocesses N — {% trans "how many processes to index with. Defaults to half the CPU count." %}
  • -c, --config — {% trans "a JSON file of options. The flags above win over it." %}

{% blocktrans %}A large overnight load, overwriting existing resources and indexing on eight processes:{% endblocktrans %} 

python manage.py etl arches-json-importer -s export.jsonl -ow -mp -mxp 8

{% blocktrans %}Multiprocessed indexing is deliberately not offered in the panel above. A load larger than the module's size limit is handed to a Celery worker, whose processes cannot themselves start child processes, so a pool there would leave the data written but never indexed. Use the command line when you want indexing spread across cores.{% endblocktrans %}