CSV import¶

Determining import complexity¶

To quickly gauge the complexity of CSV data you wish to import, the csv:check-import command can be used in the command-line. This command displays the following:

• The number of rows of data (useful when estimating the amount of processing time needed to perform the import, and whether or not you can use the user interface to perform the import)
• The number of columns (useful when estimating the amount of developer time needed to map the columns to AtoM data - see: CSV Column mapping below)
• How many instances of pipe (|) characters are found in each column (pipe characters are used by some systems to put multiple values in a single cell of data)
• Sample column values

You will need access to the command-line of the server on which AtoM is installed, and you will need to know the file path where your CSV is currently located. Run the command from the root directory of your AtoM install.

Example use:

php symfony csv:check-import lib/task/import/example/rad/example_information_objects_rad.csv

CSV Column mapping¶

AtoM was originally built to encourage broad adoption and use of the ICA’s international standards, and expanded to support other recognized standards. Consequently, all of the description templates in AtoM correspond to a recognized content or metadata exchange standard, and many of the fields in our underlying database model are also inspired by ICA standards. For your data to import successfully into AtoM, you will first need to map it to one of our existing CSV templates, which are derived from the various standards-based templates available in AtoM for description.

Mapping your data to the supplied CSV templates below implies a familiarity with the standards used in AtoM, so you can make appropriate decisions on which fields in your data map to which fields in AtoM, and how to proceed if your data does not easily map 1:1 with the standard upon which our templates are based.

For further information and source links to the standards used in AtoM, see:

• Descriptive standards
• Data entry / templates

The cell values in the top row of a CSV data file conventionally name each column. The name indicates what kind of data the column contains. If the CSV data you wish to import doesn’t include a row like this, you should insert one. You should then make the names correspond to AtoM-friendly names using the top row of data in the example CSV file(s) appropriate to your import.

Available example files are:

• ISAD archival description CSV template
• RAD archival description CSV template
• Authority record CSV template
• Authority record relationships CSV template
• Accessions CSV template
• Deaccessions CSV template
• Events CSV template
• Repository CSV template

All CSV templates can be found on the AtoM wiki:

• https://wiki.accesstomemory.org/Resources/CSV_templates

You can also find all example CSV import templates included in your AtoM installation, in: lib/task/import/example.

The order of the columns in the example CSV files is the same as the order in the AtoM interface, and should be maintained. Having the correct names in the cell values of the first row of your CSV data enables AtoM to import values in each column to the correct AtoM fields.

Verify character encoding and line endings¶

For your CSV files to import properly, you will need to ensure two things prior to importing: that the character encoding of your CSV file is set to UTF-8, and that the end-of-line characters used in your CSV conform to the Unix/Linux style of newline character.

Data transformation¶

If you are working with a CSV export from another system (such a different database), you may need to do further pre-processing to prepare your CSV. If your previous system was designed for standards-compliance to a standard that AtoM supports (see: Descriptive standards), the mapping process might be simple - but if your system used custom data fields, mapping to one of the supported standards could be trickier.

You may, for example, want to combine multiple CSV column values, that don’t cleanly map conceptually to AtoM-compatible CSV columns, into single columns so they can be put into AtoM as notes. So ColumnA and ColumnB could be combined into a generalNote column. This requires you to transform the data before importing.

Depending on the size of your import data, this work can be done manually using a spreadsheet program - simply cut and paste your data into the corresponding cell in the provided import templates. However, for larger data sets, data transformation can be done with custom programming (for example, a Python script written by a developer), open source tools such as Open Refine or Pentaho Data Integration, or via a CSV transformation script.

We have included some guidelines for creating custom CSV transformation scripts. See:

• https://wiki.accesstomemory.org/Resources/CSV_transformation

Estimating import duration¶

Once you’ve mapped the columns names in your CSV export to the corresponding AtoM-compatible CSV column names you may wish to perform a test import.

A test import gives you an idea how long the import will take to complete on your hardware. To estimate how long it will take to import 20,000 rows of CSV data, for example, you could time the import of the first 1000 records and multiply that by 20.

If your test import proves to be too slow on your hardware, or you don’t have hardware to spare, you can consider using cloud computing resources, such as Open Hosting, Amazon EC2, or Rackspace Cloud.

Testing your import¶

Once you’ve prepared your import, you may want to clone your AtoM site and test your import on the clone before importing to your production AtoM installation. This prevents you from having to delete any improperly imported data. During import testing if you want to delete all imported data you can use the command-line purge tool.

See: Purging all data in the Administrator’s manual for more information.

Back to top

Hierarchical relationships¶

Information objects often have parent-child relationships - for example, a series may be a child of the fonds to which it belongs; it has a parent fonds. If you want to import a fonds or collection into AtoM along with its lower levels of description (i.e. its children - series, files, items, etc.), you will need a way to specify which rows in your CSV file belong to which parent description.

There are two basic ways to specify which information object is the parent of an information object being imported in your CSV - either through the use of the legacyID and parentID columns (generally used for new descriptions being imported, or from descriptions being migrated from another access system), or by using the qubitParentSlug column to import new child descriptions to an existing description in AtoM.

You can use a mix of legacyId/parentId and qubitParentSlug in the same CSV, just not in the same row. So, for example, if you wanted to import a series description as a child of a description already in AtoM, as well as several files as children of the series description, you could set a legacyID for the series, use the qubitParentSlug to point to the parent fonds or collection already in AtoM, and then use the parentID column for all your lower-level file descriptions. However, using both parentID and qubitParentSlug in the same row will cause a conflict, and AtoM will prefer the qubitParentSlug so the import does not fail.

Both methods of establishing hierarchical relationships are described below.

• LegacyID and parentID
• qubitParentSlug

LegacyID and parentID¶

One way to establish hierarchical relationships during a CSV import involves the use of the parentId column to specify a legacy ID (referencing the legacyId column of a previously imported information object). This way is most often used for migrations from other access systems. Using this method, parent descriptions (e.g. fonds, collections, etc) must appear first (i.e. above) in your CSV and must include a legacyID - while child records must appear after (i.e. below) their parent records in your CSV, and must include the legacyID of the parent record in the parentID column.

Here is an example of the first three columns of a CSV file (shown in a spreadsheet application), importing a Fonds > Series > Item hierarchy:

Important

When the CSV is imported, it progresses row by row - meaning, if your CSV is not properly ordered with parent records appearing before their children, your import will fail!

Records in a CSV are imported sequentially, row by row. As each row is imported, the legacyID value is added to AtoM’s keymap table in the database. When a parentID is encountered, AtoM will check the keymap table for the corresponding legacyID. If a parentID value doesn’t refer to the legacyID of a previously imported description, AtoM’s import logic will fall back to check for an existing description with that internal object ID.

If there is still no match, then a warning will be included in the console output (shown on the Job details page), and the record will be imported as a top-level description. Be sure to double-check the results post-import!

qubitParentSlug¶

The other method of importing hierarchical data into AtoM enables you to specify an existing archival description that doesn’t have a legacyID (one, for example, that has been manually created using the AtoM web interface), and import descriptions as children of the target description(s).

To specify a parent that exists in AtoM, you must first take note of the parent information object’s slug. The “slug” is a textual identifier that is included in the URL used to view the parent description. If the URL, for example, is http://myarchive.com/AtoM/index.php/example-fonds then the slug will be example-fonds. This slug value would then be included in your import in the qubitParentSlug column in the rows of children of the parent description.

Alternately, if you are using the command-line to perform your import, you can use the --default-parent-slug option in the command-line to set a default slug value, that will be used when no qubitParentSlug or parentID values have been included for the row. For more information, see the details in the Administrator’s manual on command-line CSV export: Importing archival descriptions.

Here is an example of the first few columns of a CSV file (shown in a spreadsheet application), importing a new series to an existing fonds, and importing two new file-level descriptions to an existing series:

If desired, you can mix the use of the qubitParentSlug column with the use of the parentID column in the same CSV - for example, you could attach a new series to an existing fonds by giving it a legacyID and the slug for the existing fonds in the qubitParentSlug column, and then including lower-level files attached to the new series by adding the legacyID of the new series to the parentID column of the new files.

Important

You should not add both a parentID and a qubitParentSlug to the same row - AtoM expects one or the other. When the import encounters both columns populated in a single row, AtoM will default to use the qubitParentSlug value. In general, each row must have only one or the other - either a parent slug, or a parent ID.

Creator-related import columns (actors and events)¶

The eventActors, eventActorHistories, eventTypes, eventDates, eventStartDates, and eventEndDates columns are related to the creation of actors and events. In AtoM’s data model, an archival description is a description of a record, understood as the documentary evidence created by an action - or event. It is events that link actors (represented in AtoM by an authority record) to archival descriptions - see Entity types for more information.

The most common use for these columns is to add creation dates associated with an actor via an authority record - in this case, the eventType is Creation, the eventActor is the creator, and the various eventDates fields are the dates of creation associated with the description.

However, some standards support other types of events as well. For example, the ISAD(G) template in AtoM also supports Accummulation as an event type, while the Canadian RAD template allows direct association between actors and events, and includes many other event types, such as contribution, broadcasting, manufacturing, and more.

A brief summary of the fields is included below, followed by a longer discussion of their use:

• eventActors: Add the associated creator or other actor name here. AtoM will link to an existing authority record or create a new one. For details on linking behavior, see below: On Authority records, archival descriptions, and CSV imports.

• eventActorHistories: Add the associated creator or other actor’s administrative or biographical history here. Equivalent to ISAD(G) 3.2.2, RAD 1.7B, and/or DACS 2.7 - Administrative/Biographical history. This will be mapped to the related authority record.

• eventTypes: Type of event. Values include:

  • ISAD: Creation, Accumulation
  • RAD: Creation, Accumulation, Contribution, Collection, Broadcasting, Manufacturing, Custody, Publication, Reproduction, Distribution

• eventDates: Display dates shown in public user interface for event on

  archival description view page. May use free-text, including typographical conventions to express approximation or uncertainty (e.g. [190-?]; [ca. 1885]).

• eventStartDates: Internal ISO-8601 formatted (e.g. YYYY, YYYY-MM,

  YYYY-MM-DD) start date of event

• eventEndDates: Internal ISO-8601 formatted (e.g. YYYY, YYYY-MM,

  YYYY-MM-DD) end date of event

• eventDescriptions: Only in RAD CSV template. Adds a note to the event
• eventPlaces: Only in RAD CSV templates. Associates a place with the event.

If multiple actors/events exist for an information object, the values in these fields can be pipe-separated (e.g. using the | pipe separator between values).

On Authority records, archival descriptions, and CSV imports¶

AtoM tries to support the reusability of actor information through the maintenance of authority records that can be linked to archival descriptions and other entities. This, and the rationale for this, is outlined in greater detail in the following sections:

• AtoM, Authority records, Biographical histories, and Access points
• On Name vs. Subject Access Points

This also affects how actor names are handled during a CSV import. Some of the key behaviors are outlined below:

Creating new actor records on import

• AtoM looks for creator names in the eventActors column in the RAD and ISAD CSV import templates, as well as access point names (used as subjects) in the nameAccessPoints column during a CSV import of archival descriptions.
• Similarly, any Administrative / biographical history data in an archival description CSV import (i.e. data contained in the eventActorHistories CSV column will be mapped to the “History” field (ISAAR-CPF 5.2.2) in the related authority record (generated from the data contained in the creators column of the CSV), and then is presented in AtoM in any related descriptions where the entity is listed as a creator.
• Where multiple creator names and histories are included in an import, eventActors and eventActorHistories elements are matched 1:1 in the order they appear in the CSV, divided by pipe elements (e.g. | ). For example, if the eventActors column contains name 1|name 2, the eventActorHistories should also include history 1|history 2 to match on import. If there is no history for the first actor, you can include NULL, and AtoM will ignore the imput - e.g. name 1|name 2 should be matched with NULL|history 2 to include only a history for name 2.
• This same NULL approach can be used for any matched date values where multiple actor names are included for import - eventDates, eventStartDates, eventEndDates can all include NULL if you wish to leave these blank when associating multiple actors with an event. An example, using the RAD template:

• If a creator history element is included in a CSV import, but no creator name is included, AtoM will still automatically generate a stub authority record and map the history data to the “History” field (ISAAR-CPF 5.2.2) - the authority record will be left untitled, until the user manually adds the appropriate name to the authority record. Similarly, if there are more eventActorHistories elements included in an import than creator names included in the creators column, the final biographical/administrative history will be mapped to an untitled authority record. Because the slug is normally based on the title of the authority record, AtoM will generate a random alphanumeric string to use as the slug - and you will not be able to edit this through the user interface.

Digital object-related import columns¶

As of AtoM 2.1, two new columns have been added to the ISAD and RAD CSV import templates: digitalObjectPath and digitalObjectURI. These columns will allow you to link or upload a digital object and attach it to the new information object being created in that row of the CSV.

In AtoM, a 1:1 relationship is maintained between information objects and digital objects - meaning that for every archival description, you can only attach one digital object. If you wish, you can create new child records - a number of item descriptions as children of a file-level description; a number of part descriptions as children of an item (for multiple views of a single object, for example, or individual pages of a single book uploaded separately, etc), and so on.

In the CSV templates, the digitalObjectPath and digitalObjectURI columns are positioned after the publicationStatus column, and before the physical object-related import columns.

The digitalObjectPath column can be used to upload local digital objects - simply provide a complete path and filename to the digital object.

The digitalObjectURI column can be used to link to externally hosted, publicly available digital objects, such as those available at a specific URL on the web. You must have a path directly to the digital object which includes a file extension, and not just to a web page with a digital object located on it somewhere - it is often the equivalent of right-clicking on a digital object in your browser and selecting “View image”.

You can use a mixture of the digitalObjectPath and digitalObjectURI columns throughout your CSV (linking some information object rows to locally uploaded digital objects, and others to web-based resources), but you cannot use both columns in the same row. If AtoM encounters a CSV row where both the digitalObjectPath and digitalObjectURI columns are populated, it will favor the digitalObjectURI value, and ignore the digitalObjectPath value.

Physical object-related import columns¶

The physicalObjectName, physicalObjectLocation, and physicalObjectType columns are related to the creation of physical objects and physical storage locations related to an archival description.

These fields will also support | pipe separators to add multiple physical storage containers per row. When piping one column, each related column must also be piped correspondingly:

For more information on working with physical storage in AtoM, see: Physical storage.

Standards related fields¶

Most fields in the CSV template have been named in a fairly obvious way, translating a simplified version of the field name in our data entry templates into a condensed camelCase. For example, the Rules for Archival description’s General Material Designation is rendered in the CSV header as radGeneralMaterialDesignation. In both the RAD and ISAD templates, the Scope and Content field is marked by the CSV header scopeAndContent. However, for users seeking a full mapping of fields, consult the RAD template and ISAD template pages for further details.

The culture column indicates to AtoM the default language of the descriptions being uploaded. This column expects two-letter ISO 639-1 language code values - for example, “en” for English; “fr” for French, “it” for Italian, etc. See Wikipedia for a full list of ISO 639-1 language codes.

Other data entry notes¶

• language and languageOfDescription, like culture, expect two-letter ISO 639-1 language code values - for example, “en” for English; “fr” for French, “it” for Italian, etc. See Wikipedia for a full list of ISO 639-1 language codes. Unlike the culture column, however, these two fields will accept multiple values separated by a pipe character - for example, en|fr|it.
• The script and scriptOfDescription columns expect four-letter ISO 15924 script code values - for example, “Latn” for Latin-based scripts, “Cyrl” for Cyrillic scripts, etc. See Unicode for a full list of ISO 15924 script codes.
• Alternative identifiers and their display labels can be imported using the alternativeIdentifiers and alternativeIdentifierLabels columns. Use pipe (|) separators to add multiple values. There should be a 1:1 relationship between the number of identifier values in the alternativeIdentifiers column and corresponding labels in the alternativeIdentifierLabels column.
• An accessionNumber column can be added to create a link between an existing accession record and an archival description being imported via CSV. See the section on Accession CSV import below for more information.

Importing translations¶

As of version 2.6, AtoM will allow you to import new descriptions in multiple languages at once, as a way of adding translations to your source content during an import.

At this time, not all CSV fields support translation imports. Only those fields found in AtoM’s information_object_i18n database table can be imported as translations. These include:

• title
• alternateTitle
• radEdition
• extentAndMedium
• archivalHistory
• acquisition
• scopeAndContent
• appraisal
• accruals
• arrangement
• accessConditions
• reproductionConditions
• physicalCharacteristics
• findingAids
• locationOfOriginals
• locationOfCopies
• relatedUnitsOfDescription
• rules
• sources
• revisionHistory
• institutionIdentifier

The translation import works on the following logic: whenever AtoM encounters two CSV rows that have the same legacyId value, but different culture values, AtoM will import the second row as a translation of the first. AtoM expects two-letter ISO 639-1 culture codes to be used in the culture column - e.g. en for English, fr for French, etc. See the section above, Other data entry notes, for further information.

Preparing translations for import

To import new archival descriptions with translations:

• Make sure that every row in your CSV has a legacyId and a culture value
• Place translation rows directly below the source culture row
• Translation rows must have the same legacyId value as their source culture rows
• Translation rows must have a different culture value as their source culture rows
• Make sure that all culture values use ISO 639-1 two-letter codes
• In the translation rows, leave any columns that do not support translation blank

An example CSV:

Back to top

Fields that will support update imports¶

Currently, not all fields in AtoM’s archival description metadata templates can be updated via import. Only those fields which are found in AtoM’s primary information_object and information_object_i18n database tables will support updates in place. Below is a list of supported fields:

• title
• identifier
• levelOfDescription
• repository
• alternateTitle (in the RAD CSV template)
• radEdition (in the RAD CSV template)
• extentAndMedium
• archivalHistory
• acquisition
• scopeAndContent
• appraisal
• accruals
• arrangement
• language
• script
• accessConditions
• reproductionConditions
• physicalCharacteristics
• findingAids
• locationOfOriginals
• locationOfCopies
• relatedUnitsOfDescription
• digitalObjectPath
• digitalObjectURI
• rules
• languageOfDescription
• scriptOfDescription
• sources
• descriptionStatus
• levelOfDetail
• revisionHistory
• institutionIdentifier
• alternativeIdentifiers
• alternativIdentifierLabels

There are also additional fields that are not stored in AtoM’s primary information object database tables that can potentially receive new data via an update import. In these cases, existing data will not be replaced - instead, the update import will append new data to the existing resources. These fields typically relate to note fields (such as all the custom note types in the RAD and DACS templates), or linked entities such as terms, authority records, etc.

Below is a list of fields to which new data can be appended via an update import - any existing data will be left in place:

• accessionNumber
• radGeneralMaterialDesignation
• radTitleStatementOfResponsibilityNote
• radTitleAttributionsAndConjectures
• radTitleContinues
• radTitleSourceOfTitleProper
• radTitleVariationsInTitle
• radTitleParallelTitles
• subjectAccessPoints
• placeAccessPoints
• genreAccessPoints
• nameAccessPoints
• radNoteAccompanyingMaterial
• radNoteAlphaNumericDesignation
• radNoteConservation
• radNoteEdition
• radNotePhysicalDescription
• radNotePublisherSeries
• radNoteRights
• radNoteCast
• radNoteCredits
• radNoteSignaturesInscriptions
• generalNote
• archivistNote
• physicalObjectName
• physicalObjectLocation
• physicalObjectType

Finally, please note that eventActors (i.e. creators and other actors associated with different event types) and other related event fields are a special case. Please see the table in the section above to determine the matching criteria and resulting behavior:

• Attempting to match to existing authority records on import

Matching criteria for archival descriptions¶

AtoM uses the following cascading criteria when checking for matches on existing archival descriptions during a CSV import:

• First AtoM will look for an exact match in the legacyID and the source_name value (for more information on source_name, see above: Legacy ID mapping: dealing with hierarchical data in a CSV). During a CSV import via the user interface, the default source name value stored will be the filename of the CSV.

Tip

You can always check what source name was used for records created via an import by entering into edit mode and navigating to the Administration area of the edit page - the source name used will be diplayed there:

• If no match is found on legacyID and source_name, then AtoM will look for an exact match on title, repository, and identifier.
• If there is no exact match on all of these 3 values, then the record is considered not to have a match. Depending on user settings during import, it will either import as new, or be skipped and reported in the Job details page.

The default behavior when no match is found during an updates import is to import the record as a new description. However, AtoM does have an option in the user interface (and in the command-line options) to skip unmatched records. When this option is selected, any records that do not match existing descriptions will be ignored during the import, and reported in the console log shown on the Job details page of the related import job (see: Manage jobs for more information). This is recommended if you are intending to only import updates to existing records.

You can also narrow the scope of the matching criteria to either records linked to a specific repository (i.e. an archival institution), or a specific archival unit (i.e. a top-level description such as a fonds or collection, etc.). To avoid contradictory options, AtoM will only allow one of these parameters to be specified at a time. These options are useful for ensuring that you are matching the correct descriptions before updating them - for example, when importing updates to the records of one specific institution into a multi-repository system.

Update matches in place¶

AtoM’s first option for updates allows you to use incoming CSV data as an update to existing descriptions. Whenever a match is found for an existing description, AtoM will use the data in the CSV to update the related field in place. If a column is left blank in the updates CSV, it will be ignored (it will not overwrite existing data by erasing it).

To import a CSV as updates to existing descriptions, select the option “Update matches ignoring blank fields in CSV” from the “Update behaviours” drop-down menu on the CSV import page.

If a match is not found during the import, the default behavior is to import the CSV row as a new record. If you are only importing updates, we recommend clicking the checkbox for the “Skip unmatched records” - AtoM will then skip any unmatched CSV rows and report them in the console log shown on the Job details page of the related import job.

Specific instructions on configuring the user interface for import are included below - see: Importing updates via the user interface.

Delete matches and replace with imported records¶

AtoM’s second update option allows you to identify matched descriptions during import, and then delete the matches prior to importing the CSV data as a new record to replace it.

Note that only the matched archival description and its children are deleted during this process. Any related/linked entities (such as an authority record linked as a creator, subject/place/name/genre access points, linked accession records, physical storage locations, etc.) are not automatically deleted. If you want these fully removed, you will have to find them and manually delete them via the user interface after the import.

Once the original matched archival description has been deleted, the CSV import proceeds as if the record is new. That is to say, just as AtoM does not automatically delete related entities in the original archival description, it also not automatically re-link previously related entities. Instead, AtoM will use its default matching behaviors to determine if related entities in the import (such as a creator in the eventActors CSV column) should be linked to existing records in AtoM, or created as new records during the import. AtoM’s matching criteria for authority records is outlined above - see: Attempting to match to existing authority records on import

To import an archival description CSV of records as replacements for existing descriptions in AtoM, select the “Delete matches and replace with imported records” option from the “Update behaviours” drop-down menu on the CSV import page.

If a match is not found during the import, the default behavior is to import the CSV row as a new record. If you are only importing updates, you can click the checkbox for the “Skip unmatched records” if desired - AtoM will then skip any unmatched CSV rows and report them in the console log shown on the Job details page of the related import job.

Specific instructions on configuring the user interface for import are included below - see: Importing updates via the user interface.

Importing updates via the user interface¶

If you have double-checked the above, you should be ready to import your updates.

To import a CSV file of description updates via the user interface:

1. Click on the Import menu, located in the AtoM header bar, and select “CSV”.

2. AtoM will redirect you to the CSV import page. Make sure that the “Type” drop-down menu is set to “Archival description” .

3. Select the type of update import you want to initiate.

   To update existing archival descriptions in place, select the option “Update matches ignoring blank fields in CSV” from the “Update behaviours” drop-down menu on the CSV import page.

   

   To delete existing matched archival descriptions and replace them with the data in your CSV, select the “Delete matches and replace with imported records” option from the “Update behaviours” drop-down menu on the CSV import page.

   

4. AtoM’s default behavior when it cannot find a match during an update import is to import the CSV row as a new record. However, if you are only importing updates and don’t want to accidentally create new records when no match is found, you can check the “Skip unmatched records” checkbox. Any unmatched records will not be imported - instead, skipped records will be reported in the Job details page of the related import job (see: Manage jobs for more information).

5. To improve the default matching behavior (described in detail above: Matching criteria for archival descriptions), you can provide further criteria to help AtoM find the correct match. You can limit the matches to either the holdings of a specific repository, or if you are updating a single archival unit, you can limit matches to a specific top-level description. To prevent contradictory options, these limiters are mutually exclusive - you cannot pick a repository AND a top-level description.

   To limit your matches to the holdings of a specifc archival institution, use the drop-down menu to select the name of the related institution

   To limit your matches to a specific top-level description, place your cursor in the “Top-level description” field and begin to slowly type the name of the top-level description. This field is an autcomplete menu - as you begin to type, AtoM will display matching records in the drop-down menu that will appear below. When you see the top-level description you want to use as a match limit, click on it in the drop-down menu to select it.

6. If you do not want your files indexed during the import, you can click the checkbox labelled “Do not index imported items.” This will prevent the new records from automatically being added to AtoM’s search index.

7. When you have configured your import options, click the “Browse” button to open a window on your local computer. Select the CSV file that you would like to import as your update.

8. When you have selected the file from your device, its name will appear next to the “Browse” button. Click the “Import” button located in the button block to begin your import.

9. After your import is complete, AtoM will indicate that the import has been initiated. A notification at the top of the page will also provide you with a link to the Job details page of the related import job. Alternatively, you can click the “Back” button in the button block at the bottom of the page to return to the CSV import page, or navigate elsewhere in the application.

Tip

Want to find your recently updated records? You can use the sort button located in the top-right hand side of the archival description browse page to change the results display to be ordered by “Most recent” if it is not already - that way, the most recently added or edited descriptions will appear at the top of the results. If you have come directly here after importing your descriptions, they should appear at the top of the results.

10. If any warnings or errors are encountered, AtoM will display them on Job details page of the related import job. Generally, errors will cause an import to fail, while warnings will be logged but will allow the import to proceed anyway. Errors can occur for many reasons - please review the checklist above for suggestions on resolving the most common reasons that CSV import updates fail.

Back to top

Repository CSV columns¶

The legacyID column is a required field, used only internally. Add a unique numeric value to this column. The legacyID can be used to assist in matching multiple CSV import types. Further information is provided in the section on legacy ID mapping above - see: Legacy ID mapping: dealing with hierarchical data in a CSV.

The uploadLimit column allows a user to set a default upload limit for a repository at the time of import. This value should be a number, representing Gigabytes. For more information on the use of respository upload limits in AtoM, see: Set digital object upload limit for an archival institution.

The types column relates to the institution type found in the Identity information area of the archival institution edit page. It is linked to the “Repository Types” taxonomy in AtoM, which comes pre-populated with 22 terms. New terms added via the CSV import will be created successfully and be added to the Repository Types taxonomy. You can add multiple terms by using the | pipe character between them, like so:

Arts Organization|Community

Almost all other fields are drawn directly from the archival institution edit template in AtoM, which is based upon the International Council on Archives’ International Standard for Describing Institutions with Archival Holdings (ISDIAH). For more information on the use of each field and a link to the related standard, see: ISDIAH edit template. Generally, fields in the CSV template have been named in a fairly obvious way, translating a simplified version of the field name in our data entry templates into a condensed camelCase. For example, ISDIAH 5.3.2, Geographical and cultural context (in the Description Area) becomes geoCulturalContext in the CSV template. Consult the ISDIAH page for further help with fields.

The descStatus and descDetail columns are both also related to controlled term taxonomies in AtoM - the “Description Statuses” and “Description Detail Levels” taxonomies respectively. The descStatus column has the following default terms available for use:

• Final
• Revised
• Draft

The descDetails column has the following default terms available for use:

• Full
• Minimal
• Partial

The culture column indicates to AtoM the language of the descriptions being uploaded. This column expects two-letter ISO 639-1 language code values - for example, “en” for English; “fr” for French, “it” for Italian, etc. See Wikipedia for a full list of ISO 639-1 language codes.

Updating repository records in place via CSV import¶

AtoM’s first option for archival institution updates allows you to use incoming CSV data as an update to existing repository records. Whenever a match is found for an existing repository (based on the authorized form of name), AtoM will use the data in the CSV to update the related field in place. If a column is left blank in the updates CSV, it will be ignored (it will not overwrite existing data by erasing it).

To import a CSV as updates to existing repository records, select the option “Update matches ignoring blank fields in CSV” from the “Update behaviours” drop-down menu on the CSV import page.

If a match is not found during the import, the default behavior is to import the CSV row as a new record. If you are only importing updates, you can click the checkbox for the “Skip unmatched records” if desired - AtoM will then skip any unmatched CSV rows and report them in the console log shown on the Job details page of the related import job.

See below for step-by-step instructions on importing repository updates via the user interface.

Deleting and replacing repository records via CSV import¶

AtoM’s second update option allows you to identify matched repositories during import, and then delete the matches prior to importing the CSV data as a new record to replace it.

Note that only the matched archival institution record is deleted during this process. Any related/linked entities (such as an authority record linked as being maintained by the repository, Thematic area or other repository access points, and linked archival descriptions) are not automatically deleted. If you also want these fully removed, you will have to find them and manually delete them via the user interface after the import.

Once the original matched repository record has been deleted, the CSV import proceeds as if the record is new. That is to say, just as AtoM does not automatically delete entities related to the original archival institution, it also not automatically re-link previously related entities.

To import an archival institution CSV of records as replacements for existing repositories in AtoM, select the “Delete matches and replace with imported records” option from the “Update behaviours” drop-down menu on the CSV import page.

If a match is not found during the import, the default behavior is to import the CSV row as a new record. If you are only importing updates, you can click the checkbox for the “Skip unmatched records” if desired - AtoM will then skip any unmatched CSV rows and report them in the console log shown on the Job details page of the related import job.

See below for step-by-step instructions on importing repository updates via the user interface.

Importing repository updates in the user interface¶

If you have double-checked the above, you should be ready to import your updates.

To import a CSV file of repository updates via the user interface:

1. Click on the Import menu, located in the AtoM header bar, and select “CSV”.

2. AtoM will redirect you to the CSV import page. Make sure that the “Type” drop-down menu is set to “Archival institution” .

3. Select the type of update import you want to initiate.

   To update existing archival institutions in place, select the option “Update matches ignoring blank fields in CSV” from the “Update behaviours” drop-down menu on the CSV import page.

   

   To delete existing matched archival institutions and replace them with the data in your CSV, select the “Delete matches and replace with imported records” option from the “Update behaviours” drop-down menu on the CSV import page.

   

4. AtoM’s default behavior when it cannot find a match during an update import is to import the CSV row as a new record. However, if you are only importing updates and don’t want to accidentally create new records when no match is found, you can check the “Skip unmatched records” checkbox. Any unmatched records will not be imported - instead, skipped records will be reported in the Job details page of the related import job (see: Manage jobs for more information).

5. If you do not want your files indexed during the import, you can click the checkbox labelled “Do not index imported items.” This will prevent the new records from automatically being added to AtoM’s search index.

6. When you have configured your import options, click the “Browse” button to open a window on your local computer. Select the CSV file that you would like to import as your update.

7. When you have selected the file from your device, its name will appear next to the “Browse” button. Click the “Import” button located in the button block to begin your import.

8. After your import is complete, AtoM will indicate that the import has been initiated. A notification at the top of the page will also provide you with a link to the Job details page of the related import job. Alternatively, you can click the “Back” button in the button block at the bottom of the page to return to the CSV import page, or navigate elsewhere in the application.

9. If any warnings or errors are encountered, AtoM will display them on Job details page of the related import job. Generally, errors will cause an import to fail, while warnings will be logged but will allow the import to proceed anyway. Errors can occur for many reasons - please review the checklist above for suggestions on resolving the most common reasons that CSV import updates fail.

Back to top

Authority record fields that will support update imports¶

Currently not all fields in AtoM’s authority record metadata template can be updated via import. Only those fields which are found in AtoM’s primary actor and actor_i18n database tables will support updates in place. Additionally, because the authorizedFormOfName field is used for matching on import, editing this field will cause a match to fail - meaning it can’t be updated via CSV import. Below is a list of supported fields that can be updated in place:

• typeOfEntity
• corporateBodyIdentifiers
• datesOfExistence
• history
• places
• legalStatus
• functions
• mandates
• internalStructures
• generalContext
• digitalObjectURI
• digitalObjectPath
• descriptionIdentifier
• institutionIdentifier
• rules
• status
• levelOfDetail
• revisionHistory
• sources
• maintenanceNotes

There are also additional fields that are not stored in AtoM’s primary authority record database tables that can potentially receive new data via an update import. In these cases, existing data will not be replaced - instead, the update import will append new data to the existing resources, leaving the current data in place as well. These fields typically relate to access point fields (such as subjects, places, and occupations), as well as alternative forms of name.

Below is a list of fields to which new data can be appended via an update import - any existing data will be left in place:

• parallelFormsOfName
• standardizedFormsOfName
• otherFormsOfName
• actorOccupations
• actorOccupationNotes
• subjectAccessPoints
• placeAccessPoints

Update authority records in place via CSV import¶

AtoM’s first option for authority record updates allows you to use incoming CSV data as an update to existing authority records in place. Whenever a match is found for an existing authority record (based on the authorized form of name), AtoM will use the data in the CSV to update the related field in place. If a column is left blank in the updates CSV, it will be ignored (it will not overwrite existing data by erasing it).

To import a CSV as updates to existing authority records, select the option “Update matches ignoring blank fields in CSV” from the “Update behaviours” drop-down menu on the CSV import page.

See below for step-by-step instructions on importing repository updates via the user interface.

Delete and replace existing authority records via CSV import¶

AtoM’s second update option allows you to identify matched authority records during import, and then delete the matches prior to importing the CSV data as a new record to replace it.

Note that only the matched authority record is deleted during this process. Any related/linked entities (such as a repository linked as the authority record’s maintainer, other authority records linked via a relationship, Occupation access points, and linked archival descriptions) are not also automatically deleted. If you also want these fully removed, you will have to find them and manually delete them via the user interface after the import.

Once the original matched authority record has been deleted, the CSV import proceeds as if the record is new. That is to say, just as AtoM does not automatically delete entities related to the original archival institution, it also does not automatically re-link previously related entities.

To import an archival institution CSV of records as replacements for existing repositories in AtoM, select the “Delete matches and replace with imported records” option from the “Update behaviours” drop-down menu on the CSV import page.

See below for step-by-step instructions on importing authority record updates via the user interface.

Updating authority records via import in the user inteface¶

If you have double-checked the above, you should be ready to import your updates.

To import a CSV file of authority record updates via the user interface:

1. Click on the Import menu, located in the AtoM header bar, and select “CSV”.

2. AtoM will redirect you to the CSV import page. Make sure that the “Type” drop-down menu is set to “Archival institution” .

3. Select the type of update import you want to initiate.

   To update existing authority records in place, select the option “Update matches ignoring blank fields in CSV” from the “Update behaviours” drop-down menu on the CSV import page.

   

   To delete existing matched authority records and replace them with the data in your CSV, select the “Delete matches and replace with imported records” option from the “Update behaviours” drop-down menu on the CSV import page.

   

4. AtoM’s default behavior when it cannot find a match during an update import is to import the CSV row as a new record. However, if you are only importing updates and don’t want to accidentally create new records when no match is found, you can check the “Skip unmatched records” checkbox. Any unmatched records will not be imported - instead, skipped records will be reported in the Job details page of the related import job (see: Manage jobs for more information).

5. To improve the default matching behavior (based on the authorized form of name of the authority record), you can provide further criteria to help AtoM find the correct match. If you have linked your authority record(s) to a repository as the maintainer, then you can limit the matches to either the authority records of a specific repository. To learn more about linking an authority record to a repository, see: Link an authority record to a repository as its maintainer.

   To limit your matches to the holdings of a specifc archival institution, use the drop-down menu to select the name of the related repository.

   If you previously created your repository record via CSV import, it will sometimes not appear at first in the drop-down menu. Try to begin typing the first letters of the target repository name when your cursor is on the Limit matches drop-down to see if it appears.

6. If you do not want your files indexed during the import, you can click the checkbox labelled “Do not index imported items.” This will prevent the new records from automatically being added to AtoM’s search index.

7. When you have configured your import options, click the “Browse” button to open a window on your local computer. Select the CSV file that you would like to import as your update.

8. When you have selected the file from your device, its name will appear next to the “Browse” button. Click the “Import” button located in the button block to begin your import.

9. After your import is complete, AtoM will indicate that the import has been initiated. A notification at the top of the page will also provide you with a link to the Job details page of the related import job. Alternatively, you can click the “Back” button in the button block at the bottom of the page to return to the CSV import page, or navigate elsewhere in the application.

10. If any warnings or errors are encountered, AtoM will display them on Job details page of the related import job. Generally, errors will cause an import to fail, while warnings will be logged but will allow the import to proceed anyway. Errors can occur for many reasons - please review the checklist above for suggestions on resolving the most common reasons that CSV import updates fail.

Back to top

Linking an imported accession to an archival description¶

To link incoming accession records to an existing archival description, you can add a column named qubitParentSlug. This column will behave similarly to the qubitParentSlug column in the archival description CSV templates (described above). To link an accession row in your CSV to an existing description in your AtoM instance, simply enter the slug of the target description in the qubitParentSlug column. AtoM will locate the matching description and link the two during import, similar to how an accession created through the user interface can be linked to a description (see: Link an accession record to an archival description).

Standards related fields¶

Most fields in the CSV template have been named in a fairly obvious way, translating a simplified version of the field name in the AtoM user interface into a condensed camelCase in our data entry. For example, the “Immediate source of acquisition” field in the user interface is represented in the CSV template as the sourceOfAcquisition column.

Some fields are linked to a taxonomy of controlled values in AtoM, maintained as terms. These are generally represented in the accession record edit page in the user interface as a drop-down menu providing a limited list of options for a user to select. To learn more about managing taxonomies and terms, see: Terms. Some of the taxonomy-related fields in the CSV template include:

• acquisitionType - listed in the user inteface as “Acquisition type.” Default values are: Deposit, Gift, Purchase, Transfer.
• resourceType - listed in the user interface as “Resource type.” Default values are: Public transfer, Private transfer.
• processingStatus - listed in the user interface as “Processing status.” Default values are: Complete, Incomplete, In-Progress.
• processingPriority - listed in the user interface as “Processing priority.” Default values are: High, Medium, Low.

Physical object-related import columns¶

The physicalObjectName, physicalObjectLocation, and physicalObjectType columns are related to the creation of physical storage containers and locations related to an accession record.

For more information on working with physical storage in AtoM, see: Physical storage.

These fields will also support | pipe separators to add multiple physical storage containers per row. When piping one column, each related column must also be piped correspondingly:

Creator-related import columns (actors and events)¶

The creators column can be used to add the name(s) of the creator of the accession materials. For now, only creation events are supported, so the the eventTypes value should always be Creation. eventDates, eventStartDates, and eventEndDates columns can be used similarly to those in the archival description CSV templates - The eventDates field will map to the free-text display date field in AtoM, where users can use special characters to express approximation, uncertainty, etc. (e.g. [190-?]; [ca. 1885]), while eventStartDates and eventEndDates should include ISO 8601-formatted date values (YYYY-MM-DD, YYYY-MM, or YYYY). If there are multiple creators/events associated with the accession, these fields can all accept multiple values, separated using the pipe | character.

Donor-related import columns¶

The following columms relate to donor information in the CSV import template:

• donorName
• donorStreetAddress
• donorCity
• donorRegion
• donorCountry
• donorPostalCode
• donorTelephone
• donorFax
• donorEmail
• donorNote
• donorContactPerson

On import, data entry in these columns will be treated as a single donor. Post-import, Donor records can be managed separately from accessions in AtoM’s user interface. For more information on managing Donor records, see:

• Donor records

Other accession CSV data entry notes¶

The acquisitionDate column expects date strings to be formatted according to the ISO 8601 date format standard - that is, YYYY-MM-DD. ISO 8601 prescribes, as a minimum, a four-digit year [YYYY]. If the date range is not formatted according to ISO 8601 formatting, then AtoM will use the PHP date_parse function (which adds ‘1’ as default value to the month and day if they are missing) to modify the date to a YYYY-MM-DD format.

The culture column indicates to AtoM the default language of the accession records being uploaded. This column expects two-letter ISO 639-1 language code values - for example, “en” for English; “fr” for French, “it” for Italian, etc. See Wikipedia for a full list of ISO 639-1 language codes.

Back to top

Fields that will support update imports¶

Currently not all fields in AtoM’s accession record metadata templates can be updated via import. Only those fields that are found in AtoM’s primary accession and accession_i18n database tables will support updates in place. Below is a list of supported fields:

• title
• acquisitionDate
• sourceOfAcquisition
• locationInformation
• acquisitionType
• resourceType
• title
• archivalHistory
• scopeAndContent
• appraisal
• physicalCondition
• receivedExtentUnits
• processingStatus
• processingPriority
• processingNotes

There are also additional fields that are not stored in AtoM’s primary accesion record database tables that can potentially receive new data via an update import. In these cases, existing data will not be replaced - instead, the update import will append new data to the existing resources. These fields typically include related entities such as donors, creators and event dates, physical storage, as well as alternative identifiers.

Below is a list of fields to which new data can be appended via an update import - any existing data will be left in place:

• alternativeIdentifiers
• alternativeIdentifierTypes
• alternativeIdentifierNotes
• physicalObjectName
• physicalObjectLocation
• physicalObjectType
• donorName
• donorStreetAddress
• donorCity
• donorRegion
• donorCountry
• donorPostalCode
• donorTelephone
• donorFax
• donorEmail
• donorNote
• donorContactPerson
• creators
• eventTypes
• eventDates
• eventStartDates
• eventEndDates

The process for importing updates is identical to importing a CSV of new accession record metadata - see above for step-by-step instructions on how to import an accession record CSV in AtoM.

Back to top