Some thoughts about a minimalistic Archival Information System, part 2

 In the last post I explained some basic terms. Now it's time for the real thing. 

Choosing the right format

The first question, what should the information packages (SIP, AIP, DIP) look like? It is important that they are easy to process, easy to understand and easy to expand. Fortunately, there is RFC8493 that has the solution ready for us: BagIt.

 In (1) we store the metadata, in (2) there is space for our payload. BagIt is simple, it is a definition of a directory structure and some files that take over certain functions. Very interesting for us, if we want to store digital objects, we can store them in the BagIt payload. We can take over this area completely unchanged when processing a SIP and creating the AIP. The same is possible later when creating the DIPs from the AIPs. BagIt gives a lot of freedom. To limit ourselves, we choose UTF-8 for all metadata and text files.And we don't use fetch bags. Since BagIt is now standardized, we use version 1.

 

Metadata and AIP update considerations

 

Many AIS systems are insufficiently prepared for metadata and AIP updates. In my experience, it is important to think about how and which data is updated and what the consequences are. In order to enable the producer to submit supplements, these must be clearly assigned to an existing AIP. Either you give the producer back an ID for his first recording. This is not a good choice because the process then has a strong coupling and internals are exposed to the outside world. In addition, if a producer wants to change the AIS, there can be collisions. A better choice is to tell the producer to choose a unique ID for your data yourself and transmit it in your SIPs. Internally, we would then use these to search for the appropriate AIPs. The ID is called "ExternalID" and is the base for our internal MAIS-AIP-ID. More on that later.

In the last post I already mentioned that we have to think about the topic of versioning of AIPs. Not only because of the metadata or AIP updates, but also in the case of a PP&A, i.e. format migration. A simple idea is to introduce linked lists.

This allows us to easily implement the functionality of rolling back an AIP version as well.

A new AIP points to the predecessor in which the new version receives a reference entry in the "bag-info.txt":

 

• 'MAIS-previous-AIP' - contains AIP-ID of the current AIS (MAIS-AIP-ID)
  
• 'MAIS-migrated-AIP' - contains AIP-ID of the previous AIS if AIP was migrated from there
  
• 'MAIS-origin-AIS' - contains identifiers of the previous AIS from where the AIP was migrated

 The last two keys are optional and only needed if AIP-AIP-Transfer is needed to move digital objects from one archival information system to another.

Draft for a differential BagIt

The problem

BagIt (RFC 8493) forms the basis for Submission Information Packages (SIP) and Archival Information Packages (AIP) in many digital archives.
Especially in the library environment, it is necessary to support supplemental submissions in the Archival Information System (AIS) software. Supplements may be limited to metadata or may add new files, remove existing files, or replace existing files.
Unfortunately, there is no way to implement a differential SIP cleanly and easily in the BagIt specification.

The constraints

A design of a differential BagIt (dBagIt) should meet the following conditions:

1. existing BagIt should not be touched
2. it should be based on the BagIt structure so that the conversion effort is minimal
3. it should be easy to implement
4. it should support the "add" and "delete" operations
5. the checksum protection should be guaranteed
6. the referenced bag should be specified explicitly

The proposal of dBagIt

The basis is the structure of BagIt. The following are the changes that are mandatory.

Bag Declaration: dbagit.txt

In contrast to 2.1.1 of RFC8493 the filename is dbagit.txt

Payload Manifest

In contrast to 2.1.3 of RFC8493 each line of a payload manifest file MUST be of the form

   sign checksum filepath

where sign is either + for adding a file or - for deleting a file. 

The replacement of files is simulated by one entry each for deleting and adding.

Bag Metadata: bag-info.txt

Additional to RFC8493 the key Updates-External-Identifier becomes mandatory. It is used to reference to the original data object, which will be updated by this dBagIt.

Optional Tag Manifest

The Tag Manifest is similar to RFC8493. 

Although tag manifest files in BagIt could be used to describe additional proprietary subdirectories of a bag not specified in the RFC, it is not defined here to support changes as in the previous section on payload manifest. This facilitates the creation and processing of dBagIts. 

Implementation of the behavior

The implementation must ensure that:

1. the target object referenced by key Updates-External-Identifier exists
2. the dBagIt is valid
3. the add/delete operations are atomic and rollback-able
4. the checksums of files which should be added are correct and part of the current payload
   
5. the checksum of files which should be deleted are similar to the checksum of the files in the referenced digital object
   
6. the files in tag manifests handled correctly if proprietary extensions used
   
7. the metadata content in bag-info.txt replaced previous versions in referenced object completely
   

Future

If there is interest, I would be happy to receive feedback via art1pirat ATgmail.com. Maybe a new RFC can grow out of it.

Alternate consideration

A very simple solution could also be the use of unified 'diff'. This also allows partial changes in files, but would hardly bring any advantages with binary data and is not quite as intuitive for users who are not familiar with IT.

FAQ (Update 2022-05-18)

1. What if "delete" references a non-existing file? The complete operations via differential BagIt should be atomar and consistent. In this case the operations are rollbacked and aborted with an error. This ensure that no unintented updates will be applied.
2. Wouldn't it be nice, to avoid transferring files, to allow a simple rename instead of a replace? This would be worth considering. however, a secure rename requires the checksum, the old filename, and the new filename. That makes it complicated again. Since the case would probably not be too frequent, this could be specified later if needed.
3. How is it ensured that of several files with the same checksum, the wrong file is not deleted or replaced? Since for "delete" the checksum and the path of the already existing file must be specified, a mix-up is impossible.
4. Is it correct that when I pass metadata in baginfo.txt, it overwrites the metadata in the referenced object? If yes, why? Yes, that is so. It simplifies the design to focus only on the payload. By the way, the purpose of differential BagIt is to reduce the cost of complete transfer of all files in case of supplement deliveries. And most of the costs are usually incurred in the transfer of the payload.
   

Warum "AIPUpdate" notwendig ist

In der Diskussion mit Archivaren ist mir in letzter Zeit immer wieder aufgefallen, dass diese mit dem Begriff "AIPUpdate" nichts anzufangen wissen und daher auch nicht verstehen, warum aus der Sicht von Bibliotheken das Thema "AIPUpdate" in der Überarbeitung des OAIS Referenzmodells mit aufgenommen werden sollte.

Klassische Archive

Ein klassisches Archiv arbeitet nach dem Provinienzprinzip, d.h. Archivalien werden nach ihrer Herkunft bzw. Entstehung geordnet. Meist erfolgt diese Ordnung in Form von Akten durch die schriftgutbildende Behörde. Wenn dieses Schriftgut an das Archiv übergeben wird, so handelt es sich dabei um abgeschlossene Dokumente.

Aus Sicht der klassischen Archive sind Änderungen am Archivgut nicht mehr zu erwarten. Diese Überzeugung hat sich auch im Bereich der digitalen Langzeitarchive erhalten.

Bibliotheken und Museen

Anders die Situation in den Bibliotheken und Museen. Diese arbeiten in aller Regel nach dem Pertinenzprinzip, d.h. der Ordnung nach Sachgruppen. Durch die damit einhergehende unterschiedliche Erschließung nach Sachgruppen können Dokumente zu unterschiedlichen Zeiten unterschiedlich gut tiefenerschlossen sein. Die Erschließung ist auch nicht immer perfekt, weil für viele Quellen bestimmte Informationen erst nach und nach durch die Geschichtswissenschaften ermittelt werden können.

Hinzu kommt, dass durch die schiere Menge von Digitalisaten allein im Projekt VD18 Fehler in der Digitalisierung entstehen, die nicht immer sofort auffallen.

Außerdem müssen Bibliotheken und v.a. auch Museen digitale Dokumente (z.B. elektronische Installationen) schon heute noch während der eigentlichen Lebenszeit langzeitarchivieren.

All diese Punkte führen dazu, dass im Gegensatz zu Archiven die Langzeitarchivierung in diesem Bereich mit teilweise unvollständigen, sich noch ändernden Dokumenten oder Dokumententeilen zu tun hat.

Mit dem AIPUpdate ist es möglich, zu einem bereits im Langzeitarchiv gesicherten Stand nachträglich eine vergessene Seite hinzuzufügen, einen Fehler zu korrigieren oder Metadaten zu ergänzen.

Prinzipien AIPUpdate

Damit AIPUpdate funktioniert, bedarf es in Langzeitarchivsystemen der Einhaltung folgender Prinzipien:

1. Saubere Verwaltung eines persistenten Identifiers für die korrekte Zuordnung des Update zu im Langzeitarchiv befindlichen Vorgang
2. Versionsverwaltung der AIPs im Langzeitarchiv
3. Verbot der Löschung von "alten" AIPs (damit alle Versionen nachvollziehbar bleiben)

Da ein AIPUpdate immer auch eine Belastung für das Langzeitarchivsystem darstellt (z.B. Schreib-Lesevorgänge auf Bandspeichern, aber auch durch das Processing), sollten solche Operationen möglichst gebündelt ausgeführt werden.