Some thoughts about a minimalistic Archival Information System, part 2

 In the last post I explained some basic terms. Now it's time for the real thing. 

Choosing the right format

The first question, what should the information packages (SIP, AIP, DIP) look like? It is important that they are easy to process, easy to understand and easy to expand. Fortunately, there is RFC8493 that has the solution ready for us: BagIt.

 In (1) we store the metadata, in (2) there is space for our payload. BagIt is simple, it is a definition of a directory structure and some files that take over certain functions. Very interesting for us, if we want to store digital objects, we can store them in the BagIt payload. We can take over this area completely unchanged when processing a SIP and creating the AIP. The same is possible later when creating the DIPs from the AIPs. BagIt gives a lot of freedom. To limit ourselves, we choose UTF-8 for all metadata and text files.And we don't use fetch bags. Since BagIt is now standardized, we use version 1.

 

Metadata and AIP update considerations

 

Many AIS systems are insufficiently prepared for metadata and AIP updates. In my experience, it is important to think about how and which data is updated and what the consequences are. In order to enable the producer to submit supplements, these must be clearly assigned to an existing AIP. Either you give the producer back an ID for his first recording. This is not a good choice because the process then has a strong coupling and internals are exposed to the outside world. In addition, if a producer wants to change the AIS, there can be collisions. A better choice is to tell the producer to choose a unique ID for your data yourself and transmit it in your SIPs. Internally, we would then use these to search for the appropriate AIPs. The ID is called "ExternalID" and is the base for our internal MAIS-AIP-ID. More on that later.

In the last post I already mentioned that we have to think about the topic of versioning of AIPs. Not only because of the metadata or AIP updates, but also in the case of a PP&A, i.e. format migration. A simple idea is to introduce linked lists.

This allows us to easily implement the functionality of rolling back an AIP version as well.

A new AIP points to the predecessor in which the new version receives a reference entry in the "bag-info.txt":

 

• 'MAIS-previous-AIP' - contains AIP-ID of the current AIS (MAIS-AIP-ID)
  
• 'MAIS-migrated-AIP' - contains AIP-ID of the previous AIS if AIP was migrated from there
  
• 'MAIS-origin-AIS' - contains identifiers of the previous AIS from where the AIP was migrated

 The last two keys are optional and only needed if AIP-AIP-Transfer is needed to move digital objects from one archival information system to another.

Draft for a differential BagIt

The problem

BagIt (RFC 8493) forms the basis for Submission Information Packages (SIP) and Archival Information Packages (AIP) in many digital archives.
Especially in the library environment, it is necessary to support supplemental submissions in the Archival Information System (AIS) software. Supplements may be limited to metadata or may add new files, remove existing files, or replace existing files.
Unfortunately, there is no way to implement a differential SIP cleanly and easily in the BagIt specification.

The constraints

A design of a differential BagIt (dBagIt) should meet the following conditions:

1. existing BagIt should not be touched
2. it should be based on the BagIt structure so that the conversion effort is minimal
3. it should be easy to implement
4. it should support the "add" and "delete" operations
5. the checksum protection should be guaranteed
6. the referenced bag should be specified explicitly

The proposal of dBagIt

The basis is the structure of BagIt. The following are the changes that are mandatory.

Bag Declaration: dbagit.txt

In contrast to 2.1.1 of RFC8493 the filename is dbagit.txt

Payload Manifest

In contrast to 2.1.3 of RFC8493 each line of a payload manifest file MUST be of the form

   sign checksum filepath

where sign is either + for adding a file or - for deleting a file. 

The replacement of files is simulated by one entry each for deleting and adding.

Bag Metadata: bag-info.txt

Additional to RFC8493 the key Updates-External-Identifier becomes mandatory. It is used to reference to the original data object, which will be updated by this dBagIt.

Optional Tag Manifest

The Tag Manifest is similar to RFC8493. 

Although tag manifest files in BagIt could be used to describe additional proprietary subdirectories of a bag not specified in the RFC, it is not defined here to support changes as in the previous section on payload manifest. This facilitates the creation and processing of dBagIts. 

Implementation of the behavior

The implementation must ensure that:

1. the target object referenced by key Updates-External-Identifier exists
2. the dBagIt is valid
3. the add/delete operations are atomic and rollback-able
4. the checksums of files which should be added are correct and part of the current payload
   
5. the checksum of files which should be deleted are similar to the checksum of the files in the referenced digital object
   
6. the files in tag manifests handled correctly if proprietary extensions used
   
7. the metadata content in bag-info.txt replaced previous versions in referenced object completely
   

Future

If there is interest, I would be happy to receive feedback via art1pirat ATgmail.com. Maybe a new RFC can grow out of it.

Alternate consideration

A very simple solution could also be the use of unified 'diff'. This also allows partial changes in files, but would hardly bring any advantages with binary data and is not quite as intuitive for users who are not familiar with IT.

FAQ (Update 2022-05-18)

1. What if "delete" references a non-existing file? The complete operations via differential BagIt should be atomar and consistent. In this case the operations are rollbacked and aborted with an error. This ensure that no unintented updates will be applied.
2. Wouldn't it be nice, to avoid transferring files, to allow a simple rename instead of a replace? This would be worth considering. however, a secure rename requires the checksum, the old filename, and the new filename. That makes it complicated again. Since the case would probably not be too frequent, this could be specified later if needed.
3. How is it ensured that of several files with the same checksum, the wrong file is not deleted or replaced? Since for "delete" the checksum and the path of the already existing file must be specified, a mix-up is impossible.
4. Is it correct that when I pass metadata in baginfo.txt, it overwrites the metadata in the referenced object? If yes, why? Yes, that is so. It simplifies the design to focus only on the payload. By the way, the purpose of differential BagIt is to reduce the cost of complete transfer of all files in case of supplement deliveries. And most of the costs are usually incurred in the transfer of the payload.