Montag, 29. Juni 2026

Umzug

Das Blog ist seit geraumer Zeit vollständig auf meine private Domain umgezogen und unter http://andreas-romeyke.de/blog erreichbar.

Freitag, 25. Oktober 2024

Der Wert von digitalen Objekten - Fragment einer Untätigkeitsbewertung

Als Mitarbeiter eine digitalen Langzeitarchivs ist man Kummer gewohnt und weiß, dass man oft dicke Bretter bohren muss, um Institutionen davon zu überzeugen, dass bestimmte digitale Objekte langzeit gesichert werden sollten.

Besonders schlecht steht es dabei um sogenannte born-digitals, originär digitale Objekte, die bei Verlust unwiederbringlich verloren sind.

Ein fatalistischer Ansatz ist es, "wenn es für die Nachwelt verloren gegangen ist, dann ist es so". Die darwinistische Erweiterung davon lautet "wenn es für die Nachwelt verloren gegangen ist, war es nicht wertvoll genug".

Was bedeutet also "dieses Objekt ist wertvoll"?

Diese Frage ist nicht nur aus versicherungstechnischer Sicht interessant, sondern könnte argumentativ helfen, die Berechtigung für digitale Langzeitarchivierung zu untermauern und Prozessentscheidungen zu versachlichen.

Wie bestimmt man den Wert von etwas, von dem man erst weiß, dass es gebraucht wird, wenn es nicht zugreifbar ist?

Wiederbeschaffung

Für die Wertermittlung gibt es verschiedene Ansätze. Einer wäre der, für die Kosten einer "Wiederbeschaffung". Für Retrodigitalisate würde dies den Kosten entsprechen, die für eine erneute Digitalisierung entstehen würden. Wenn Bücher im Regal fehlen, dann für wie viel Aufwand man für die Beschaffung ausgeben müsste.

Für unbekannte Dateiformate wären dies die Kosten, die für die Analyse des Dateiformates entstehen würden. In dem Zusammenhang bekommt man manchmal die Aussage zu hören: "Im Zweifel setzen wir da einen Doktorrand dran, der das für ein halbes Jahr untersucht".

Im Fall des Siebeck-Nachlass hatten wir es mit Disketten für eine Panasonic-Schreibmaschine zu tun. Diese zu interpretieren erfordert mindestens die Beschaffung einer solchen Schreibmaschine, die zur Zeit in den einschlägigen Portalen nicht zu bekommen ist. Für das reverse-engineering des Datenformates müsste man sicherlich auch nochmal ein halbes Jahr ansetzen.

Nicht entstandene Werke

Ein anderer Ansatz wäre zu ermitteln, wieviele Arbeiten aufgrund des Verlustes nicht zu dem Objekt entstanden sind. Sprich, man schaut sich an, wieviele Artikel, Forschungsarbeiten etc., sich mit einem digitalen Objekt beschäftigen. Gäbe es dieses Objekt nicht, gäbe es auch die darauf aufbauenden Arbeiten nicht.

Wert als reziproke Funktion der Häufigkeit

Aus den vorherigen Überlegungen ergibt sich, dass ein Objekt um so wertvoller ist, desto seltener es ist. Dabei ist "selten" Ausdruck von fehlender Redundanz.

Wenn ein Buch in einer Auflage von 10.000 Stück mal 10€ gekostet hat, und jetzt nur noch 2 Exemplare existieren, kann man dann

10€/2 * 10000 = 50000€

rechnen?

Aussicht

Hmmm. Sind diese Ansätze sinnvoll? So richtig nicht. In der Praxis würde man sich vielleicht typische Vertreter bestimmter Objekttypgruppen herausziehen und einen Wert bestimmen.

Wie auch immer, wenn jemand eine Idee hat, wie man es besser machen könnte, immer her damit!

Donnerstag, 16. März 2023

Some thoughts about a minimalistic Archival Information System, part 3

In the last blog post we considered what the basic structure of the information packages should look like and how we will deal with versioning. In the following I would like to describe further cornerstones of a minimalistic archival information system. These will then form the basis for a first implementation, which would go beyond the scope of this blog. As soon as there is news worth reporting, I will announce it here.

Data management

Other archival information systems sometimes make it too easy for themselves and use a database to manage information about the AIPs in the archive. In principle there is nothing wrong with this, but it often seems that it is forgotten that a basic principle of information packages is the intellectual unit (IE) of data and metadata. What does this mean? The idea is that an IE should be able to stand on its own at all times. Following this principle has two consequences. First, hierarchically nested IEs cannot exist unless self-contained IEs are encapsulated like a box of boxes. In other words: IEs that only contain references to other IEs are not possible because they would not be viable on their own.

The second consequence is that all metadata must always be in a consistent state, regardless of the state of the archive information system. In other words, there must be no contradictions between the information stored in the AIP and the information in the system's database.

Why is this important? The Archival Information Packages are ultimately the time capsules that will outlast the Archive. If everything breaks, but a copy of an AIP is still found on tape, it contained all the information needed to interpret the data to be preserved.

So for the management of the AIPs we define the following:

the AIP is the basis for everything. If there are inconsistencies in the archival information system, we first ask the AIPs.
In order to speed up the processing, we can use a database. But then there must be a way to generate the database from the AIPs.
If the AIP is the basis for everything, then we need a mechanism that ensures that if there are errors in the creation of an AIP or in the creation of a new AIP version, these can be rolled back.
The archive can and should only assume responsibility for the data entrusted to it if an AIP or a new AIP version could be successfully generated.

Things that make life easier

What has proven to be very helpful is the following:

We should only allow 1:1 mappings. This means that a SIP contains exactly one digital object
We do without nested IEs.
We ignore for now that copy operations are expensive. It follows that AIP updates always consist of SIPs with complete data and metadata.

Architectural decisions

A minimalist archival information system (MAIS) should have the following properties:

Implemented as open source for study, improvement and reuse
Command line oriented so there is a clear interface.
Concentration on the essentials, therefore no routing, but suitable for parallel use.
Fast and small enough not to waste resources.
Avoiding XML to keep code simple and metadata human-readable
BagIt as base for SIPs, AIPs and DIPs
Preservation Planning and Action as an external operation on a set of AIPs
Implementation in a programming language that can be used for all common operating systems without contortions.

And now?

I plan to tackle the programming in the coming weeks and months. I will probably not go into detail about the individual steps of programming here. As soon as there is something presentable, I'll let you know. Otherwise let me know what your experiences are, which details are important to you with an AIS, especially if it should be particularly lightweight.

Dienstag, 7. März 2023

Some thoughts about a minimalistic Archival Information System, part 2

In the last post I explained some basic terms. Now it's time for the real thing.

Choosing the right format

The first question, what should the information packages (SIP, AIP, DIP) look like? It is important that they are easy to process, easy to understand and easy to expand. Fortunately, there is RFC8493 that has the solution ready for us: BagIt.

In (1) we store the metadata, in (2) there is space for our payload. BagIt is simple, it is a definition of a directory structure and some files that take over certain functions. Very interesting for us, if we want to store digital objects, we can store them in the BagIt payload. We can take over this area completely unchanged when processing a SIP and creating the AIP. The same is possible later when creating the DIPs from the AIPs. BagIt gives a lot of freedom. To limit ourselves, we choose UTF-8 for all metadata and text files.And we don't use fetch bags. Since BagIt is now standardized, we use version 1.

Metadata and AIP update considerations

Many AIS systems are insufficiently prepared for metadata and AIP updates. In my experience, it is important to think about how and which data is updated and what the consequences are. In order to enable the producer to submit supplements, these must be clearly assigned to an existing AIP. Either you give the producer back an ID for his first recording. This is not a good choice because the process then has a strong coupling and internals are exposed to the outside world. In addition, if a producer wants to change the AIS, there can be collisions. A better choice is to tell the producer to choose a unique ID for your data yourself and transmit it in your SIPs. Internally, we would then use these to search for the appropriate AIPs. The ID is called "ExternalID" and is the base for our internal MAIS-AIP-ID. More on that later.

In the last post I already mentioned that we have to think about the topic of versioning of AIPs. Not only because of the metadata or AIP updates, but also in the case of a PP&A, i.e. format migration. A simple idea is to introduce linked lists.

This allows us to easily implement the functionality of rolling back an AIP version as well.

A new AIP points to the predecessor in which the new version receives a reference entry in the "bag-info.txt":

'MAIS-previous-AIP' - contains AIP-ID of the current AIS (MAIS-AIP-ID)
'MAIS-migrated-AIP' - contains AIP-ID of the previous AIS if AIP was migrated from there
'MAIS-origin-AIS' - contains identifiers of the previous AIS from where the AIP was migrated

The last two keys are optional and only needed if AIP-AIP-Transfer is needed to move digital objects from one archival information system to another.

Donnerstag, 2. März 2023

Some thoughts about a minimalistic Archival Information System, part 1

Many of those who are dealing with the digital preservation of objects for the first time and who work in small memory organizations are often helpless in the face of the vast range of functions and requirements of current archival information systems.
Students of library or archival science often appear to be similarly overwhelmed when they are supposed to learn what constitutes archival software.

This has motivated me to write down thoughts on a minimalist archive information system. Because it really doesn't need much.

The basic terms

An archive essentially has three roles: the submitter, called the producer, the user, also called the consumer, and the problem solver who maintains the archive, also called the technical analyst.

When digital objects are transferred to the archive, it is called the ingest process. When they are requested from the archive, then this is the access process.

The digital objects to be preserved are provided with all the necessary information for the archive ingest and are packaged in a predefined structure. This is called a Submission Information Package (SIP). You can actually imagine this just like in real life. For example, if you want to store a vase, you put it in a box, label it and put it on a shelf.

In the archive it is checked whether (allegorically) the vase is in the box and intact, and if there is a stamp and signature that says that the content of the package is indeed a vase. A file number and a storage location is assigned and the box goes sealed and neatly labeled on the shelf. The "box" is called Archival Information Package (AIP). With the seal, the archive takes responsibility.

At some point, when the user would like to see the vase from the archive again, the archive would process the request and send the vase and accompanying information to the user. This is then called a Dissemination Information Package (DIP).

In addition to this simple "I store something safely and retrieve it again at some point" approach, an archive fulfills another task that is not so obvious: it ensures that objects entrusted to it are kept usable.

What does that mean in the digital world?
If it is possible in principle to store a digital object securely with bit accuracy, even over a very long period of time (bitstream archival), it still can age because the environment for using this object is no longer available.

There are essentially three concepts for keeping digital objects usable (content preservation): hardware museum, emulation or format migration.

Hardware museums (e.g. a slot machine museum) try to keep old equipment running in a controlled environment. To do that, they have to build up a stock in time and build up knowledge on how to maintain and repair these devices.

With an emulation, I try to recreate the environment for the digital object so that it feels at home and doesn't notice any difference from the previous, real world. A very good example of emulators is e.g. MAME, but also various others, the e.g. retro computers like the Amiga or C64, so that old programs from their time can run on them. Here, too, I need knowledge about what the environment to be emulated looks like and how I can recreate it with today's means.

When migrating the format, I try to find a new form that retains the essential properties (significant properties) in good time and to transfer files from a digital object to a newer data format.

From this point onwards, it is assumed that this is the preferred way of maintaining usability.

It follows that an Archival Information System (AIS) must be able to support this process of format migration. The process (also called Preservation Planning and Action) results in a new version of the Archival Information Package being created. The AIS must be able to manage this.

That would basically be all there is to Archival Information Systems if it hadn't been for the librarians.

Unlike archivists, where a record is complete and closed, librarians understand the concepts of supplements and metadata submissions. A page that has fallen out has turned up here, a letter has been discovered there in an estate, or it has dawned on some people that there is now money for costly in-depth indexing. Ergo, librarians expect people to think about how to handle metadata and data updates on existing AIPs (called metadata update and AIP update). This is not trivial, since some AIPs are also very large and you want to avoid pointless copying. For such an update, we also need a good way for producers to tell the archive which AIP needs to be added or updated.

However, AIPs are already versioned in the case of format migration, the same can be used here as well. Any change to the AIP creates a new version of an AIP. And so that you can't accidentally break anything, you should always be able to go back to an old version. And since that is also error-prone, the result of the rollback process will simply be a new version.

That's it. It's nothing more. Easy, isn't it?

Donnerstag, 12. Mai 2022

Draft for a differential BagIt

The problem

BagIt (RFC 8493) forms the basis for Submission Information Packages (SIP) and Archival Information Packages (AIP) in many digital archives.
Especially in the library environment, it is necessary to support supplemental submissions in the Archival Information System (AIS) software. Supplements may be limited to metadata or may add new files, remove existing files, or replace existing files.
Unfortunately, there is no way to implement a differential SIP cleanly and easily in the BagIt specification.

The constraints

A design of a differential BagIt (dBagIt) should meet the following conditions:

1. existing BagIt should not be touched
2. it should be based on the BagIt structure so that the conversion effort is minimal
3. it should be easy to implement
4. it should support the "add" and "delete" operations
5. the checksum protection should be guaranteed
6. the referenced bag should be specified explicitly

The proposal of dBagIt

The basis is the structure of BagIt. The following are the changes that are mandatory.

Bag Declaration: dbagit.txt

In contrast to 2.1.1 of RFC8493 the filename is dbagit.txt

Payload Manifest

In contrast to 2.1.3 of RFC8493 each line of a payload manifest file MUST be of the form

   sign checksum filepath

where sign is either + for adding a file or - for deleting a file.

The replacement of files is simulated by one entry each for deleting and adding.

Bag Metadata: bag-info.txt

Additional to RFC8493 the key Updates-External-Identifier becomes mandatory. It is used to reference to the original data object, which will be updated by this dBagIt.

Optional Tag Manifest

The Tag Manifest is similar to RFC8493.

Although tag manifest files in BagIt could be used to describe additional proprietary subdirectories of a bag not specified in the RFC, it is not defined here to support changes as in the previous section on payload manifest. This facilitates the creation and processing of dBagIts.

Implementation of the behavior

The implementation must ensure that:

the target object referenced by key Updates-External-Identifier exists
the dBagIt is valid
the add/delete operations are atomic and rollback-able
the checksums of files which should be added are correct and part of the current payload
the checksum of files which should be deleted are similar to the checksum of the files in the referenced digital object
the files in tag manifests handled correctly if proprietary extensions used
the metadata content in bag-info.txt replaced previous versions in referenced object completely

Future

If there is interest, I would be happy to receive feedback via art1pirat ATgmail.com. Maybe a new RFC can grow out of it.

Alternate consideration

A very simple solution could also be the use of unified 'diff'. This also allows partial changes in files, but would hardly bring any advantages with binary data and is not quite as intuitive for users who are not familiar with IT.

FAQ (Update 2022-05-18)

What if "delete" references a non-existing file? The complete operations via differential BagIt should be atomar and consistent. In this case the operations are rollbacked and aborted with an error. This ensure that no unintented updates will be applied.
Wouldn't it be nice, to avoid transferring files, to allow a simple rename instead of a replace? This would be worth considering. however, a secure rename requires the checksum, the old filename, and the new filename. That makes it complicated again. Since the case would probably not be too frequent, this could be specified later if needed.
How is it ensured that of several files with the same checksum, the wrong file is not deleted or replaced? Since for "delete" the checksum and the path of the already existing file must be specified, a mix-up is impossible.
Is it correct that when I pass metadata in baginfo.txt, it overwrites the metadata in the referenced object? If yes, why? Yes, that is so. It simplifies the design to focus only on the payload. By the way, the purpose of differential BagIt is to reduce the cost of complete transfer of all files in case of supplement deliveries. And most of the costs are usually incurred in the transfer of the payload.

Freitag, 3. Dezember 2021

Detectorist - Part two "A crumb of knowledge"

A crumb of knowledge

In the first part I described how I came to know how to read the floppy disks (using kryoflux). Now I would like to give an intermediate state about the floppy disk format of the Panasonic typewriter - in the quiet hope that someone could uncover the last secret.

I found the most important clue while researching a successor model - the Panasonc KX-W1000. I stumbled across the follow old blog post https://surrey.lug.org.uk/panasonic-kx-w1000.

My findings

Even if it didn't lead to a full success, there were some interesting insights. The floppy image is strongly related to FAT12.

Here is my summary.

The filesystem is based on FAT12 with proprietary extensions.

Header / MBR

The first bytes are: 0x00 00 00 4B 58 2D 57 31 35 31 30 20 31 2E 30 30 20, which corresponds to the string "KX-W1510 1.00" from the third byte onwards.

The first 256 bytes are very similar to a MBR of old DOS floppies:

0000:0000 | 00 00 00 4B  58 2D 57 31  35 31 30 20  31 2E 30 30 | ...KX-W1510 1.00
0000:0010 | 20 F9 00 00  00 00 00 00  00 00 00 00  00 00 00 00 |  ù..............
0000:0020 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:0030 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:0040 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:0050 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:0060 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:0070 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:0080 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:0090 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:00A0 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:00B0 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:00C0 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:00D0 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:00E0 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................
0000:00F0 | 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 | ................

FATs

There are two equal blocks which probably represent FATs, once at address 0x200:

0000:0200 | F9 FF FF 03  40 00 05 B0  00 07 80 00  09 A0 00 FF | ùÿÿ.@..°..... .ÿ
0000:0210 | FF FF 0D E0  00 0F 00 01  FF 8F 01 13  40 01 15 60 | ÿÿ.à....ÿ...@..`
0000:0220 | 01 17 F0 FF  19 90 02 1B  10 02 1D E0  01 1F 00 02 | ..ðÿ.......à....
0000:0230 | FF 2F 02 23  F0 FF 25 60  02 2D 80 02  2C A0 02 2B | ÿ/.#ðÿ%`.-.., .+
0000:0240 | F0 FF FF EF  02 35 00 03  31 20 03 33  F0 FF 36 F0 | ðÿÿï.5..1 .3ðÿ6ð
0000:0250 | FF 37 80 03  39 F0 FF 3B  C0 03 3D E0  03 FF 0F 04 | ÿ7..9ðÿ;À.=à.ÿ..
0000:0260 | 41 20 04 43  F0 FF 45 60  04 47 80 04  49 F0 FF 4B | A .CðÿE`.G..IðÿK
0000:0270 | C0 04 4D E0  04 4F F0 FF  51 20 05 53  40 05 55 F0 | À.Mà.OðÿQ .S@.Uð
0000:0280 | FF 57 80 05  59 A0 05 5B  F0 FF 5D E0  05 69 B0 07 | ÿW..Y .[ðÿ]à.i°.
0000:0290 | 7F 20 06 63  40 06 65 F0  FF 6E 80 06  6B A0 06 FF | . .c@.eðÿn..k .ÿ
0000:02A0 | CF 06 6D F0  FF 7E 00 07  71 20 07 73  40 07 FF 6F | Ï.mðÿ~..q .s@.ÿo
0000:02B0 | 07 77 80 07  79 A0 07 FF  CF 07 7D 60  08 80 30 08 | .w..y .ÿÏ.}`..0.
0000:02C0 | 81 20 08 FF  4F 08 85 80  08 87 F0 FF  FF AF 08 8B | . .ÿO.....ðÿÿ¯..
0000:02D0 | F0 08 8D E0  08 90 20 09  91 40 09 93  F0 FF FF 6F | ð..à.. ..@..ðÿÿo
0000:02E0 | 09 A2 F0 09  99 A0 09 9B  C0 09 9D F0  FF A1 00 0A | .¢ð.. ..À..ðÿ¡..
0000:02F0 | B2 60 0A A3  40 0A A5 F0  FF AC 80 0A  A9 A0 0A AB | ²`.£@.¥ðÿ¬..© .«
0000:0300 | F0 FF AD E0  0A AF 00 0B  B1 B0 0B BA  40 0B B5 C0 | ðÿ.à.¯..±°.º@.µÀ
0000:0310 | 0C B7 80 0B  B9 60 0C C3  C0 0B BD E0  0B BF 00 0C | .·..¹`.ÃÀ.½à.¿..
0000:0320 | C1 20 0C CA  40 0C C5 80  0C C7 90 0C  E8 00 0D CB | Á .Ê@.Å..Ç..è..Ë
0000:0330 | F0 FF CD E0  0C CF B0 0D  D1 50 0D D3  40 0D DE F0 | ðÿÍà.Ï°.ÑP.Ó@.Þð
0000:0340 | FF D7 E0 0E  D9 70 0E E2  C0 0D DD F0  FF DF 00 0E | ÿ×à.Ùp.âÀ.Ýðÿß..
0000:0350 | E1 50 0E E3  40 0E E6 90  0E FB C0 0E  FF AF 0E EB | áP.ã@.æ..ûÀ.ÿ¯.ë
0000:0360 | F0 FF ED 60  0F EF 00 0F  F1 20 0F F3  40 0F F5 F0 | ðÿí`.ï..ñ .ó@.õð
0000:0370 | FF F7 80 0F  F9 A0 0F FF  CF 0F FD E0  0F 08 01 00 | ÿ÷..ù .ÿÏ.ýà....

once at 0x800:

0000:0800 | F9 FF FF 03  40 00 05 B0  00 07 80 00  09 A0 00 FF | ùÿÿ.@..°..... .ÿ
0000:0810 | FF FF 0D E0  00 0F 00 01  FF 8F 01 13  40 01 15 60 | ÿÿ.à....ÿ...@..`
0000:0820 | 01 17 F0 FF  19 90 02 1B  10 02 1D E0  01 1F 00 02 | ..ðÿ.......à....
0000:0830 | FF 2F 02 23  F0 FF 25 60  02 2D 80 02  2C A0 02 2B | ÿ/.#ðÿ%`.-.., .+
0000:0840 | F0 FF FF EF  02 35 00 03  31 20 03 33  F0 FF 36 F0 | ðÿÿï.5..1 .3ðÿ6ð
0000:0850 | FF 37 80 03  39 F0 FF 3B  C0 03 3D E0  03 FF 0F 04 | ÿ7..9ðÿ;À.=à.ÿ..
0000:0860 | 41 20 04 43  F0 FF 45 60  04 47 80 04  49 F0 FF 4B | A .CðÿE`.G..IðÿK
0000:0870 | C0 04 4D E0  04 4F F0 FF  51 20 05 53  40 05 55 F0 | À.Mà.OðÿQ .S@.Uð
0000:0880 | FF 57 80 05  59 A0 05 5B  F0 FF 5D E0  05 69 B0 07 | ÿW..Y .[ðÿ]à.i°.
0000:0890 | 7F 20 06 63  40 06 65 F0  FF 6E 80 06  6B A0 06 FF | . .c@.eðÿn..k .ÿ
0000:08A0 | CF 06 6D F0  FF 7E 00 07  71 20 07 73  40 07 FF 6F | Ï.mðÿ~..q .s@.ÿo
0000:08B0 | 07 77 80 07  79 A0 07 FF  CF 07 7D 60  08 80 30 08 | .w..y .ÿÏ.}`..0.
0000:08C0 | 81 20 08 FF  4F 08 85 80  08 87 F0 FF  FF AF 08 8B | . .ÿO.....ðÿÿ¯..
0000:08D0 | F0 08 8D E0  08 90 20 09  91 40 09 93  F0 FF FF 6F | ð..à.. ..@..ðÿÿo
0000:08E0 | 09 A2 F0 09  99 A0 09 9B  C0 09 9D F0  FF A1 00 0A | .¢ð.. ..À..ðÿ¡..
0000:08F0 | B2 60 0A A3  40 0A A5 F0  FF AC 80 0A  A9 A0 0A AB | ²`.£@.¥ðÿ¬..© .«
0000:0900 | F0 FF AD E0  0A AF 00 0B  B1 B0 0B BA  40 0B B5 C0 | ðÿ.à.¯..±°.º@.µÀ
0000:0910 | 0C B7 80 0B  B9 60 0C C3  C0 0B BD E0  0B BF 00 0C | .·..¹`.ÃÀ.½à.¿..
0000:0920 | C1 20 0C CA  40 0C C5 80  0C C7 90 0C  E8 00 0D CB | Á .Ê@.Å..Ç..è..Ë
0000:0930 | F0 FF CD E0  0C CF B0 0D  D1 50 0D D3  40 0D DE F0 | ðÿÍà.Ï°.ÑP.Ó@.Þð
0000:0940 | FF D7 E0 0E  D9 70 0E E2  C0 0D DD F0  FF DF 00 0E | ÿ×à.Ùp.âÀ.Ýðÿß..
0000:0950 | E1 50 0E E3  40 0E E6 90  0E FB C0 0E  FF AF 0E EB | áP.ã@.æ..ûÀ.ÿ¯.ë
0000:0960 | F0 FF ED 60  0F EF 00 0F  F1 20 0F F3  40 0F F5 F0 | ðÿí`.ï..ñ .ó@.õð
0000:0970 | FF F7 80 0F  F9 A0 0F FF  CF 0F FD E0  0F 08 01 00 | ÿ÷..ù .ÿÏ.ýà....
0000:0980 | 00 00 00 00  00 00 00 00  00 00 00 00  FF 0F 00 00 | ............ÿ...

Umlauts ans Special chars

Umlauts and Special chars are mapped as follows:

ä → 0x7b
ö → 0x7c
ü → 0x7d
Ä → 0x5b
Ö → 0x5c
Ü → 0x5d
ß → 0x85
hyphen → 0xbc

Open Questions

What is still completely unclear is how the FATs are constructed. They do look like FAT12 entries, the first bytes 0xf9 0xff 0x03... and the frequently occurring 0xff suggest this, yet there seems to be no connection between the addresses of the text fragments in the image and the FAT byte sequences.

In the directory entries everything points to the fact that byte 26 indicates the start cluster and bytes 28-29 the file size, the connection with the FAT and the actual offset (or cluster) to the data I could not decipher yet.

The meaning of offset 0x100 is unclear.

If you have any ideas how to read the FATs, or how to interpret the bytes 26, 28-29 of the directory entries, or what the cluster size should be, feel free to write me.

If you are the owner of such an old typewriter, it would be helpful to have a clean-room floppy copy, i.e. a freshly formatted floppy with a small test text, so that I can reverse engineer the data format even better.

Just contact me at art1piratatgoogledotcom

Supportive Links

https://archive.org/details/MSXTechnicalDataBook/page/n269/mode/2up

https://github.com/Konamiman/MSX2-Technical-Handbook/blob/master/md/Chapter3.md#3--structure-of-disk-files

https://manualsbrain.com/ja/products/panasonic-kx-w1510/

Thanks

my thanks goes to

Panasonic Museum
David Murray, the 8-bit Guy
John Wash, for his analysis of kx-w1000 floppies, https://surrey.lug.org.uk/panasonic-kx-w1000
Jason Kleiner
to the Developers and Maintainers of Okteta, Kaitai, wxHexEditor, Debian, Kryoflux…

Sonntag, 28. November 2021

Detectorist - Part one "First indications"

First indications

In an estate there are several CDROMs, DVDs and especially floppy disks. We were able to read most of them with Linux, including the floppy disks. Only on the last 8 floppy disks did we have a hard time.

On one of the floppy disks a small inscription peeked out, referring to a Panasonic electronic typewriter. There was none in the estate, no other information was available.

Eight floppy disks, 3,5", double density, not readable.

Time passed, constantly haunted by the voice in my mind: "There's something on the disks, only what?"

We managed to purchase a Kryoflux controller (see https://kryoflux.com/, there are also free opensource alternatives). This is a special disk controller that allows you to record the magnetic flux as the read heads move over the disk.

After the first attempts, I was able to create an image file with the following command:

./dtc -fIMAGEFILE -dd1 -g2 -i4

The options mean:

"-dd1" - double density
"-g2" - double sided
"-i4" - MFM sector image 40/80+ tracks

A look at the image using the hex editor showed that I was right with my intention. After the first three zero bytes, the string "KX-W1510 v1.00" followed (and to my happy surprise, a lot of readable text fragments).

Yep, there is exactly one electronic typewriter series from Panasonic.

Disillusionment

I was able to find a manual at https://manualsbrain.com/en/manuals/1814281/. And yes, the machine used 3.5" floppy disks, double sided, double density with a capacity of 713,000 characters, but unfortunately without an exact description of the disk format and the file system.

I then contacted Panasonic support - no success. I started researching patent databases in Japan, the USA and Germany - nothing. I wrote to the Panasonic museum in Japan, but unfortunately they could not help me.

A proprietary disk format, which was forgotten after 30 years.

In the next part I report what I could find out about the disk system of the Panasonic typewriter KX-W1510, and where I (still) fail...

Donnerstag, 1. April 2021

Backup is digital long-term preservation!

Exponential growth

https://www.statista.com/chart/17727/global-data-creation-forecasts/

An important observation is that the number of files produced each year continues to increase worldwide (see https://en.wikipedia.org/wiki/Information_explosion). And with it the number of digital objects increases in the same measure, for which we must decide: Keep or throw away?

The truth is, the discard scenario becomes the more likely one with each passing year.

Magnificent diversity

Another observation is that about 90 new file formats are added every year.
And the file formats that are being dropped are already in place.

The truth is, no one can build up format knowledge for this yet.

A fuzzy concept

When talking to colleagues, the topic of validation does not play a role. For one thing, no one is clear about what "valid" means. Valid against a specification? Valid against a profile? Valid because it can be opened by programs? On the other hand, nothing happens after that. If a file is broken, it is still archived. If it is not broken, fine.

The truth is, validation is useless.

Success factors

Do you know how the success of digital preservation is measured? I'll tell you, in terabytes per year. If the numbers go up, that's a good thing to sell to politicians. Whether it was difficult to prepare digital objects for long-term availability doesn't matter. Whether born-digitals are more at risk, never mind.

Is that the truth?

Overrated

It used to be said that long-term digital archiving could only be handled by organizations with a minimum of resources. Look around and you'll find dozens of one-man orchestras and part-time archives. And do you think that as the amount of data increases, so do the human resources? Oh, come on!

You know the truth!

That's too exhausting

If you've ever heard of format migration as a principle of long-term preservation, you've read in textbooks phrases like

To ensure format migration, the significant properties of groups of objects that must be preserved must be determined.

Have you ever seen an archive that has actually determined and documented significant properties?

The truth is, significant properties are determined after the fact from technical metadata.

Summary

So what is digital long-term preservation? Only an expensive backup.

Mittwoch, 27. Januar 2021

Impossible - or how I learned to read data storage media at the speed of light and what it's good for

When I receive data carriers from an inheritance, I want to get a quick overview of what is on the floppy disk, the CDROM, the USB stick or the hard disk drive so that I can look at the interesting things first.

But I only know what is there when I read the media, right? A typical chicken and egg problem.

https://openclipart.org/detail/212857/sci-fi-scanner-device

I discovered the crucial clue to the solution in a 2014 talk by Simon Garfinkel "Digital Forensics Innovation: Searching A Terabyte of Data in 10 minutes" (http://simson.net/ref/2014/2014-02-21_RPI_Forensics_Innovation.pdf)

What is Random Sampling?

Random sampling is nothing more than looking at only every n-th part of a total set and inferring the big picture.

To find out what is on a medium, it would be sufficient to look at random blocks and determine for them, based on their byte structure, whether they fall into the categories "empty", "random", "text", "video" or "undef".

Exactly this approach is implemented in the Perl module File::FormatIdentification::RandomSampling, which can be found on CPAN under https://metacpan.org/pod/File::FormatIdentification::RandomSampling.

The category "empty" is dominated by sequences of zero bytes, in the category "random" the byte values are almost equally distributed, in the category "text" values for the characters "a-z" from the ASCII character set appear frequently, "video" contains frequent byte sequences resulting from the basic structure of MPEG. And under "undef" everything else is subsumed.

Example

The above Perl module contains the program crazy_fast_image_scan.pl. The following simple call:

perl -I lib bin/crazy_fast_image_scan.pl --percent=0.000001 --image=/dev/mapper/laptop--vg-home

provides the following output:

Scanning Image /dev/mapper/laptop--vg-home with size 728982618112, checking 1423 sectors
scanning [...]
Estimate, that the image '/dev/mapper/laptop--vg-home'
has percent of following data types:
    44.6% random/encrypted/compressed
    35.6% undef
    11.0% empty
    5.4% video/audio
    3.5% text

The complete output is even more extensive. It is important to note that the examined partition was 668GB in size and was scanned in just 15s.

Limits

Importantly, the output provides only a rough estimate of what might be on the media. The choice of the sample size (here: via the --percentage parameter) determines the informative value of the estimate, as well as the duration until a result can be delivered.

More ideas

In the above module, I have implemented an experimental output of the MIME-Types potentially present on the media. This is not very stable yet and needs more work, but it can help to estimate even better whether the files on a disk are interesting enough to prioritize it. Here is an example output:

The next mimetype estimation is experimental and needs further work:
    87.9% unknown
    3.5% application/pdf
    1.1% video/quicktime
    0.8% image/gif
    0.8% text/java
    0.7% application/msword
    0.6% text/markdown
    0.6% application/vnd.openxmlformats-officedocument.wordprocessingml.document
    0.6% application/xml
    0.4% application/msaccess
    0.4% application/navimap
    0.4% application/rtf
    0.3% image/png
    0.2% application/arj
    0.1% application/vnd.ms-powerpoint
    0.1% text/html

The approach is to determine the MIME-Type of the files for a test corpus using other tools, determine typical bytegram values and pass the whole thing to a decision tree learner. If you are interested, you are welcome to contribute to the module.

Happy scanning!

Montag, 10. August 2020

It is nonsense to consider significant properties only at file level

As it looks, most archives raise significant properties at the file level (by the way, they often mean technical properties, which is not the same. But this is a topic for another blog post). But this is insufficient and I will give two examples.

Example 1 - Retro-digitised material

If monographs are scanned, as we do in-house, in order to preserve the originals and make them accessible to users, images are created.If you look at these image files, you can determine the following significant characteristics

readable
accessible for OCR analysis
reproducible
maybe even true to color

These properties can then be used to define technical parameters that can be found in certain requirement profiles and can lead, for example, to the recommendation of the TIFF file format.

In the above consideration, the list of the significant property "the order of the scans should correspond to the original" (pagination) is missing. This property could be implemented by combining all scan pages into one file format, e.g. as BigTIFF or PDF/A. However, there may be good reasons not to include all pages in one file. What next? The remaining option is to add a file describing the structure of the digitized material in addition to the TIFF files. This can be a METS XML file, for example. METS is a good choice because it was created for this very purpose. Hmmm, is METS not a metadata format? And doesn't metadata belong outside of the payload? And isn't METS used by several archive information systems to map the AIPs? So can I not pack the structuring data into it?

Stop!

It is true, METS is a metadata format. And it is true that METS is often used to describe container structures in SIPs or AIPs. But we have to distinguish between metadata describing the IE (i.e. the payload) and metadata inherently belonging to the payload. This is not easy, but here the significant properties help us: If the METS is used, as in our example, to represent the significant property "pagination", then the METS is part of the IE, otherwise it is not.

Now you might be tempted to get sloppy and just put the "pagination" into the METS of the AIP. Is that a good idea? No. Because IE should be kept available and usable. The AIP should only contain the metadata necessary to ensure availability. But when a user later accesses the payload via DIP, he should have everything together, i.e.: an intellectual unit as it was actually intended. This is the principle of independence.

I admit that sounds abstract and difficult. But let us try an analogy. If I have loose pages where the order is important, then the order is important, whether the page is archived or not. For example, I tie them to a book or use other techniques. This is my intellectual unit that I want to archive. I put the whole thing in a box and write on it what is in it and what happened to the box or the content during archiving. This is then my AIP. If I want to hand over the contents of this box to someone later, they don't necessarily have to be interested in what happened to the box, they can take the contents and work with them and know exactly in which order the pages follow each other.

Example 2 - Web page

I would like to present a second example to illustrate another aspect. Let us assume that we are to archive a very specific web page, which for the sake of simplicity consists of an HTML document, CSV files and graphic files. If you look at the web page, there is always a link in the text between one of the CSV files and one graphic file. The assignment could be the visualization of an experiment. It is only important to the department that the values, the textual content and the assignment to the graphic are not lost. Together with the department we determined the significant properties and after a lot of effort we transferred the website (IE) into the long-term archive. After some time we found out that the graphic files were subject to format obsolescence and had to be migrated to a new format. We decide on the new image archive format PNG/A and migrate the old files.

But is this sufficient? No. The HTML document still contains the file name of the old format. Should we change the file name or leave it as it is? The principle of least surprise speaks for "change". But if we change the file names during the migration, we impossibly have to change the file names in the HTML document as well.

Let's summarize

Significant properties belong at the level of IE recorded. They are not file dependent.
Metadata, which is essential to represent the relationship of objects within an IE, is mandatory part of an IE
Format migrations can result in changes to other parts of the IE, even if they are not migrated themselves
Metadata and data that are inside an IE must never refer to data or metadata outside
Metadata outside of an IE, however, may already reference metadata and data of an IE.

Whew, that was a lot of thinking, but I hope it was worth thinking about it.

Mittwoch, 22. Juli 2020

Format recognition, new analysis options?

Previous work

In an older article (see https://kulturreste.blogspot.com/2018/10/heres-tool-make-it-work.html) I have already done an analysis of PRONOM signatures. Since today the module for this exists on CPAN, see https://metacpan.org/pod/File::FormatIdentification::Pronom for details.

In addition to the statistics on PRONOM signatures, the Perl package comes with two more helper scripts that can make the work of a long-term archivist easier.

Format identification

On the one hand, we have the functionality of classic format recognition. The script delivers all hits. In the output the quality of the RegEx is indicated. This does not say how well the PRONOM signature matches the file, but how specifically it is created.

Here is an example output for a TIFF file, which was wrongly recognized as GeoTIFF by Droid:

perl -I lib bin/pronomidentify.pl -s DROID_SignatureFile_V96.xml -b /tmp/00000007.tif

/tmp/00000007.tif identified as Tagged Image File Format with PUID fmt/353 (regex quality 1)
/tmp/00000007.tif identified as Geographic Tagged Image File Format (GeoTIFF) with PUID fmt/155 (regex quality 2)

Colorized output of possible signature hits in the hexeditor wxHexEditor

Under Linux you can use the editor wxHexEditor to analyze files. It allows you to create tag-files, in which you can define sections that are marked with colors and annotated.

The script pronom2wxhexeditor creates such a file. In the following you can see the call and a screenshot.

perl -I lib bin/pronom2wxhexeditor.pl -s DROID_SignatureFile_V96.xml -b /tmp/00000007.tif

What next?

Well, it's up to us as a community to use the existing tools and use their possibilities to improve our daily work. Anyone who has suggestions for improvement or ideas is welcome to share them with us.

I would be especially happy if servant spirits would take the pronoun statistics to their chest and help improve the pronoun signatures.

It makes sense to start with the orphaned signatures and to check multiple used signatures again.

Montag, 13. Juli 2020

Why it is a stupid idea to consider CSV as a valid long-term preservation file format

Take CSV!

It's so nice and quick and easy to say. Take CSV!

For simple cases that may be true. CSV files look so simple, so innocent, so sweet. Yet by their very nature they are insidious, vicious, and resemble a bloody walk into the deepest dungeons of classic role-players.

Let us begin our journey.

Innocent simplicity

You take a separator, e.g. the comma, use it to separate your values. Pour both into readable form. Done.

Okay. We need a second separator to show us the next line. But then, done! It's a CSV.

Hmm. There was something. Line separator. Now, is that line feed, carriage return or carriage return and line feed? It depends. For example, what operating system you're running.

The monster is growing

It is not a bad idea to separate values of a list by commas. Especially for Americans, this feels quite natural.

In other parts of the world, the decimal places of fractional numbers are separated by commas. Good, then we'll give the spreadsheets the opportunity to define the separator freely. Problem solved.

Well, not quite. It could be in other contexts that somehow the separator could appear in the individual values of a list. Good, then we'll introduce quoting. We define a character that allows us to recognize whether a separator is a separator or just a text component of a list value. Apostrophes would fit. That was easy, wasn't it?

Short break

So, to sum up. CSV files are easy. You need a separator, which can be a comma or anything else. We have a second separator that separates the lines. Usually there are three variations. We need quoting to see that a value cannot be confused with a separator.

Yeah, it may have been a little more complex than it looked at first. But what is there to make it worse?

Little toothy pegs!

Hmm, what if I want to store a text like this as a value after the raw value 1:

And he said "Oh, no!"

In the text, we have a comma, which would be protected by quoting, But we also have quotation marks, which we need for our quoting. No problem, then we double the quotation mark at that point to indicate that the text is not finished. So in the CSV it looks like this now:

1, "And he said ""Oh, no!""

I got it.

But, wait, what happens if my text consists of a single quotation mark?

1,""""

You're lucky. It seems to be working.

Wait, so what if I have a lot of quotation marks? As in

""""""

This is translated to

1, """"""""""""""

It works, too.

The problem is in the details

Now, a nasty little devil might get the idea to construct a text as value that contains line breaks, for example this one:

Evil Text
",
",

That would then:

1, "Evil text
"","
"",

Oops! If I now stubbornly read this in line by line, I would have read strange lines.

Good thing there is real software out there that reads and parses CSV files cleanly from the beginning. Not that anyone here still uses 'grep' and co.

The Abyss

Have we actually talked about character encoding yet? ASCII, Latin-1, UTF32? UTF8? With or without byte-order mark? No. Let's turn back. We still have a chance.

Later, at the pub.

I admit it was a terrible trip. Now, over a cold beer, we can laugh about it. But our hearts were already in our mouth. We had no idea what to expect.

If only there had been a sign that said what character encoding, what line end encoding, what separators for lines and columns we could expect, yes, then we would have been able to understand CSV and we would have been spared the horror. But the horror comes from the darkness, from the premonitions of the unknown.

Therefore, be warned!

Don't use CSV, it could get you!

Dienstag, 18. Februar 2020

format zoo for videos - a bad idea in digital preservation

Background

In an article on https://axfelix.github.io/ffv1, reasons are given not to apply the existing normalization of born-digital videos to FFV1, but to convert to lossy codecs instead. Elsewhere I even heard that normalization is not applied at all because it requires so many resources.

Why is normalization a good idea after all?

Normalization ensures that a manageable set of file formats remains from the huge format zoo, which can be handled well in the future. Normalization therefore reduces the organizational complexity above all.

And why should you use Matroska/FFV1?

FFV1 has the disadvantage of imposing higher storage requirements on its users, but in my opinion, the following points outweigh it:

FFV1 is much less complex than h264 (read "reduced technical complexity")
FFV1 (like other lossless codecs) allows automatic format migration (see also RAWcooked) — this reduces organizational complexity
FFV1 is freely available, widely used, well documented and standardized

The point that FFV1 is also more resistant to bit rot is just the icing on the cake.

Summary

Incidentally, personnel cost is the cost driver in digital preservation, as opposed to the pure storage cost.

Hence, the ultimate question is: how expensive is storage capacity in relation to the reduced technical and organizational complexity?

Mittwoch, 29. Mai 2019

Legacy media

This is the reason why you have to pay special attention to legacy digital media. Defective tracks of a floppy disk, special hardware (and knowledge) is necessary here.

Montag, 1. April 2019

Vorsicht vor Bitfischchen - Bestandserhaltung im digitalen Zeitalter

Schädlingsbekämpfung ist ein immerwährendes Problem in Bibliotheken und Archiven. Silberfischchen, Papierfischchen und andere Übeltäter laben sich an den Beständen und richten dabei beträchtliche Schäden an.

Da die Schädlingsbekämpfung nicht als explizite Aufgabe im OAIS-Referenzmodell aufgeführt ist, haben einige digitale Langzeitarchive hier bisher deutliche Defizite. Inzwischen spüren aber auch diese Einrichtungen immer deutlicher, dass die Schädlingsbekämpfung nicht vernachlässigt werden darf.

Angelockt von umfangreichen digitalen Beständen nisten sich Bitfischchen und Käfer (in der Fachsprache "Bugs" genannt) in Kabelhaufen ein und vermehren sich dort ungestört. Das Nahrungsangebot durch den reichlich vorhandenen Kabelsalat ist gut, und so wachsen die Populationen schnell an. Reste von Junk sowie Binärmüll-Krümel verschärfen das Problem zusätzlich.

Nicht nur die Anzahl der Fischchen, sondern auch ihre lange Lebensdauer ist ein Problem. Viele von Ihnen werden acht bis zehn Jahre alt, Microfichechen sogar noch deutlich älter.

Im moderigen Milieu vieler digitaler Archive fühlen sich auch Magnetbandwürmer wohl, die sich vor allem an den Daten auf WORM-Tapes laben. Daten, die nicht von den kleinen Plagegeistern zerstört werden, zerfallen in der fauligen Umgebung durch den Bitrot zu unlesbarem Datenkompost, der die Datenleitungen verstopft und so die Verarbeitung stört.

Eine gute Seite hat die neue Plage allerdings: findige Informatiker haben herausgefunden, dass Bitfischchen hervorragend zur Herstellung von Bitfett geeignet sind. Sie nutzen es, um Leitungsverbindungen zu schmieren und so die Reibung bei der Datenübertragung zu reduzieren, was wiederum positiv auf den Durchsatz auswirkt.

Freitag, 5. Oktober 2018

Here's a tool, make it work!

In the last post you may have already noticed it. To analyze the hits of DROID signatures I wrote a small Perl script which converts Droid signatures into Perl Regular Expressions and writes the matches into tag files of the hex editor wxHexEdit so that you can see which signatures were used where in a file.

From this small script a bigger Perl module called "File::FormatIdentification::Pronom" was created. It should not replace Droid, Fido or Siegfried. It only serves to analyze which patterns can be optimized and gives statistics about how to improve the Pronom database in the future.
In the following a statistic of the current Droid signature is shown, so that you get a feeling, what is possible.

perl -I lib/ bin/pronom_statistics.pl ../DROID_SignatureFile_V94.xml
Statistics of file ../DROID_SignatureFile_V94.xml
=======================================

Countings
---------------------------------------
Count of PUIDs:                        1670
         internal IDs:                 1441
         regular expressions:          1730
         file endings:                 1167
         PUIDs with file endings only: 503
         (56,76,167,168,169,194,195,212,594,681,682,683,684,691,717,760,780,879,996,1435)
         orphaned internal IDs:        20
         (56,76,167,168,169,194,195,212,594,681,682,683,684,691,717,760,780,879,996,1435)

Quality of internal IDs
---------------------------------------
1-best quality internal ID (PUID, name):       110 (fmt/75, Drawing Interchange File Format (ASCII)) -> 4.882;3.135
        combined regex: (?=((\x0A)|(\x0D\x0A)(0))SECTION((\x0A)|(\x0D\x0A)(\x20\x202)((\x0A)|(\x0D\x0A)(HEADER)((\x0A)|(\x0D\x0A))))((\x0A)|(\x0D\x0A)(9))\$ACADVER((\x0A)|(\x0D\x0A)(\x20\x201)((\x0A)|(\x0D\x0A)(AC1009)((\x0A)|(\x0D\x0A))))((\x0A)|(\x0D\x0A)(0))ENDSEC((\x0A)|(\x0D\x0A)))(?=(((\x0A)|(\x0D\x0A)(0))EOF((\x0A)|(\x0D\x0A)))\Z)
2-best quality internal ID (PUID, name):       105 (fmt/70, Drawing Interchange File Format (ASCII)) -> 4.736;2.833
        combined regex: (?=0\x0D\x0ASECTION\x0D\x0A\x20\x202\x0D\x0AHEADER\x0D\x0A9\x0D\x0A\x24ACADVER\x0D\x0A\x20\x201\x0D\x0AAC((1001)|(2\x2E21)|(2\x2E22)(\x0D\x0A))0
ENDSEC
)(?=(0\x0D\x0AEOF\x0D\x0A)\Z)
3-best quality internal ID (PUID, name):       104 (fmt/69, Drawing Interchange File Format (ASCII)) -> 4.644;2.833
        combined regex: (?=0\x0D\x0ASECTION\x0D\x0A\x20\x202\x0D\x0AHEADER\x0D\x0A9\x0D\x0A\x24ACADVER\x0D\x0A\x20\x201\x0D\x0AAC2\x2E10\x0D\x0A0\x0D\x0AENDSEC\x0D\x0A)(?=(0\x0D\x0AEOF\x0D\x0A)\Z)
4-best quality internal ID (PUID, name):       103 (fmt/68, Drawing Interchange File Format (ASCII)) -> 4.644;2.833
        combined regex: (?=0\x0D\x0ASECTION\x0D\x0A\x20\x202\x0D\x0AHEADER\x0D\x0A9\x0D\x0A\x24ACADVER\x0D\x0A\x20\x201\x0D\x0AAC1\x2E50\x0D\x0A0\x0D\x0AENDSEC\x0D\x0A)(?=(0\x0D\x0AEOF\x0D\x0A)\Z)
5-best quality internal ID (PUID, name):       102 (fmt/67, Drawing Interchange File Format (ASCII)) -> 4.644;2.833
        combined regex: (?=0\x0D\x0ASECTION\x0D\x0A\x20\x202\x0D\x0AHEADER\x0D\x0A9\x0D\x0A\x24ACADVER\x0D\x0A\x20\x201\x0D\x0AAC1\x2E40\x0D\x0A0\x0D\x0AENDSEC\x0D\x0A)(?=(0\x0D\x0AEOF\x0D\x0A)\Z)

1-worst quality internal ID (PUID, name):       1299 (fmt/950, MIME Email) -> -1.993;-2.91;-2.776;-2.776;-2.29
        combined regex: (?=\A.{0,16384}(((V)|(v)(\x2D)((IME)|(ime)(M)))ersion: 1\.0))(?=\A.{0,16384}(To\x3A\x20))(?=\A.{0,16384}(From\x3A\x20))(?=\A.{0,16384}(Date\x3A\x20))(?=\A.{0,16384}(Content\x2DType\x3A\x20))
2-worst quality internal ID (PUID, name):       527 (fmt/358, Internet Data Query File) -> -2.806;-2.743;-2.629;-2.981
        combined regex: (?=\A.{0,3424}(\x5BQuery\x5D).*(((S)|(s)(i)((C)|(c)))cope=))(?=\A.{0,3424}(\x5BQuery\x5D).*(((C)|(c)(i)((C)|(c)))olumns=))(?=\A.{0,3424}(\x5BQuery\x5D).*(((T)|(t)(i)((C)|(c)))emplate=\/))(?=\A.{0,3424}(\x5BQuery\x5D).*(((R)|(r)(i)((C)|(c)))estriction=.?(\x25)))
3-worst quality internal ID (PUID, name):       532 (fmt/363, SEG Y Data Exchange Format) -> -3.351;-4.196
        combined regex: (?=\A.{0,320}(\x40{22}))(?:(?=\A.{3200}(\x00\x00.{15}([^\x00])|(?=\A.{3600}(\x00\x00.{15}([^\x00])).{3}([^\x00])(.{2}(\x00[\x01-\x08])|.{2}(\x01\x00))))
4-worst quality internal ID (PUID, name):       533 (fmt/363, SEG Y Data Exchange Format) -> -3.351;-4.196
        combined regex: (?=\A.{0,320}(\x40{22}))(?:(?=\A.{3200}(\x00\x00.{15}([^\x00])|(?=\A.{3600}(\x00\x00.{15}([^\x00])).{3}([^\x00])(.{2}(\x00[\x01-\x08])|.{2}(\x01\x00))))
5-worst quality internal ID (PUID, name):       835 (fmt/532, Drawing Interchange File Format (ASCII)) -> -3.614;-3.842
        combined regex: (?=\A.{1,3}((0).{1,2}SECTION.{1,2}(\x20\x202).{1,2}(HEADER)).+((9).{1,2}\$ACADVER.{1,2}(\x20\x201).{1,2}(AC1027)).+((0).{1,2}ENDSEC))(?=((0).{1,2}EOF).{1,3}\Z)


Regular expressions
---------------------------------------
Count of multiple used regular expressions: 67
         common regex group no 0:
            regex='(((\x0A)|(\x0D)|(\x0D\x0A)(0))EOF).{0,2}\Z'
            internal IDs: 111,112,113

[…]

I would be pleased about feedback. The code is available under http://andreas-romeyke.de/software.html#_file_formatidentification_pronom .

Have fun!

Montag, 17. September 2018

A file is a TIFF is a MP3 is a…

In den letzten Tagen sind uns einige Dateien aufgefallen, die in der Formatidentifizierung hängengeblieben sind. Diese wurden von Droid als TIFF (fmt/353) und als MP3 (fmt/134) erkannt.

Die Frage, die sich uns stellte: Lag ein Fehler vor, oder handelt es sich tatsächlich um Dateien, die man anhand der Pronom-Signaturen sowohl als TIFF als auch als MP3 interpretieren könnte?

Um diese genauer zu untersuchen, haben wir uns ein Perl-Script¹ geschrieben. welches die Muster aus der Droid-Signaturen Datei verwendet und die entsprechenden Treffer im HexEditor sichtbar macht. Hier ein Screenshot:


wxHexeditor, Screenshot mit spezieller Tags-Datei

Wie man sieht, treffen mehrere Muster. Zum einen das Muster für TIFF-Dateien, indem am Anfang der Magicbyte-String "0x4949" vorkommt. Zum anderen auch eines der Rezepte, die einen MP3-Datenstrom beschreiben.

Bei Wikipedia findet man unter XXX folgende Darstellung eines MP3-Frames. Das Muster in der Droid-Signatur trifft, da 8 Frames hintereinander vorkommen:


MP3-Struktur, Quelle: Wikipedia, sh. https://commons.wikimedia.org/wiki/File:Mp3filestructure.svg (CC-BY/GFDL)

Diese Datei ist ein gutes Beispiel dafür, daß nicht die Muster in der Pronom-Datenbank das Problem sind, sondern dateiformat-spezifische Eigenschaften es notwendig machen, den Ingest-Prozess so zu gestalten, dass dieser mit mehreren Treffern in der Formatidentifikation umgehen kann.

Siehe hierzu auch unser Beitrag "Formatidentifikation vs. Formatvalidierung - Wem glauben wir eigentlich?" unter https://kulturreste.blogspot.com/2016/06/formatidentifikation-vs.html

--
¹ Das Perlscript stellen wir demnächst zur Verfügung

Montag, 29. Juni 2026

Freitag, 25. Oktober 2024

Was bedeutet also "dieses Objekt ist wertvoll"?

Wiederbeschaffung

Nicht entstandene Werke

Wert als reziproke Funktion der Häufigkeit

Aussicht

Donnerstag, 16. März 2023

Data management

Things that make life easier

Architectural decisions

And now?

Dienstag, 7. März 2023

Choosing the right format

Metadata and AIP update considerations

Donnerstag, 2. März 2023

The basic terms

Donnerstag, 12. Mai 2022

The problem

The constraints

The proposal of dBagIt

Bag Declaration: dbagit.txt

Payload Manifest

Bag Metadata: bag-info.txt

Optional Tag Manifest

Implementation of the behavior

Future

Alternate consideration

FAQ (Update 2022-05-18)

Freitag, 3. Dezember 2021

A crumb of knowledge

My findings

Header / MBR

FATs

Directory

Umlauts ans Special chars

Open Questions

Supportive Links

Thanks

Sonntag, 28. November 2021

First indications

Disillusionment

Donnerstag, 1. April 2021

Exponential growth

Magnificent diversity

A fuzzy concept

Success factors

Overrated

That's too exhausting

Summary

Mittwoch, 27. Januar 2021

What is Random Sampling?

Example

Limits

More ideas

Montag, 10. August 2020

Example 1 - Retro-digitised material

Example 2 - Web page

Let's summarize

Mittwoch, 22. Juli 2020

Previous work

Format identification

Colorized output of possible signature hits in the hexeditor wxHexEditor

What next?

Montag, 13. Juli 2020

Take CSV!

Innocent simplicity

The monster is growing

Short break

Little toothy pegs!

The problem is in the details

The Abyss

Later, at the pub.

Dienstag, 18. Februar 2020

Background

Why is normalization a good idea after all?

And why should you use Matroska/FFV1?

Summary

Mittwoch, 29. Mai 2019

Montag, 1. April 2019