sigtool behaviour with ?? wildcard

If I use sigtool to create a Pronom style signature from a byte sequence thus:

sigtool -b -p "04{1}[01:0C][01:1F]{28}([41:5A]|[61:7A]){10}(43|44|46|4C|4E)"

The results splits correctly into the XML form…

  <SubSequence Position="1" SubSeqMaxOffset="0" SubSeqMinOffset="0">
    <Sequence>04</Sequence>
    <RightFragment MaxOffset="1" MinOffset="1" Position="1">[01:0C][01:1F]</RightFragment>
    <RightFragment MaxOffset="28" MinOffset="28" Position="2">[41:5A]</RightFragment>
    <RightFragment MaxOffset="28" MinOffset="28" Position="2">[61:7A]</RightFragment>
    <RightFragment MaxOffset="10" MinOffset="10" Position="3">43</RightFragment>
    <RightFragment MaxOffset="10" MinOffset="10" Position="3">44</RightFragment>
    <RightFragment MaxOffset="10" MinOffset="10" Position="3">46</RightFragment>
    <RightFragment MaxOffset="10" MinOffset="10" Position="3">4C</RightFragment>
    <RightFragment MaxOffset="10" MinOffset="10" Position="3">4E</RightFragment>
  </SubSequence>
</ByteSequence>

but if I swap the {1} for ?? which should be syntactically identical, the processing doesn’t split a fragment at the ?? like it does with {1} and the question marks pass through into the XML output, which DROID then hangs when processing.

sigtool -b -p "04??[01:0C][01:1F]{28}([41:5A]|[61:7A]){10}(43|44|46|4C|4E)"

  <SubSequence Position="1" SubSeqMaxOffset="0" SubSeqMinOffset="0">
    <Sequence>04</Sequence>
    <RightFragment MaxOffset="0" MinOffset="0" Position="1">??[01:0C][01:1F]</RightFragment>
    <RightFragment MaxOffset="28" MinOffset="28" Position="2">[41:5A]</RightFragment>
    <RightFragment MaxOffset="28" MinOffset="28" Position="2">[61:7A]</RightFragment>
    <RightFragment MaxOffset="10" MinOffset="10" Position="3">43</RightFragment>
    <RightFragment MaxOffset="10" MinOffset="10" Position="3">44</RightFragment>
    <RightFragment MaxOffset="10" MinOffset="10" Position="3">46</RightFragment>
    <RightFragment MaxOffset="10" MinOffset="10" Position="3">4C</RightFragment>
    <RightFragment MaxOffset="10" MinOffset="10" Position="3">4E</RightFragment>
  </SubSequence>
</ByteSequence>

Note the questions marks on RightFragment position 1, and the offset being 0 rather than 1.

There are about 47 Pronom signatures using ?? syntax currently.

[nishihatapalmer]

DROID already supports putting an entire signature into a ByteSequence directly, via a Sequence attribute:

  <InternalSignature ID="3" Specificity="Specific">
   <ByteSequence Reference="BOFoffset" Sequence="04??[01:0C][01:1F]{28}([41:5A]|[61:7A]){10}(43|44|46|4C|4E)"/>
  </InternalSignature>

It will then compile it into the actual objects DROID uses. It can include * wildcards and the entire syntax. This was done to let container signatures use the full range of syntax available, without having to manually figure out the XML object structure that PRONOM currently produces.

The current system is pretty awful really. PRONOM is essentially pre-compiling the signatures into what it thinks DROID wants internally! Fixing in stone how DROID works in the actual signature file. But DROID has moved on substantially since then, and the decisions PRONOM makes are no longer optimal for DROID. So DROID actually changes what PRONOM tells it to do, reverse engineering the signatures so it can optimise them properly. It should really just give the signature in full, and let DROID figure out the best way to search for it.

However - the big problem with changing the signature file format is backwards compatibility with earlier versions of DROID. Maybe there could be a “new format” URL, and the older format is still published so older versions of DROID continue to work?

[steve-daly]

Fixed and available in 6.6-rc1. Thanks @nishihatapalmer