Cipher detection interface

Problem

We have a limited number of ciphers that can be detected, and adding a new one means retraining the AI. Also, some ciphers can be detected far more easily algorithmically, and encodings can be checked by attempting to decode them.

To fix this problem, we have a tacked-on interface for filtering out various encodings, but that does;n’t solve the root problem.

However, the AI does provide us with an order of testing, which would be hard to replicate

Solution

Present an interface like the following:

from abc import ABC, abstractmethod
from typing import Optional, Dict

class Decoder(ABC):
    """Represents the undoing of some encoding"""
    @staticmethod
    @abstractmethod
    def decode(self, ctext: str, config: Dict[str, object]) -> Optional[str]: pass

class Cracker(ABC):
    @staticmethod
    @abstractmethod
    def scoreLikelihood(self, ctext: str, config: Dict[str, object]) -> float:
        """This should give a probability of this being this cipher"""
        pass
    @staticmethod
    @abstractmethod
    def attemptCrack(self, ctext: str, config: Dict[str, object]) -> Optional[str]:
        """This should attempt to crack the cipher, and use the config["checker"] where appropriate"""
        pass

The new steps should be

1. Ciphey should gather in a collection of modules exposing various Decoders and Crackers
2. Ciphey should apply all the Decoders it can to the ciphertext, and store this in a list
3. It should then walk through all the crackers, running scoreLikelihood on all of the decoded ciphertexts, applying attemptCrack to ones with a likelihood above a certain threshold, and storing the remaining ones in a collection ordered by likelihood.
4. If we still haven’t got anything, give up

[Cyclic3]

I think moving the AI to classify families of ciphers is a great idea! It has proven to be remarkably effective an

I would, however, like to address some of your points:

“Applying” a decoder should be low overhead if the decoder doesn’t match. Your example of a morse decoder slowing it down would only apply if it was encoded in something that looked like morse, which would probably mean that it was morse.

Whilst we’re on that topic: SHA1 only looks like SHA512 when you have no idea how long the actual datastream is, as they both look like random data. If we cannot decode it, then we haven’t a hope in hell of cracking it, and if we can, then we can tell which one it is. The insight provided by the AI is minimal on that, unless it has somehow managed to find a way of distinguishing hashes from random data, in which case the Journal of Cryptology is probably a better place to discuss this :p

I still believe there should be implementations that circumvent the AI completely. For instance, DES can be distinguished from random data by some difficult statistical analysis that the AI will not be able to learn.

Taking on board the points you raised, I would like to propose yet another alternative scheme:

class Detector(ABC):
    @abstractmethod
    def what(self) -> str:
        """Returns the cipher that this object attempts to detect"""
        pass
    @abstractmethod
    def scoreLikelihood(self, ctext: str, config: Dict[str, object]) -> Dict[string, float]::
         """Should return a dictionary of (cipher_name: score), using config["checker"] as appropriate"""
    @abstractmethod
    def isIntensive(self):
        """Return True if this will take a significant amount of time by some metric"""
        pass

class Cracker(ABC):
    @abstractmethod
    def what(self) -> str:
        """Return the cipher that this object attempts to crack"""
        pass
    @abstractmethod
    def attemptCrack(self, ctext: str, config: Dict[str, object]) -> Optional[str]:
        """This should attempt to crack the cipher, and use the config["checker"] where appropriate"""
        pass

(the Decoder interface remains the same)

The family AI should be a Detector that returns the score for each family, maybe scaled to get it to hit the threshold on particularly likely outputs.

Anyway, here is the algorithm:

1. We gather the Decoders, Detectors, and Crackers from the various modules. When we gather the non-Cracker objects, we put them in a structures of type Dict[str, List[OBJECT_NAME]], which allows simple lookup, and keep intensive detectors separate.
2. We then iterate through the non-intensive ones in parallel, storing the results in scores: Dict[str, float], which will keep track of the probabilities as we go through.
   • If we get a scores above some threshold (maybe use 1 or something), we should attempt the crackers for that, and remove that entire cipher from the system (by deleting it from the relevant dictionaries).
   • If we do not exceed the threshold, look it up in scores, and if our score is above the existing value, update it with the new one.
3. If we exhaust the non-intensive detectors, we then iterate through the intensive ones in the same way, using the same scores dictionary.
4. Now we sort the scores dictionary into a list, and iterate through them in descending order, until we run out of ciphers.

This could perhaps be improved by also taking into account the complexity of cracking, but I’m not sure how to do that without massively overcomplicating the algorithm. We can hopefully rely on more secure algorithms being harder to detect instead :p

[SkeletalDemise]

Closing this because Ciphey’s interface was finished a long time ago.