django and transitions [previously about model extensions]

We documented the results of this discussion in the FAQ. Make sure to check it out!

Occasionally,

we discuss generic model functionality which might be handy for some users. Just to keep track of these requests, here is the current candidate list:

edit: I split the list into separate comments. This way it might be easier to “upvote” the extension you are looking forward to.

Please comment/“upvote” if you like to see any of these features integrated into transitions

Already implemented: 0.6.0:

• finite state support/TaggableModel (see #130 and #230)
• timeout support (see #198 and #229)

[aleneum]

Hi @jxskiss and @proofit404,

I just finished some test runs and wanted to share the results. The full code can be found here and is based on what jxskiss provided. This is the most important stuff:

# As a model, I used a simple one field model. `Item` was extended with a *MixIn*
class Item(<MIXIN>, models.Model):
    state = models.CharField(max_length=16, default='new')

# This is a Singleton meta class from
# https://stackoverflow.com/questions/6760685/creating-a-singleton-in-python
class Singleton(type):
    _instances = {}
    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
        return cls._instances[cls]

class GlobalMachine(transitions.Machine):
    __metaclass__ = Singleton

    # # In case you want to use set instead of list to store models
    # def __init__(self, *args, **kwargs):
    #     super(GlobalMachine, self).__init__(*args, **kwargs)
    #     self.models = SetWrapper()

# Since transitions expects a list, extend set by a method 'append'
# You just need that in case you want to use set instead of list for storing models in Machine
class SetWrapper(set):

    def append(self, item):
        self.add(item)

# instead of adding a machine to each entry, we use a global machine here
# the constructor is only called ONCE and ignored after the global
# instance has been created.
# Note that we do not add a model or set an initial state;
# this will be done for each model entry instead
class ItemSingletonMachineMixin(object):
    def __init__(self, *args, **kwargs):
        super(ItemSingletonMachineMixin, self).__init__(*args, **kwargs)
        machine = GlobalMachine(
            model=None,
            states=ItemStatus.SM_STATES,
            transitions=ItemStatus.SM_TRANSITIONS,
            initial=None,
            auto_transitions=False,
            send_event=True
        )
        # save item to be hashable
        self.save()
        machine.add_model(self, initial=self.state)

Evaluation

I tested 5 kinds of models with different MixIns:

• Item: add a transitions.Machine instance to each record/item
• ItemSingleton: add each model to a global transitions.Machine
• ItemSingletonSet: Extend transitions.Machine to use (a subclass of) set instead of list to store models; this increases look up speed in add_model
• ItemNoMachine: just a minimal model without a state machine (for referencing purposes)
• ItemFysom: add a customized fysom machine to each record/item
• ItemFysomClass: use fysom as a class variable, extend the model with an __getattribute__ wrapper which passes the object to the static machine.

note that I wrote “add a customized fysom machine” since imho fysom.StateMachine is not handled as a singleton in the items but gains his waaaay better memory footprint by being more lightweight in general and facilitating static methods where ever possible. But maybe I overlooked something. Edit: I did. It’s a class attribute and not bound to an instance as in my current gist. I will update the comparison soon. I added ItemFysomClass which sticks to jxskiss actual idea.

Process

I wrote some helper functions to create instances of Item in memory until the django process exceeded a previously defined memory limit. I also tracked the time how long it took to create the instances (you will see why). This was done on my laptop while I did other things. So its not 100% scientific. Additionally, I added self.save() to each __init__ method of the mixins. Creation times are slightly increased because of this. But it was necessary to make the resulting model hashable (which is required for ItemSingletonSet as you will see later).

Results

With a memory limit of 250MB I could create:

• 3120 instances of Item
• 18680 instances of ItemSingleton
• 18740 instances of ItemSingletonSet
• 14300 instances of ItemFysom
• 278500 instances of ItemFysomClass

Discussion

As already mentioned by jxskiss, adding a transitions.Machine to each Item will result in a big memory footprint and is therefore not encouraged. Using a lightweight state machine such as the customized version of fysom with static methods produces much better results. However, using transitions.Machine as a Singleton allows to even fit more instances into 250MB (about 30%). The least overhead can be achieved by passing the model to a static state machine as illustrated in jxskiss’s ItemFysomClass. With the help of a small convenience wrapper (__getattribute__) and a transition name prefix (e.g. ‘sm_’), the memory footprint of the added state machine logic is trivial as the actual memory consumption is more or less determined by the record/entry and it’s field. Both test cases, ItemFysomClass and ItemNoMachine resulted in almost the same amount of objects.

Note that the tested model was close to the smallest possible model. In case your entry includes more fields and complex datatypes, the impact of a model decoration with a state machine will be reduced. These results may vary depending on when and how Python does the garbage collection and/or rearranges stuff in memory.

Using Singleton seems to considerably reduce the object overhead when using transitions. However, using just one Machine has its drawbacks:

transitions.Machine uses list to keep track of added models. This becomes an issue when lots of models are added consecutively since Machine.add_model always checks if the model is already in Machine.models. This takes longer the more models are already in the list. At the end of the test, it takes up to 0.6 seconds to add 10 instances Item to the machine. If this does not satisfy an intended use case I propose 2 solutions:

Solution 1: Skip check

A subclass of Machine could override add_model and basically copy the whole method bust just remove if model not in self.models to skip the test.

Solution 2: Use set instead of list

This approach has been tested with ItemSingletonSet and as you see it improves the execution time of add_model dramatically. Drawbacks of using set instead of list: set is unordered and model instances cannot be retrieved by index. Currently, at some points transitions expects Machine.models to be a list but we can work around this by subclassing set.

Feedback welcome

These tests were focused on memory footprint. I also tested for basic functionality but cannot guarantee everything will work without further adaptions to a specific use case. Feel free to ask questions or provide further feedback to improve the interplay of django and transitions. If you think this comment should be part of the faq, please tell me.

[aleneum]

django database integration (see #111/#142)