Using list2padded for sequences

I have referred to https://github.com/explosion/thinc/blob/master/examples/03_pos_tagger_basic_cnn.ipynb to get a better understanding of the thinc layers. The model is as follows in the example

model = strings2arrays() >> with_array(
        HashEmbed(nO=width, nV=vector_width, column=0)
        >> expand_window(window_size=1)
        >> ReLu(nO=width, nI=width * 3)
        >> ReLu(nO=width, nI=width)
        >> Softmax(nO=nr_classes, nI=width)
    )

Can someone please explain what a string2arrays does? The documentation says that it takes a sequence of sequence of string and produces a list[int2D].

The input X_train is something like [[“this”,“is”,“awesome”],[“thinc”,“is”,“cool”]]. What does the strings2arrays transform this example to? I am unable to wrap my head around what exactly strings2arrays does and how it transforms the input from a List[List] (2D lists) to List[Int2D] technically (3D lists/sequence).

[honnibal]

@naveenjafer Ah you should use the PyTorchLSTM layer currently. I’m nearly done fixing the “native” LSTM implementation, but the one that’s there currently is a draft and isn’t very reliable.

In your model there, you have the LSTM inside the with_array block, which is not correct — you need the Padded format to be input for the LSTM. I think the type-checking should highlight a problem with your architecture (although the errors are sometimes hard to read).

There are a couple of ways you could do the transformations. One is:

model = (
    strings2arrays()
    >> list2padded()
    >> with_array(HashEmbed(nO=width, nV=vector_width, column=0))
    >> PyTorchLSTM(nO=100, nI=width)
    >> with_array(Softmax(nO=nr_classes, nI=width))
    >> padded2list()
)

A couple of people have found the sequence formats confusing, so I’m thinking of ways to make it a bit simpler. For now, a good general strategy is to keep the data in a List[Array] format and use the with_ transforms around blocks of your network. This is an easy way to get everything working, and then you can think about refactoring the transforms slightly if there’s a more efficient way to do things.

For instance, here’s how you could have your network:

model = (
    strings2arrays()
    >> with_array(HashEmbed(nO=width, nV=vector_width, column=0))
    >> with_padded(PyTorchLSTM(nO=100, nI=width))
    >> with_array(Softmax(nO=nr_classes, nI=width))
)

[honnibal]

@naveenjafer Sorry I didn’t see this before.

Yes I did mean just the multi-head attention layer, which wouldn’t have any state associated to it. The transformer would then be built out of it and several other pieces. Anyway, I think the list2padded issue here should be fixed. so I’ll close this issue as old.