Longest sequence and truncation of sentence

Hi, I wonder how the maximum length is set before getting an embedding given a sentence. Let s be a sentence such as s = [x1, x2, x3, ----, xN]. Is there a maximum length parameter n such that if N>n, then all tokens in indices above n are removed? s would be mapped to map(s) = [x1, x2, —,xn] (This what we can see often in BERT-like models).

From this code:

            longest_seq = 0

            for idx in length_sorted_idx[batch_start: batch_end]:
                sentence = sentences[idx]
                tokens = self.tokenize(sentence)
                longest_seq = max(longest_seq, len(tokens))
                batch_tokens.append(tokens)

            features = {}
            for text in batch_tokens:
                sentence_features = self.get_sentence_features(text, longest_seq)
      

I am confused about what get_sentence_features does which is defined here (I do not get what _first_module corresponds to actually):

def get_sentence_features(self, *features):
        return self._first_module().get_sentence_features(*features)

[nreimers]

Hi @dataislife BERT like models have a limit of usually 512 tokens. In the sentence transformer models, you can set your own limit, which is usually set to 128 tokens.

A sentence is broken down to tokens and word pieces. Anything above the limit (e.g. 128) is truncated, i.e., only the first 128 word pieces are used in the default setting.

Best Nils

[sagar1411986]

@nreimers Thanks.