What does .encode() actually do to extract the embeddings ?

Looked at the code, it reads .encode() obtains the embeddings by looping through all hidden states (i.e. sequence outputs from all layers).

Question: But the shape of the output embeddings is 1D, it just has the (hidden dim size,) i.e (512, or 768,), as opposed to 2D (input_ids, hidden dim size) for text models and (pixel_values, hidden dim size) for vision models. So are the sentence embeddings obtained are got by adding an MLP head on top of the sequence output of all layers ? or by adding an MLP head on top of the pooler output of the sequence output of all layers? Please clarify.

Also the MLP head is the classic one i.e. simple linear layer within, out features + classic tanh activation, and no dropouts of any sort ?

Please advice …

[nreimers]

Just the mean of the last layer

[nreimers]

In most cases mean pooling is performed: All output embeddings are averaged to give the fixed representation