Loss Calculation in Image Captioning

Hi , am new to PyTorch in model.py you do ,

embeddings = self.embed(captions)
embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)

that means you are only passing the context(from the EncoderCNN ) as input to the first time step, hence your outputs will always be 1+captions size as you are running the lstm for captions+1 number of time steps in the decoder , but looks like you have targets and outputs of same size. What am I missing here ? Thanks in advance .

[okanlv]

Decoder should have the same number of elements in both its input and output. Input in your example contains 7 words, whereas the output contains 5 words. Every word (including feature) is used to predict the next word, so it is like the following.

Input: feature <start> there is a cat

Output: <start> there is a cat <end>

[fmehralian]

Hey guys, The reason behind concatenating the feature vector with the input (here) is not clear for me. Would you please help me figure out the difference between this implementation and when we pass the feature vector as the hidden value of the first LSTM cell, as depicted below:

1. LSTM(cat(image_feature, input)) [link](

) 2) LSTM(input, hidden=image_feature) [link](

)