Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
auto model encodings for a text snippet returns different floating values across different batch sizes
See original GitHub issueEnvironment info
transformersversion: 4.4.1- Platform: Linux-4.19.112±x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: yes (but the bug issue is irrespective of it)
- Using distributed or parallel set-up in script?: no
Who can help
@LysandreJik, @patrickvonplaten
Information
Model I am using : bert-base-cased and sentence-transformers/distilbert-base-nli-stsb-mean-tokens
Consider the following code:
# pip install transformers
import torch
device = "cuda" if torch.cuda.is_available() else "cpu"
print(device)
import transformers
from transformers import AutoModel, AutoTokenizer
name = "sentence-transformers/distilbert-base-nli-stsb-mean-tokens"
model = AutoModel.from_pretrained(name)
tokenizer = AutoTokenizer.from_pretrained(name)
model.to(device)
model.eval()
from tqdm.autonotebook import trange
for ntimes in trange(1, 200, 1, desc="ntimes", disable=False):
s = ['This framework generates embeddings for each input sentence' for _ in range(ntimes)]
f = tokenizer(s, padding=True, truncation='longest_first', return_tensors="pt", max_length=128)
f = f.to(device)
with torch.no_grad():
out = model(**f, return_dict=False)
t = out[0] # token_embedding
print(str(ntimes).zfill(4), t[0][0][:5].tolist())
The testing setup is as follows: For every batch size considered in 1 to 200, the model output (last layer’s output) for first sentence is taken and compared. Ideally, it is expected to be same but depending on the batch size, the output varies. Although the differences are after several decimal places, it still creates an issue when rounding off or when used for exact-text-match tasks. An example when comparing first 5 values of CLS token’s positional representation is printed below:
batch_size first_5_values
0001 [-0.4345831274986267, 0.19430403411388397, -0.008721709251403809, 0.16533663868904114, -0.21307958662509918]
0002 [-0.4345831274986267, 0.19430403411388397, -0.008721709251403809, 0.16533663868904114, -0.21307958662509918]
0003 [-0.4345839023590088, 0.19430400431156158, -0.008721785619854927, 0.16533628106117249, -0.21307939291000366]
0004 [-0.4345839023590088, 0.19430400431156158, -0.008721785619854927, 0.16533628106117249, -0.21307939291000366]
0005 [-0.4345839023590088, 0.19430400431156158, -0.008721785619854927, 0.16533628106117249, -0.21307939291000366]
0006 [-0.4345828890800476, 0.19430409371852875, -0.0087218526750803, 0.1653369963169098, -0.2130797803401947]
0007 [-0.43458378314971924, 0.19430388510227203, -0.008721890859305859, 0.16533657908439636, -0.21307970583438873]
0008 [-0.43458378314971924, 0.19430388510227203, -0.008721890859305859, 0.16533657908439636, -0.21307970583438873]
0009 [-0.43458378314971924, 0.19430388510227203, -0.008721890859305859, 0.16533657908439636, -0.21307970583438873]
0010 [-0.43458378314971924, 0.19430388510227203, -0.008721890859305859, 0.16533657908439636, -0.21307970583438873]
0011 [-0.43458303809165955, 0.19430424273014069, -0.0087218526750803, 0.16533637046813965, -0.21307967603206635]
0012 [-0.43458303809165955, 0.19430424273014069, -0.0087218526750803, 0.16533637046813965, -0.21307967603206635]
0013 [-0.4345836043357849, 0.19430403411388397, -0.008721555583178997, 0.16533656418323517, -0.21307919919490814]
0014 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0015 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0016 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0017 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0018 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0019 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0020 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0021 [-0.4345839321613312, 0.19430312514305115, -0.008722005411982536, 0.16533663868904114, -0.21307897567749023]
0022 [-0.43458428978919983, 0.1943034529685974, -0.008722092024981976, 0.16533707082271576, -0.21307975053787231]
0023 [-0.43458428978919983, 0.1943034529685974, -0.008722092024981976, 0.16533707082271576, -0.21307975053787231]
0024 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0025 [-0.43458348512649536, 0.19430328905582428, -0.008722656406462193, 0.16533656418323517, -0.21307960152626038]
0026 [-0.43458348512649536, 0.19430328905582428, -0.008722656406462193, 0.16533656418323517, -0.21307960152626038]
0027 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0028 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0029 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0030 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0031 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0032 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0033 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0034 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0035 [-0.4345836639404297, 0.19430400431156158, -0.008721727877855301, 0.16533680260181427, -0.2130795419216156]
0036 [-0.4345836639404297, 0.19430400431156158, -0.008721727877855301, 0.16533680260181427, -0.2130795419216156]
0037 [-0.4345836639404297, 0.19430400431156158, -0.008721727877855301, 0.16533680260181427, -0.2130795419216156]
0038 [-0.4345836639404297, 0.19430400431156158, -0.008721727877855301, 0.16533680260181427, -0.2130795419216156]
0039 [-0.4345836639404297, 0.19430400431156158, -0.008721727877855301, 0.16533680260181427, -0.2130795419216156]
0040 [-0.4345836639404297, 0.19430400431156158, -0.008721727877855301, 0.16533680260181427, -0.2130795419216156]
0041 [-0.43458354473114014, 0.1943034529685974, -0.008721929043531418, 0.1653369963169098, -0.2130795270204544]
0042 [-0.43458354473114014, 0.1943034529685974, -0.008721929043531418, 0.1653369963169098, -0.2130795270204544]
0043 [-0.43458354473114014, 0.1943034529685974, -0.008721929043531418, 0.1653369963169098, -0.2130795270204544]
0044 [-0.43458375334739685, 0.1943032294511795, -0.008721861988306046, 0.1653372347354889, -0.21307994425296783]
0045 [-0.43458375334739685, 0.1943032294511795, -0.008721861988306046, 0.1653372347354889, -0.21307994425296783]
0046 [-0.43458375334739685, 0.1943032294511795, -0.008721861988306046, 0.1653372347354889, -0.21307994425296783]
0047 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0048 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0049 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0050 [-0.43458375334739685, 0.19430378079414368, -0.008721306920051575, 0.1653364896774292, -0.21308039128780365]
0051 [-0.43458375334739685, 0.19430378079414368, -0.008721306920051575, 0.1653364896774292, -0.21308039128780365]
0052 [-0.43458375334739685, 0.19430378079414368, -0.008721306920051575, 0.1653364896774292, -0.21308039128780365]
0053 [-0.4345836043357849, 0.1943032443523407, -0.008722072467207909, 0.16533635556697845, -0.21307991445064545]
0054 [-0.4345836043357849, 0.1943032443523407, -0.008722072467207909, 0.16533635556697845, -0.21307991445064545]
0055 [-0.4345836043357849, 0.1943032443523407, -0.008722072467207909, 0.16533635556697845, -0.21307991445064545]
0056 [-0.43458402156829834, 0.19430415332317352, -0.008722043596208096, 0.16533638536930084, -0.2130793184041977]
0057 [-0.43458402156829834, 0.19430415332317352, -0.008722043596208096, 0.16533638536930084, -0.2130793184041977]
0058 [-0.43458348512649536, 0.19430400431156158, -0.008722235448658466, 0.16533590853214264, -0.2130792737007141]
0059 [-0.43458348512649536, 0.19430400431156158, -0.008722235448658466, 0.16533590853214264, -0.2130792737007141]
0060 [-0.43458348512649536, 0.19430400431156158, -0.008722235448658466, 0.16533590853214264, -0.2130792737007141]
0061 [-0.43458348512649536, 0.19430400431156158, -0.008722235448658466, 0.16533590853214264, -0.2130792737007141]
0062 [-0.4345838725566864, 0.19430425763130188, -0.008721861988306046, 0.16533733904361725, -0.21307975053787231]
0063 [-0.4345838725566864, 0.19430425763130188, -0.008721861988306046, 0.16533733904361725, -0.21307975053787231]
0064 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0065 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0066 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0067 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0068 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0069 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0070 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0071 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0072 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0073 [-0.4345824718475342, 0.1943037360906601, -0.00872176606208086, 0.16533659398555756, -0.2130804806947708]
0074 [-0.4345824718475342, 0.1943037360906601, -0.00872176606208086, 0.16533659398555756, -0.2130804806947708]
0075 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0076 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0077 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0078 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0079 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0080 [-0.43458354473114014, 0.19430452585220337, -0.008721709251403809, 0.16533659398555756, -0.21307997405529022]
0081 [-0.43458354473114014, 0.19430452585220337, -0.008721709251403809, 0.16533659398555756, -0.21307997405529022]
0082 [-0.43458354473114014, 0.19430452585220337, -0.008721709251403809, 0.16533659398555756, -0.21307997405529022]
0083 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0084 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0085 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0086 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0087 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0088 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0089 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0090 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0091 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0092 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0093 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0094 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0095 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0096 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0097 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0098 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0099 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0100 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0101 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0102 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0103 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0104 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0105 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0106 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0107 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0108 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0109 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0110 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0111 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0112 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0113 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0114 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0115 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0116 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0117 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0118 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0119 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0120 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0121 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0122 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0123 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0124 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0125 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0126 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0127 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0128 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0129 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0130 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0131 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0132 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0133 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0134 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0135 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0136 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0137 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0138 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0139 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0140 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0141 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0142 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0143 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0144 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0145 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0146 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0147 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0148 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0149 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0150 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0151 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0152 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0153 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0154 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0155 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0156 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0157 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0158 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0159 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0160 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0161 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0162 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0163 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0164 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0165 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0166 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0167 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0168 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0169 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0170 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0171 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0172 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0173 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0174 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0175 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0176 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0177 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0178 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0179 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0180 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0181 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0182 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0183 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0184 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0185 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0186 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0187 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0188 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0189 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0190 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0191 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0192 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0193 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0194 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0195 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0196 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0197 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0198 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0199 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
To reproduce
Run the snippet of code provided above
Detailed snippets are available at this colab notebook
Expected behavior
It is expected that an input has same representation irrespective of the batch size used to obtain it.
Issue Analytics
- State:
- Created 3 years ago
- Comments:7 (3 by maintainers)
Top Results From Across the Web
Trying to infer the `batch_size` from an ambiguous collection
I am not sure where it come from. I did not do any changes on my code. The error keeps getting different batch...
Read more >CLIP - Hugging Face
We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action...
Read more >How to Develop an Encoder-Decoder Model for Sequence-to ...
The approach involves two recurrent neural networks, one to encode the source sequence, called the encoder, and a second to decode the encoded ......
Read more >Word embeddings | Text - TensorFlow
Machine learning models take vectors (arrays of numbers) as input. When working with text, the first thing you must do is come up...
Read more >Package google.cloud.automl.v1
Returns a Model in the response field when it completes. ... Because this field returns a value at model training time, for different...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Yes, you’re right, the difference is between batch size == 1 and batch size > 1! Talking about it with team members, we guess it’s because the kernels used to compute the results differ according to the dimensions, as they’re optimized differently.
For batch size = 1, the model input would essentially be in one dimension (the vector of tokens), while for batch size > 1, the model input would essentially be in two dimension (an array of tokens).
Imo this is more of a PyTorch issue (if it’s an issue in the first place) than a
transformersissue!This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.