DPR usage of BertPooler

Environment info

• transformers version: 4.8.2
• Platform: Linux-5.8.0-50-generic-x86_64-with-debian-bullseye-sid
• Python version: 3.7.4
• PyTorch version (GPU?): 1.5.1 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed

Who can help

RAG, DPR: @patrickvonplaten, @lhoestq

Information

DPR initializes BertModel with a BertPooler module which is not used in the end

Although this seems consistent with the original implementation, it is confusing for the user. One would expect that the pooled_output will come from the BertPooler module, if it is present, and the last layer of the model. Moreover it wastes memory and compute.

How to fix

Simply add the add_pooling_layer=False flag in https://github.com/huggingface/transformers/blob/master/src/transformers/models/dpr/modeling_dpr.py#L178 Some other parts of the code need also to be fixed, like https://github.com/huggingface/transformers/blob/master/src/transformers/models/dpr/modeling_dpr.py#L205 should be sequence_output = outputs[0]

[PaulLerner]

Ok, I’ll let you know. I’m quite busy atm.

[PaulLerner]

DPR has an optional projection layer in the original implementation but it is only applied on the sequence output, not on BertPooler’s output.