Support fp16 for inference

🚀 Feature request - support fp16 inference

Right now most models support mixed precision for model training, but not for inference. Naively calling model= model.haf() makes the model generate junk instead of valid results for text generation, even though mixed-precision works fine in training.

If there’s a way to make the model produce stable behavior at 16-bit precision at inference, the throughput can potentially double on most modern GPUS.

Motivation

Double the speed is always attractive, especially since transformers are compute-intensive.

[JulesGM]

has this been solved?

[johncookds]

Hi, I’ve noticed the same issue of the model randomly generating junk when using autocast within a custom generate() method with the only change below (fp16 is a boolean). From the above comments I thought this approach should’ve worked.

if fp16:
	with torch.cuda.amp.autocast():
		outputs = self(**model_inputs)
else:
	outputs = self(**model_inputs)

The current model I’ve tested it on is a huggingface gpt2 model finetuned on a personal dataset. Without fp16 the generate works perfectly. The dataset is very specific and the model is supposed to generate symbols+numbers, so it’s clear when it starts spitting out words during fp16 inference.