GPT-J-6B endpoint

I’m kicking off a TRC sub to see how far we can get with a custom GPT-J endpoint.

https://github.com/kingoflolz/mesh-transformer-jax

[cirecrux]

Just going to leave this comment here for anyone with the same problem:

RuntimeError: Resource exhausted: Failed to allocate request for 32.00MiB (33554432B) on device ordinal 0: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well).

Assuming that you’re following the fine-tune-guide from mesh-transformer-jax like I am, the problem is that the TPU has to be set up as v3-8 and not v2-8.

I wasted a whole day trying to figure out what I got wrong since I only know enough about code to copy paste error messages into the search bar. The author of the issue (avaer) is a genius for getting it to work somehow on a v2-8. Anyways, good luck out there.

[lalalune]

Hosted inference on HF is a snap, pin the model to keep it running hot.

https://huggingface.co/bigscience/T0pp

https://huggingface.co/EleutherAI/gpt-j-6B

async function query(data) {
	const response = await fetch(
		"https://api-inference.huggingface.co/models/EleutherAI/gpt-j-6B",
		{
			headers: { Authorization: "Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" },
			method: "POST",
			body: JSON.stringify(data),
		}
	);
	const result = await response.json();
	return result;
}

query({"inputs": "Can you please let us know more details about your "}).then((response) => {
	console.log(JSON.stringify(response));
});