Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Implement PyTorch and/or TensorFlow sequence classification architectures for causal language models

See original GitHub issue

🚀 Feature request

The architecture GPT2ForSequenceClassification was added in #7501 in PyTorch. It would be great to have it in TensorFlow (cf. issues #7622), but it would also be great to have it for other causal models: ~OpenAI GPT~, ~CTRL~ (PR opened @elk-cloner), ~TransfoXL~ (PR opened @spatil6)

Below is a list of items to follow to make sure the integration of such an architecture is complete:

Implement XXXForSequenceClassification in modeling_xxx.py or TFXXXForSequenceClassification in `modeling_tf_xxx.py
Test that architecture in tests/test_modeling_xxx.py or tests/test_modeling_tf_xxx.py
Add that architecture to __init__.py and docs/source/model_doc/xxx.rst.

Taking a look at the code changes in #7501 would be a good start.

A very good first issue to get acquainted with the library and its architectures!

Issue Analytics

State:
Created 3 years ago
Comments:18 (10 by maintainers)

Top GitHub Comments

2reactions

spatil6commented, Dec 2, 2020

Ok thanks @LysandreJik.

I’m waiting for this PR #8714 to get merge. Once done, I’ll raise PR for these models as well.

1reaction

LysandreJikcommented, Nov 23, 2020

I believe CTRL and TransfoXL are still available. Feel free to open a PR!

Top Results From Across the Web

Text classification with an RNN - TensorFlow

This text classification tutorial trains a recurrent neural network on the IMDB large movie review dataset for sentiment analysis.

Build Your First Text Classification model using PyTorch

In this article learn how to solve text classification problems and build text classification models and implementation of text ...

NLP From Scratch: Translation with a Sequence to ... - PyTorch

With a seq2seq model the encoder creates a single vector which, in the ideal case, encodes the “meaning” of the input sequence into...

Deep Learning Models - GitHub

A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks. Traditional Machine Learning. Title ...

PyTorch vs TensorFlow — spotting the difference

Here we introduce datasets module which contains wrappers for popular datasets used to benchmark deep learning architectures. Also nn.Module is used to build...