How to incorporate covariate information using TransformerTempFlowEstimator

Description

I am currently using the Australian retail trade turnover data set to get familiar with PyTorch-TS in general and with TransformerTempFlowEstimator in particular. The data looks as follows:

Each series (133 series in total) has 417 months of training observations and is uniquely identified using two keys:

• State: The Australian state (or territory)
• Industry: The industry of retail trade

All series show quite some positive dependencies, as the correlation matrix shows:

As such, TransformerTempFlowEstimator seems to be a good option. I want to make use of both State and Industry as covariates in the model. For each categorical covariate, a generalized linear mixed model is fit to the outcome and the coefficients are returned as the encodings. The cardinality of State and Industry is [7, 20]. After bringing the data into the right format, I create the train data as follows:

train_ds = ListDataset([{FieldName.TARGET: target, 
                         FieldName.START: start,
                         FieldName.ITEM_ID: item_id,
                         FieldName.FEAT_DYNAMIC_REAL: feat_dynamic_real,
                         FieldName.FEAT_STATIC_REAL: feat_static_real,
                         FieldName.FEAT_TIME: time_feat
                        } 
                        for (target, 
                             start, 
                             item_id, 
                             feat_dynamic_real, 
                             feat_static_real, 
                             time_feat
                            ) in zip(target_train,
                                     start_train,
                                     item_id_train,
                                     feat_dynamic_real_train,
                                     feat_static_real_train,
                                     time_feat_train
                                    )],
                      freq = "1M") 

feat_static_real_train contain the embeddings and time_feat_train the month information. To transform the data into a multivariate data set, I use

grouper_train = MultivariateGrouper(max_target_dim = 133) # as there are 133 unique series 
train_ds = grouper_train(train_ds)

However, after using grouper_train(train_ds), none of the covariate information is included anymore. To bring them back, I use

train_ds.list_data[0]["feat_dynamic_real"] = feat_dynamic_real_train
train_ds.list_data[0]["feat_static_real"] = feat_static_real_train

I then train the model as follows:

np.random.seed(123)
torch.manual_seed(123)
trainer = Trainer( epochs = 40) 

estimator = TransformerTempFlowEstimator(input_size = 401,
                                         freq = "1M", 
                                         prediction_length = 24,
                                         context_length = 48,
                                         target_dim = 133,
                                         cardinality = [7, 20],
                                         trainer = trainer)                              
predictor = estimator.train(training_data = train_ds)

The model summary is

predictor.__dict__["prediction_net"]*
pts.model.transformer_tempflow.transformer_tempflow_network.TransformerTempFlowPredictionNetwork(act_type="gelu", cardinality=[7, 20], conditioning_length=200, context_length=48, d_model=32, dequantize=False, dim_feedforward_scale=4, dropout_rate=0.1, embedding_dimension=5, flow_type="RealNVP", hidden_size=100, history_length=60, input_size=401, lags_seq=[1, 12], n_blocks=3, n_hidden=2, num_decoder_layers=3, num_encoder_layers=3, num_heads=8, prediction_length=24, scaling=True, target_dim=133)

I also compared the forecast to some competing models, even though I am not sure that all models are correctly specified (i.e., covariate information, no parameter tuning).

Given the strong dependencies between the different series, I would suspect that TransformerTempFlowEstimator should outperform models that treat the series as being independent.

Question

Based on the above summary, I have the following questions concerning the proper use of TransformerTempFlowEstimator:

• How can covariates be included, in particular categorical information.
• Does the model automatically include, e.g., month and/or age information that it itself derives from the data or do we need to pass it using time_features in the function call.
• Does the model automatically derive holiday information from the data, or do we need to derive it ourselves as described here.
• Does the model automatically select an appropriate lag-structure from the data, or do we need to derive it ourselves as described here.
• Which of the following field names are currently supported:

 "FieldName.START = 'start'",
 "FieldName.TARGET = 'target'",
 "FieldName.FEAT_STATIC_CAT = 'feat_static_cat'",
 "FieldName.FEAT_STATIC_REAL = 'feat_static_real'",
 "FieldName.FEAT_DYNAMIC_CAT = 'feat_dynamic_cat'",
 "FieldName.FEAT_DYNAMIC_REAL = 'feat_dynamic_real'",
 "FieldName.FEAT_TIME = 'time_feat'",
 "FieldName.FEAT_CONST = 'feat_dynamic_const'",
 "FieldName.FEAT_AGE = 'feat_dynamic_age'",
 "FieldName.OBSERVED_VALUES = 'observed_values'",
 "FieldName.IS_PAD = 'is_pad'",
 "FieldName.FORECAST_START = 'forecast_start'"]

[kashif]

No, the holidays get converted to dynamic real features and depending on the kernel you use a particular date gets smoothed out so the model knows when a particular date is approaching and has passed… the reason the dynamic cat is not used is because I never found a need for it yet… but as soon as I do I will add it…

[NielsRogge]

Thank you for the quick reply.

For people wondering (quick summary):

• the model you use itself already creates a bunch of covariates, which are defined in the create_transformation function of the model. For example, DeepVAREstimator already creates covariates such as Fourier time features (if you’re not providing time_features yourself when initializing the model), age features, and observed values as seen here. Also, lagged features are created (as seen by the lag_seq variable), if you’re not providing them yourself when initializing the model.
• if you want to add additional covariates (such as holiday information or other dynamic real features), add them to your dataset objects as shown in section 1.3 of GluonTS’ extended tutorial.
• If you’re using MultivariateGrouper to group the various time series, you have to add the features again after grouping (as shown above).
• when initializing DeepVAREstimator, set use_feat_dynamic_real/use_feat_static_cat/use_feat_static_real to True (depending on which you are using). Also, if you’re using categorical features, set cardinality, which is a list containing the number of unique values for each categorical feature, and embedding_dimension which is a list with embedding dimensions you want to use for each of the additional features. For each additional categorical feature you add, an embedding layer is created as seen here.

@kashif is there a reason use_dynamic_feat_cat is not supported in DeepVAR? Isn’t holiday information a dynamic (i.e. time-dependent) categorical feature?