What are your thoughts on using latents for additional classification task

Hi! I was wondering if you have thought about aggregating seasonal and growth latents for additional tasks (for example classification)? What are the possible ways to bring latents into single feature vector in your opinion? The easiest one would be just get the mean along layers and time dimensions but that seams to be too naive. Another idea I had it to use Cross Attention mechanic with single time query key to aggregate latents:

all_latents = torch.cat([latent_growths, latent_seasonals], dim=-1)
all_latents = rearrange(all_latents, 'b n l d -> (b l) n d')
# q = nn.Parameter(torch.randn(all_latents_dim))
q = repeat(q, 'd -> b 1 d', b = all_latents.shape[0])
agg_latent = cross_attention(query=q, context=all_latents)
agg_latent = rearrange(all_latents, '(b l) n d -> b (l n) d')
agg_latent = agg_latent.mean(dim=1) # may be we should have done it before cross attention?

Would be great to hear your thoughts

[inspirit]

super thanx!

[lucidrains]

@inspirit done https://github.com/lucidrains/ETSformer-pytorch/blob/main/etsformer_pytorch/etsformer_pytorch.py#L373