How to read alignment graph?

Trying to understand what the axes and color legend means

• From reading other issues, I understand that a diagonal line means good alignment, but what exactly is happening at each x,y value and color of the graph?
• Is the color legend the alignment value, and higher means better?

[NTT123]

There are two important things here.

(1) the encoder (y-axis) which at each step takes an input character and its current state, to output a real vector representing the status of the brain at that moment. The input length here is about 100, so there are about 100 vectors generated by the encoder.

(2) the decoder (x-axis) takes all the vectors (the y-axis) to generate audio frames (mel-spectrogram). The decoder also works step-by-step, at each step (there are about 80-90 steps here) it would decide which vectors (on the y-axis) are important to create audio frames at that particular moment. Bright colors mean to focus more here (on the y-axis) and vice versa.

In short, the encoder reads input characters step-by-step and outputs status vectors. The decoder reads all status vectors and generates audio frames step-by-step.

A good alignment simply means: An “A” sound generated by the decoder should be the result of focusing on the vector generated by the encoder from reading “A” character. The diagonal line is the result when audio frames are created by focusing on the correct input characters in order.

[NTT123]

@reallynotabot,

(1) the number of decoding steps is determined by the training audio sample. For example, each decoding step generates, say, 5 frames, if an audio sample is about 400 audio frames, then there are 400/5 = 80 decoding steps.

(2) the decoder has to learn which vectors are important. This is the training. Technically, this is an attention mechanism, see here for details.

(3) At each decoding step, the whole y-axis is a weighted sum. All the colors add up to 1.0. The focused vector is actually the weighted average of all encoder’s status vectors.

Yes, vector 60 adds very little to the average vector (because the weight is very small) at the 20 decoding step.