Large loopcounts in storyboard loops may allocate lots of memory

Describe the bug:

Ayane - Nageki no Mori is an example of a ranked storyboard that has a loop with a loopcount of 9999999. The sprite itself isn’t visible for that long, but the events in the loop stretch out to about 1.9 years. This causes lazer to continue allocating memory for the events until at least my computer crashes.

It gets stuck over at this function, which can be confirmed with a breakpoint. Note that closing osu in this state will leave the thread to continue allocating memory, so watch out for that.

In both diff files:

Sprite,Foreground,Centre,"sb\collabpart\_002.png",650,430.75
 L,0,9999999
  R,5,0,3000,-0.01,0.01
  R,5,3000,6000,0.01,-0.01

I don’t know if stable iterates through the loops differently, but it works fine there. There is no outro on stable because loops don’t affect outro time, but in the design editor the invisible sprite can be observed moving way beyond the end of the map. Haven’t tested to see if there is any limit, although not sure if a simple limit would be the correct solution.

Screenshots or videos showing encountered issue:

osu!lazer version:

Latest release: 2021.410.0

Logs:

not needed

[peppy]

The PR linked fixes loops at a framework level. Unfortunately, storyboard command and loop generation is being done locally at the osu! side:

https://github.com/ppy/osu/blob/e330052852a8692c769ed2152a5980a16679b576/osu.Game/Storyboards/StoryboardSprite.cs#L127-L160

I’ll take a quick look into whether we can switch this to using framework native loop support.

[peppy]

Here’s a very partly working diff which will fix the issue osu!-side:

diff --git a/osu.Game/Storyboards/CommandLoop.cs b/osu.Game/Storyboards/CommandLoop.cs
index 29e034d86c..9b0457d9f4 100644
--- a/osu.Game/Storyboards/CommandLoop.cs
+++ b/osu.Game/Storyboards/CommandLoop.cs
@@ -40,11 +40,10 @@ public CommandLoop(double startTime, int repeatCount)
 
         public override IEnumerable<CommandTimeline<T>.TypedCommand> GetCommands<T>(CommandTimelineSelector<T> timelineSelector, double offset = 0)
         {
-            for (int loop = 0; loop < TotalIterations; loop++)
+            foreach (var command in base.GetCommands(timelineSelector, offset))
             {
-                double loopOffset = LoopStartTime + loop * CommandsDuration;
-                foreach (var command in base.GetCommands(timelineSelector, offset + loopOffset))
-                    yield return command;
+                command.LoopCount = TotalIterations;
+                yield return command;
             }
         }
 
diff --git a/osu.Game/Storyboards/CommandTimeline.cs b/osu.Game/Storyboards/CommandTimeline.cs
index 0650c97165..a6659f82b6 100644
--- a/osu.Game/Storyboards/CommandTimeline.cs
+++ b/osu.Game/Storyboards/CommandTimeline.cs
@@ -54,6 +54,7 @@ public class TypedCommand : ICommand
             public Easing Easing { get; set; }
             public double StartTime { get; set; }
             public double EndTime { get; set; }
+            public int LoopCount { get; set; }
             public double Duration => EndTime - StartTime;
 
             public T StartValue;
diff --git a/osu.Game/Storyboards/CommandTimelineGroup.cs b/osu.Game/Storyboards/CommandTimelineGroup.cs
index d198ed68bd..bea640cb57 100644
--- a/osu.Game/Storyboards/CommandTimelineGroup.cs
+++ b/osu.Game/Storyboards/CommandTimelineGroup.cs
@@ -107,6 +107,7 @@ public virtual IEnumerable<CommandTimeline<T>.TypedCommand> GetCommands<T>(Comma
                     new CommandTimeline<T>.TypedCommand
                     {
                         Easing = command.Easing,
+                        LoopCount = command.LoopCount,
                         StartTime = offset + command.StartTime,
                         EndTime = offset + command.EndTime,
                         StartValue = command.StartValue,
diff --git a/osu.Game/Storyboards/StoryboardSprite.cs b/osu.Game/Storyboards/StoryboardSprite.cs
index 982185d51b..fb7ff10016 100644
--- a/osu.Game/Storyboards/StoryboardSprite.cs
+++ b/osu.Game/Storyboards/StoryboardSprite.cs
@@ -5,6 +5,7 @@
 using System.Collections.Generic;
 using System.Linq;
 using osu.Framework.Graphics;
+using osu.Framework.Graphics.Transforms;
 using osu.Game.Storyboards.Drawables;
 using osuTK;
 
@@ -98,7 +99,7 @@ public double EndTimeForDisplay
 
         private delegate void DrawablePropertyInitializer<in T>(Drawable drawable, T value);
 
-        private delegate void DrawableTransformer<in T>(Drawable drawable, T value, double duration, Easing easing);
+        private delegate TransformSequence<Drawable> DrawableTransformer<in T>(TransformSequence<Drawable> drawable, T value, double duration, Easing easing);
 
         public StoryboardSprite(string path, Anchor origin, Vector2 initialPosition)
         {
@@ -138,22 +139,23 @@ public void ApplyTransforms(Drawable drawable, IEnumerable<Tuple<CommandTimeline
             generateCommands(generated, getCommands(g => g.Rotation, triggeredGroups), (d, value) => d.Rotation = value, (d, value, duration, easing) => d.RotateTo(value, duration, easing));
             generateCommands(generated, getCommands(g => g.Colour, triggeredGroups), (d, value) => d.Colour = value, (d, value, duration, easing) => d.FadeColour(value, duration, easing));
             generateCommands(generated, getCommands(g => g.Alpha, triggeredGroups), (d, value) => d.Alpha = value, (d, value, duration, easing) => d.FadeTo(value, duration, easing));
-            generateCommands(generated, getCommands(g => g.BlendingParameters, triggeredGroups), (d, value) => d.Blending = value, (d, value, duration, _) => d.TransformBlendingMode(value, duration),
-                false);
 
-            if (drawable is IVectorScalable vectorScalable)
-            {
-                generateCommands(generated, getCommands(g => g.VectorScale, triggeredGroups), (_, value) => vectorScalable.VectorScale = value,
-                    (_, value, duration, easing) => vectorScalable.VectorScaleTo(value, duration, easing));
-            }
-
-            if (drawable is IFlippable flippable)
-            {
-                generateCommands(generated, getCommands(g => g.FlipH, triggeredGroups), (_, value) => flippable.FlipH = value, (_, value, duration, _) => flippable.TransformFlipH(value, duration),
-                    false);
-                generateCommands(generated, getCommands(g => g.FlipV, triggeredGroups), (_, value) => flippable.FlipV = value, (_, value, duration, _) => flippable.TransformFlipV(value, duration),
-                    false);
-            }
+            // generateCommands(generated, getCommands(g => g.BlendingParameters, triggeredGroups), (d, value) => d.Blending = value, (d, value, duration, _) => d.TransformBlendingMode(value, duration),
+            //     false);
+            //
+            // if (drawable is IVectorScalable vectorScalable)
+            // {
+            //     generateCommands(generated, getCommands(g => g.VectorScale, triggeredGroups), (_, value) => vectorScalable.VectorScale = value,
+            //         (_, value, duration, easing) => vectorScalable.VectorScaleTo(value, duration, easing));
+            // }
+            //
+            // if (drawable is IFlippable flippable)
+            // {
+            //     generateCommands(generated, getCommands(g => g.FlipH, triggeredGroups), (_, value) => flippable.FlipH = value, (_, value, duration, _) => flippable.TransformFlipH(value, duration),
+            //         false);
+            //     generateCommands(generated, getCommands(g => g.FlipV, triggeredGroups), (_, value) => flippable.FlipV = value, (_, value, duration, _) => flippable.TransformFlipV(value, duration),
+            //         false);
+            // }
 
             foreach (var command in generated.OrderBy(g => g.StartTime))
                 command.ApplyTo(drawable);
@@ -225,8 +227,13 @@ public void ApplyTo(Drawable drawable)
 
                 using (drawable.BeginAbsoluteSequence(command.StartTime))
                 {
-                    transform(drawable, command.StartValue, 0, Easing.None);
-                    transform(drawable, command.EndValue, command.Duration, command.Easing);
+                    // TODO: how the FUCK to apply a loop to this.
+                    // command.LoopCount
+                    var sequence = transform(drawable.Animate(), command.StartValue, 0, Easing.None);
+                    sequence = transform(sequence, command.EndValue, command.Duration, command.Easing);
+
+                    if (command.LoopCount > 0)
+                        sequence.Loop(command.LoopCount);
                 }
             }
         }

Unfortunately this isn’t the end of our worries. Because storyboards are allowed to rewind, things grind to a half quite quickly as the loop is duplicating itself and increasing the number of transforms until it gets to a point of slowing down to shit:

Going to require further thought and probably further framework side work as well.