Poor Benchmark Performance

I’m evaluating various C# templating libraries. I saw that the Cottle library had a benchmark that included RazorLight, so I added RazorEngineCore to it.

The results are very bad, so I’m thinking there must be something wrong with how I wrote the benchmark.

Here are some of the results:

As you can see, RazorEngineCore is ~321 times slower than the next slowest library.

Here’s the implementation:

const string content = @"
<ul id='products'>
@foreach (var product in Model.Products)
{
  <li>
    <h2>@product.Name</h2>
    <p>@product.Description.Substring(0, System.Math.Min(product.Description.Length, 15)) - Only @product.Price.ToString(""f1"", System.Globalization.CultureInfo.CreateSpecificCulture(""en-US""))$</p>
  </li>
}
</ul>";

var context = new { Products = CompareEngine.Products };

return () =>
{
    var engine = new RazorEngine();
    return () => engine.Compile(content).Run(context);
};

For comparison, here’s the code for the RazorLight benchmark:

const string content = @"
<ul id='products'>
@foreach (var product in Model.Products)
{
  <li>
    <h2>@product.Name</h2>
    <p>@product.Description.Substring(0, System.Math.Min(product.Description.Length, 15)) - Only @product.Price.ToString(""f1"", System.Globalization.CultureInfo.CreateSpecificCulture(""en-US""))$</p>
</li>
}
</ul>";

var context = new { CompareEngine.Products };

return () =>
{
    var engine = new RazorLightEngineBuilder().UseProject(new StringRazorLightProject(content)).Build();
    return () => engine.CompileRenderAsync(string.Empty, context).Result;
};

[DanielStout5]

What about this?

In RazorEngineCore you do:

string fileName = string.IsNullOrWhiteSpace(options.TemplateFilename) ? Path.GetRandomFileName() : options.TemplateFilename;
RazorSourceDocument document = RazorSourceDocument.Create(templateSource, fileName);

I.e. it’s creating a file in the filesystem

Whereas the equivalent use of RazorSourceDocument in RazorLight is this:

RazorSourceDocument source = RazorSourceDocument.ReadFrom(stream, projectItem.Key);

(And note that in the above benchmark, it uses TextSourceRazorProjectItem, not the file system project item - so stream above comes from new MemoryStream(Encoding.UTF8.GetBytes(_content));)

So I don’t think the issue is just that RazorLight is caching here and that’s why it does better on the benchmark - it looks to me like RazorEngineCore will be reading and writing to the filesystem whereas RazorLight will just be using data held in memory. That alone seems like it could explain the benchmark performance.

This is the benchmark code for RazorLight:

var engine = new RazorLightEngineBuilder().UseProject(new StringRazorLightProject(content)).Build();
return () => engine.CompileRenderAsync(string.Empty, context).Result;

It’s not calling UseMemoryCachingProvider, so my impression is that it won’t cache the templates. I’ll dig into that a bit further to confirm - I haven’t looked much at the RazorLight source code.

Update: I wrote this test in the RazorLight repo:

[Fact]
public async Task CompileTextSource()
{
	var content = "Hi @Model.Name";
	var model = new { Name = "Bob" };
	var eng = new RazorLightEngineBuilder().UseProject(new TextRazorLightProject(content)).Build();
	var str1 = await eng.CompileRenderAsync("", model);

	var eng2 = new RazorLightEngineBuilder().UseProject(new TextRazorLightProject(content)).Build();
	var str2 = await eng.CompileRenderAsync("", model);
}

private class TextRazorLightProject : RazorLightProject
{
	private readonly string content;
	public TextRazorLightProject(string content)
	{
		this.content = content;
	}

	public override Task<IEnumerable<RazorLightProjectItem>> GetImportsAsync(string templateKey)
	{
		return Task.FromResult(Enumerable.Empty<RazorLightProjectItem>());

	}

	public override Task<RazorLightProjectItem> GetItemAsync(string templateKey)
	{
		return Task.FromResult(new TextSourceRazorProjectItem(templateKey, content) as RazorLightProjectItem);
	}
}

Which is how the Cottle benchmark works. I stepped through the code compiled that same template multiple times and I can confirm RazorLight is NOT caching the template with the above code (because IsCachingEnabled returns false) which means it really is hundreds of times faster than RazorEngineCore. My guess is that it’s because of the lack of writing to the filesystem which I described above.

[DanielStout5]

I don’t think an informal benchmark like that is an accurate representation of performance - like you say, when you set it up like that, the order of execution matters (and that doesn’t account for GC, JIT optimizations, etc). That’s why benchmark frameworks like BenchmarkDotNet (https://github.com/dotnet/BenchmarkDotNet) are used.

But no worries - I won’t take up any more of your time. Best of luck with your library!