Just some ideas for making this library better

First off all, i really like this library, the intent-based handling is really intuitive, i really liked it! 👍 I woud like to help to make this library better and i had a few ideas:

• Use endpoint routing
• Controller pattern
• System.IO.Pipelines support

Endpoint routing

I noticed that since asp net 3.0, there is a endpoint middleware which simplifies routing configuration and looks like this:

app.UseEndpoints(endpoints =>
{
    endpoints.MapRazorPages();
    endpoints.MapControllers();
    endpoints.MapHealthChecks("/health")
    endpoints.MapHub<ChatHub>("/chathub");
    endpoints.MapGet("/helloworld", async context => await context.Response.WriteAsync("Hello World!"));
});

and i would propose switching to this, combined with controller pattern, for example:

app.UseEndpoints(endpoints =>
{
    // new approach while also using controller pattern:
    endpoints.MapTusController<MyTusController>("/myfiles");
    endpoints.MapTusController<VideoFileTusController>("/videofiles");

    // alternative new approach (more similar to old one):
    endpoints.MapTus("/files", httpContext => new DefaultTusConfiguration());
});

//old approach:
app.UseTus(httpContext => new DefaultTusConfiguration())

this allows registering multiple controllers at different endpoints

Controller pattern

The current pattern uses TusDiskStore and events, which works, but things like DI accessing contexts and authorization are more difficult, for example you have to use the OnAuthorizeAsync event instead of doing authorization together with ITusStore logic. I would propose to combine the authorization and request logic and TusDisk Store to something like a “TusController” that works similar to the asp net mvc controlers, but is tus-compliant. I wrote some pseudo code of how i imagined it would look like:

public class MyTusController : TusController, ITusCreationController // keep the interface based extension support, just like with TusDiskStore
{

    // dependency injection is easily possible
    private readonly DbContext _dbContext;
    private readonly FileStorageService _fileStorage;
    private readonly IAuthorizationService _authService;

    public MyTusController(DbContext dbContext, FileStorageService fileStorage, IAuthorizationService authService)
    {
        _dbContext = dbContext;
        _fileStorage = fileStorage;
        _authService = authService;
    }


    public override async Task<ITusActionResult> GetFileAsync(string fileId)
    {
        // access to httpcontext, tuscontext and user properties
        this.HttpContext;
        this.TusContext; // this could store file info, metadata etc.
        this.User;

       // authorize
       var authResult = _authService.AuthorizeAsync(this.User, "MyAuthPolicy");
       if (!authResult.Success) return TusUnauthorized();

        // ... do some logic
        ITusFile file = await _fileStorage.GetFile(fileId);

        // return result
        return TusFile(file);

        // other result types may be:
        return TusSuccess();
        return TusNotFound();
        return TusUnauthorized();
        return TusNotFound();
        // etc...
    }

    public override async Task<ITusActionResult> CreateFileAsync()
    {
        // ... do some logic
        var filename = this.TusContext.Metadata["filename"];
        ITusFile file = _fileStorage.CreateFile(filename);
        _dbContext.Add(file);

        // in this case we return success with the file id
        return TusSuccess(file.Id);
    }

    //...

    // methods that arent on a per-request basis stay the same as in TusDiskStore
    public async Task<DateTimeOffset?> GetExpirationAsync(string fileId, CancellationToken _)
    {
        //...
    }
}

System.IO.Pipelines support

this is basically a better alternative to system.io.streams. The problem with system.io.streams is that if i want to write to a memorystream, a copy will be created in the internal buffer, and during reading, a copy of the internal copy will be returned to the consumer. Thats just unneccrssary copying steps and buffer allocaions, and system.io.pipelines solves this by allowing to write and read directly to the raw memory using Memory<byte> GetMemory(int sizeHint). The asp net core webserver engine kestrel already swithed to system.io.pipelines and i would reccomend TusDotNet to do the same.

What are your thoughts on this? 😃

[Hawxy]

Any feedback on my comments above? Specifically endpoint routing, authorization and the store implementation would be greatly appreciated.

Not the OP but I’ll provide some feedback as I recommend this library on a regular basis:

Endpoint routing

Endpoint routing support would be great!

we need to support multiple pipelines (Owin, ASP.NET Core 1.x, 2.x etc)

Maybe it’s worth considered how valuable the extra targets are at this point. NET Core 1.x has been End-of-Life for years now. .NET Core 2.1 LTS is EoL as of august this year. At the bare minimum there should be a .NET Standard 2.0 target. David Fowler has been asking library developers to target .NET Standard 2.0 for ages because 1.x brings in a bucketload of nonsense for .NET Core consumers and results in security false-positives for those of us using vulnerability scanners like Snyk.

I also have to question if anyone is going out of their way to integrate tus into .NET Framework projects at this point in the lifecycle.

Store implementation

I think this a simple solution that allows for proper DI without overcooking it. Works for me.

This would allow for easy basic integration while still allowing more complex scenarios like today.

Microsoft’s library guidance for Endpoint integration has some thoughts on this topic: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/routing?view=aspnetcore-5.0#guidance-for-library-authors

If the extension simply returns IEndpointRouteBuilder, then the consumer can manage configuration of authorization themselves without any additional work being required on your behalf. Correct me if I’m wrong, but I’m pretty sure the modern authorization policy configuration covers just about everything the OnAuthorizeAsync event can.

[smatsson]

Hi!

Thank you for your kind words, your engagement and, frankly, your patience with this issue! I wholeheartedly like that you bring new points of view to things that I take for granted. 😃

There are things in your post that I like and some that I think need some more understanding from my part. I’ll split my answer into each of your headlines. Feel free to correct me if I misunderstood something!

Endpoint routing

This has been discussed earlier and while it would give us some perks, like using variables in our route, it would give us two ways of doing the same thing, which I’m not really a fan of. As background, endpoint routing support hasn’t been added due to that we need to support multiple pipelines (Owin, ASP.NET Core 1.x, 2.x etc) while endpoint routing is only available for ASP.NET Core 3.x+, i.e. we would still need to keep our old code. I’m open for adding this to newer versions of tusdotnet, e.g. netstandard2.1/net5. This in turn would require new build targets etc which is a bit of a pain, but totally doable. As pointed out in #118 it’s not super intuitive why it sometimes does not work and since endpoint routing is the new defacto standard I think it would be good to integrate with it.

Controller pattern

I like the idea but I think that the example is oversimplifying things. The tus protocol is not as straight forward as the pseudo code makes it look. It’s not just a matter of “CreateFileAsync” and return “TusSuccess” and be done with it. There are actually lots of ifs and buts. The current architecture of tusdotnet separates the handling of the protocol (TusProtocolHandlerIntentBased) and the storage of file (e.g ITusStore implemented by TusDiskStore). This is an important separation as it allows for different storing backends that are themselves responsible for how to store files, how ids are generated etc. As an example we might need to generate different types of IDs depending on if the store stores files locally (e.g. TusDiskStore) or is using some database with primary key constraints etc.

Pros:

• Could use response formatters, authorize attribute, DI etc that one is used to
• People are more used to this approach as they write controllers all day long
• Automatic support for e.g. authorization filters.

Cons:

• Added complexity with MVC given filters, model binders etc that could interupt the request flow
• Very hard to maintain with that many extension points
• Breaks the protocol/store barrier, where the store can be replaced with something other than writing files on disk.
• Harder to use than just doing app.UseTus (subjective I agree)
• Breaking changes all over 😃

When it comes to DI I think you have several valid points which makes tusdotnet harder than necessary to use:

• The store implementation is split across several interfaces, resulting in multiple things needed to be injected into your service/controller.
• Authorization needs to be custom. This is due to ASP.NET having multiple ways of doing authorization/authentication between pipelines.
• The current way to do DI for tusdotnet is to either subclass DefaultTusConfiguration and use your implementation or use the HttpContext provided for each event. I can see how this is not intuitive for people when they are used to doing controllers and constructor arguments.

I could see a compromise of the two patterns becoming a reality where the events are part of the controller’s methods and could be overriden. I’m not sure if this brings anything new to the table though, except for constructor injection that people are used to? Example:

public class MyController : TusController
{
    public MyController(/*dependency injected stuff*/) {}

    public override async Task<IActionResult> OnBeforeCreateFile(string fileId, Metadata metadata, CancellationToken cancellationToken)
    {
        if(SomethingIsWrong(metadata))
            return BadRequest();

        // tusdotnet would take the Ok response here and transform it to the correct response according to the protocol. Same with BadRequest above.
        return Ok();
    }
}

Possible solutions for these issues

Note that these would ony be available on newer pipelines, e.g. Core 3.1 or .NET 5.

Store implementation

Implement a kind of “wrapper” or “proxy” that one can use to inject the store into services/controllers. Something like this (name stolen from your example):

// Somewhere during setup of DI
ITusStore store = new TusDiskStore(...);
var storageService = new TusStorageService(store);
services.AddSingleton<TusStorageService>(storageService);
services.AddSingleton<ITusStore>(store);

// During setup of tusdotnet
app.UseTus(context => new DefaultTusConfiguration { Store = context.GetRequiredService<ITusStore>() });

// In some custom service

public class MyService
{
    TusStorageService _storage;
    public MyService(TusStorageService storage)
    {
        _storage = storage;
    }

    public async Task SomeMethod(string fileId)
    {
        // Capabilities would be used to check if the current store supports something
        if(storage.Capabilities.DeleteFile)
        {
            await storage.DeleteFileAsync(fileId);
        }
    }
}

Authorization

The problem with using the controller pattern and using an Authorize attribute is that it’s not a 1:1 match between what the client wishes to do and a method in a controller. For example you cannot do the below as there is no way of determining if the client is going to concatenate multiple files or create a new file (maybe we don’t care?). We could limit ourselves to the http method but that would not cover all currently supported extensions of the protocol (and more might be added in the future)

public class MyController : TusController
{
    [Authorize("PolicyForCreatingFiles")]
    [HttpPost]
    public IActionResult HandlePostRequest()
    {
        // Side question: What code would you as a developer put here?
    }
}

One solution that I can think of to help with this issue is to create some kind of builder that can “automatically” create the OnAuthorize event handler. Something like this:

app.UseTus(httpContext => new DefaultTusConfiguration
{
    Events = new Events 
    {
        OnAuthorize = OnAuthorizeContext.RequirePolicy("MyPolicy").ForIntent(Intent.WriteFile).RequirePolicy("MyWriteFilePolicy").Create()
    }
})

What do you think? It’s not as easy as just using the Authorize attribute but it gives the developer both easier access and full customizability.

System.IO.Pipelines support

This has been discussed earlier aswell but I think that your statement is a bit of an oversimplification. Pipelines and Streams are just two different types of abstraction around data and work in different ways. I wouldn’t say that Pipelines is necessarily “better” than streams in all cases. It all depends on what you are going to use it for. For instance, streams can be seekable (i.e. you can go back and forth in the data) while pipelines is “one way”.

Since pipelines can be used as a buffer manager around data (e.g. a http requests) it makes sense that Kestrel uses it as the memory needed for parsing incoming http requests can be optimized in a way so that data no longer needed can be discarded (or rather returned to the pool). In tusdotnet there is really no parsing of incoming data (except for some headers where we already get a full string). All we do is take the data from the http request, load it into a buffer, and when the buffer is full we write the data to disk. We could definitely start using pipelines instead but I think the performance implications of this would be minimal, except maybe for slightly lower memory usage for slightly slower uploads. Since changing the ITusStore contract is a breaking change for external stores, I’m hestiant to make that change. That being said, there are probably many places in the tusdotnet code that can be optimized. I know for a fact that we do some unnecessary memory copies in TusDiskStore. Feel free to provide a POC for changing to pipelines where a performance gain can be observed and I’ll gladly consider the change. Maybe one could change the implementation of the stream provided to the stores so that one can get the “raw” pipereader instead?