Question : custom POST operation that return Collection of Entities using API Platform extensions

Hello,

I’m facing a problem, I can’t find a way to make a custom research POST endpoint using ApiPlatform :

Problem details

I need to make a big search query with a custom filter that takes an array of strings up to 300000 strings. This values represents an amount of approximately 4,29 MB of data. The query will return entities matching these strings by serializing them.

Example of current problematic request :

GET https://something.domain.tld/api/objects/search?thefilter=[123, 456, 789, lots of etc.]

The problem here as using this big filter in a GET request is it generates a huge URL that is not handled well by web browsers and lots of cloud providers services (ex : AWS Cloudfront). Cloud providers are restricting the URL size and headers size. Passing these values into a GET request body is not possible because lots of libraries and cloud providers services will block a such request.

I cannot pass these values into headers, URL parameters or into the request body of a GET request. I cannot split this query into multiple queries because this will result in too many queries.

Example of current expected behavior :

POST https://something.domain.tld/api/objects/search

PAYLOAD :
{
    "the_filter": ["123", "456", "789", "lots of etc."]
}

Considered solutions

So, I am considering two options :

• Find a way to use a data provider with a POST request by injecting extensions into the data provider .
• Making a POST request with a custom controller that uses API Platform extensions (by injecting extensions : like injecting extensions into a data provider ?

Questions

I have 3 questions :

• Is there a way to call a data provider with a POST request without making API platform trying to INSERT something on the database ?
• Is there a way to make a POST request with a custom controller and uses API Platform extensions into the controller (Pagination and Filter) like this part of the documentation but in the controller ?
• If you have a more elegant solution, please suggest it 😃

Current progress

• I didn’t succeed to find a way to use dataprovider with POST request
• I suceed to find a way to inject API Platform extensions into a controller (edited)

Thanks a lot for reading, Best regards,

[ES-Six]

- This comment is deprecated : see next comment bellow for a better solution.

After searching, a lot, I found a better solution :

• Make a standard GET endpoint
• And make a custom POST endpoint that calls the GET endpoint and pass the data in the payload
• And use a custom decorator to sync the documentation (filter parameters and results) of the POST endpoint using the GET endpoint openapi_context.

Explantation

The get endpoint provides a standard API Platform search endpoint.

The POST endpoint is a sort of redirection that allows sending payload data to the standard API Platform GET endpoint by making the API call itself the GET endpoint with the payload data received by the POST endpoint.

Exemple code

The example endpoint definition in an entity :

// in file src/Entity/YourEntity.php
// Declare normally a GET request then declare a POST request with a custom controller and a "SyncDocWith" tag in the openapi context.
collectionOperations: [
        ...
        'standardSearchQuery' => [
            "method" => "GET",
            "path" => "/the_get_endpoint",
            "openapi_context" => [
                "summary" => "Search objects wwith a GET request",
            ],
            "normalization_context" => ["groups" => ["object:read"]],
            "pagination_items_per_page" => 20
        ],
        'customBigSearchQuery' => [
            "method" => "POST",
            "path" => "/search/the_post_endpoint",
            "openapi_context" => [
                "tags" => ["Search", "SyncDocWith" => [
                    "method" => "GET",
                    "path" => "/optional_prefix/the_get_endpoint" // if your API has a prefix before the endpoints, you will need to include this prefix in the path here
                ]],
                "summary" => "Search objects with a POST request (and enable the ability to pass lot of datas in the payload to some custom filters)",
            ],
            "controller" => CustomSearchController::class,
            "normalization_context" => ["groups" => ["object:read"]]
        ],
        ...

The controller of the custom POST endpoint :

// in file src/Controller/CustomSearchController.php
<?php

namespace App\Controller;

use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\RequestStack;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Contracts\HttpClient\HttpClientInterface;

class CustomSearchController extends AbstractController
{
    public function __construct(private RequestStack $requestStack,
                                private HttpClientInterface $client) { }

    public function __invoke()
    {
        $request = $this->requestStack->getCurrentRequest();
        // Tips : replace the "localhost:8000" depending your needs and your environment
        // localhost:8000 is the API hostname and port where the API can be accessed localy.
        // This controller make an internal request locally on the server to pass the payload datas to the standard API Platform GET endpoint to allow using payload data in custom filters when an API is hosted behind CloudFront (because CloudFront block any GET request with payload data inside).
        $response = $this->client->request(
            'GET',
            "http://localhost:8000/api/the_get_endpoint?{$request->getQueryString()}",
             [
                 'body' => $this->requestStack->getCurrentRequest()->getContent(),
                 'headers' => [
                     'Authorization' => $request->headers->get('Authorization'), // Add headers you want to 
                     'Accept' => $request->headers->get('Accept')
                 ]
             ]
        );
        return new Response($response->getContent(), $response->getStatusCode());
    }
}

The custom openapi_context decorator

<?php
// in file src/OpenApi/HideRouteDecorator.php
namespace App\OpenApi;

use http\Exception\RuntimeException;
use Symfony\Component\Serializer\Normalizer\NormalizerInterface;

final class SyncDocDecorator implements NormalizerInterface
{
    private $decorated;

    public function __construct(NormalizerInterface $decorated)
    {
        $this->decorated = $decorated;
    }

    public function normalize($object, $format = null, array $context = [])
    {
        $docs = $this->decorated->normalize($object, $format, $context);

        foreach ($docs['paths'] as $pathName => $path) {
            foreach ($path as $operationName => $operation) {
                $tags = $operation['tags'] ?? [];
                if (is_array($tags) && in_array('SyncDocWith', array_keys($tags))) {
                    unset($docs['paths'][$pathName][$operationName]['tags']['SyncDocWith']);
                    if (!is_array($tags['SyncDocWith'])) {
                        throw new RuntimeException("Error : linked operation must be an array");
                    }
                    if (!isset($tags['SyncDocWith']['method'])) {
                        throw new RuntimeException("Error : linked operation array must include a method property");
                    }
                    if (!isset($tags['SyncDocWith']['path'])) {
                        throw new RuntimeException("Error : linked operation array must include a path property");
                    }
                    $linkpedOperation = $this->getOperationByName($tags['SyncDocWith'], $docs);
                    if ($linkpedOperation === null) {
                        throw new RuntimeException("Error : the linked operation {$tags['SyncDocWith']['method']} {$tags['SyncDocWith']['path']} was not found");
                    }
                    $linkedPathName = $linkpedOperation["pathName"];
                    $linkedOperationName = $linkpedOperation["operationName"];
                    $docs['paths'][$pathName][$operationName]['parameters'] = $docs['paths'][$linkedPathName][$linkedOperationName]['parameters'];
                    $docs['paths'][$pathName][$operationName]['responses'] = $linkpedOperation['path'][$linkpedOperation['operationName']]['responses'];
                    var_dump($docs['paths'][$pathName][$operationName]['requestBody']["content"]);
                    $docs['paths'][$pathName][$operationName]['requestBody']["content"] = [
                        'application/json' => [
                            'schema' => [
                                'type'=>'object'
                            ]
                        ]
                    ];
                }
            }
        }

        return $docs;
    }

    private function getOperationByName(array $operationToFind, iterable $docs): array|null {
        foreach ($docs['paths'] as $pathName => $path) {
            foreach ($path as $operationName => $operation) {
                if (strtolower($operationName) === strtolower($operationToFind["method"]) && $pathName === $operationToFind['path']) {
                    return [
                        "pathName" => $pathName,
                        "operationName" => $operationName,
                        "operation" => $operation,
                        "path" => $path
                    ];
                }
            }
        }
        return null;
    }

    public function supportsNormalization($data, $format = null)
    {
        return $this->decorated->supportsNormalization($data, $format);
    }
}

The custom decorator service definition :

    # in file config/services.yaml

    App\OpenApi\SyncDocDecorator:
        decorates: 'api_platform.openapi.normalizer.api_gateway'
        arguments: [ '@App\OpenApi\SyncDocDecorator.inner' ]
        autoconfigure: false

Environment

This code was tested with :

API Platform : 2.6.5 PHP: 8

Pros

• OpenAPI parameters and results examples are fully synced with the ones of the GET endpoint
• Very lightweight in RAM even if you have lots of objects because the POST endpoint is just making an HTTP GET request to the API itself.
• This approach allows your API to use lots of filters because you can use your own custom filters that read data in the payload. The only limit here is your API max_upload_size .

Approach caveat

• Custom post endpoint will be approximately 100~150ms slower than the GET endpoint due to the internal API request made by the custom controller. (but it doesn’t matter as the goal is to use a large amount of filters at the same time).

[ES-Six]

+ This is the most up to date solution for now.

@MaximeTaqt,

I did experimentations with DTO and it is working well. Input allow me do generate a documentation exemple of how to use the search endpoint and output is a reference to the auto generated GET endpoint schema.

For me, DataTransformer are a good solution, because they intend to transform data, so I use them to transform a search request to a Collection result ^^

In API Platform, we can use datatransformer to only transform an input DTO class to something that will be passed to a serializer class 😃

So this make DTO suitable because they are used to transform data and the Serializer is used after in this case 😃

It does respect the process according to the API Platform documentation and exemples : https://api-platform.com/docs/core/dto/

Here is an exemple of what I did :

My consts :

// In src/Service/SharedConst.php
...
    public const OPENAPI_PAGE_PARAMETER_NAME = 'page';
    public const OPENAPI_PAGE_PARAMETER = [
        "name" => self::OPENAPI_PAGE_PARAMETER_NAME,
        "in" => "query",
        "required" => false,
        "type" => "integer",
        'description' => 'The collection page number',
        "schema" => [
            "type" => "integer",
            "default" => 1
        ]
    ];

    public const OPENAPI_ITEMS_PER_PAGE_PARAMETER_NAME = 'itemsPerPage';
    public const OPENAPI_ITEMS_PER_PAGE_PARAMETER = [
        "name" => self::OPENAPI_ITEMS_PER_PAGE_PARAMETER_NAME,
        "in" => "query",
        "required" => false,
        "type" => "integer",
        'description' => 'The number of items per page',
        "schema" => [
            "type" => "integer",
            "default" => 30
        ]
    ];

    public const INTEGER_SCHEMA = ["type" => "integer"];
    public const ARRAY_OF_INTEGER_SCHEMA = [
        "type" => "array",
        "items" => self::INTEGER_SCHEMA
    ];
...

The entity :

/ in file src/Entity/YourEntity.php
// Declare normally a GET request with appropriate filters configured for it
// then declare a POST request using the same Output schema than the GET request in the documentation
// Like this :
collectionOperations: [
        ...
        'standardSearchQuery' => [
            "method" => "GET",
            "path" => "/resource",
            "openapi_context" => [
                "summary" => "Search objects wwith a GET request",
            ]
        ],
        'customBigSearchQuery' => [
            "method" => "POST",
            "path" => "/resource/search",
            'status' => 200,
            "openapi_context" => [
                "summary" => "Search objects with a POST request",
            ],
           "read" => false,
            "input" => SearchResourceInput::class,
            "output" => self::class,
            "openapi_context" => [
                "parameters" => [
                    SharedConst::OPENAPI_PAGE_PARAMETER,
                    SharedConst::OPENAPI_ITEMS_PER_PAGE_PARAMETER
                ],
                'responses' => [
                    '200' => [
                        'description' => 'Search for resources',
                        'content' => [
                            'application/json' => [
                                'schema' => [
                                    '$ref' => '#/components/schemas/Resource.read' // This is the exact schema used by the GET endpoint
                                ]
                            ]
                        ]
                    ]
                ]
            ]
        ],
        ...

The DTO class :

<?php

namespace App\Dto;

use ApiPlatform\Core\Annotation\ApiProperty;
use App\Service\SchemaConst;
use Symfony\Component\Serializer\Annotation\SerializedName;
use Symfony\Component\Validator\Constraints as Assert;

final class SearchResourceInput {
    // note : DTO approach allow us to validate the query which is a good point
   // The $id property (equivalent to 'exact' search filter in API Platform)
   // See why I rename it here : https://github.com/api-platform/api-platform/issues/343
    #[Assert\AtLeastOneOf(
        constraints: [
            new Assert\IsNull(),
            new Assert\Sequentially(
                constraints: [
                    new Assert\Type(type: 'int'),
                    new Assert\NotNull()
                ]
            ),
            new Assert\Sequentially(
                constraints: [
                    new Assert\All(
                        constraints: [
                            new Assert\Type(type: 'int'),
                            new Assert\NotNull()
                        ]
                    ),
                    new Assert\Count(min: 1)
                ]
            ),

        ]
    )]
    #[Assert\NotBlank(allowNull: true)]
    #[ApiProperty(attributes: [
        'openapi_context' => SchemaConst::ARRAY_OF_INTEGER_SCHEMA,
        'json_schema_context' => SchemaConst::ARRAY_OF_INTEGER_SCHEMA
    ])]
    public mixed $resourceId = null;

    // Another searchable property (equivalent to 'partial' search filter in API Platform)
    #[Assert\Type(type: 'string')]
    #[Assert\NotBlank(allowNull: true)]
    public mixed $city = null;
}

The DataTransformer class :

<?php

namespace App\DataTransformer;

use ApiPlatform\Core\Bridge\Doctrine\Orm\Extension\EagerLoadingExtension;
use ApiPlatform\Core\Bridge\Doctrine\Orm\Extension\FilterEagerLoadingExtension;
use ApiPlatform\Core\Bridge\Doctrine\Orm\Extension\FilterExtension;
use ApiPlatform\Core\Bridge\Doctrine\Orm\Extension\OrderExtension;
use ApiPlatform\Core\Bridge\Doctrine\Orm\Extension\PaginationExtension;
use ApiPlatform\Core\Bridge\Doctrine\Orm\Extension\QueryResultCollectionExtensionInterface;
use ApiPlatform\Core\Bridge\Doctrine\Orm\Util\QueryNameGenerator;
use ApiPlatform\Core\DataTransformer\DataTransformerInterface;
use ApiPlatform\Core\Validator\ValidatorInterface;
use App\Repository\Interfaces\GenericDataProviderRepositoryInterface;
use App\Service\SharedConst;
use Doctrine\ORM\EntityManagerInterface;
use JsonException;
use Symfony\Component\HttpFoundation\RequestStack;
use Symfony\Component\HttpKernel\Exception\UnprocessableEntityHttpException;

final class SearchResourceDataTransformer implements DataTransformerInterface
{
    public function __construct(
        private RequestStack $requestStack,
        private EntityManagerInterface $em,
        private ValidatorInterface $validator,
        // Autowirering collectionExtensions to DTO here :
        private FilterExtension $filterExtension, 
        private FilterEagerLoadingExtension $filterEagerLoadingExtension,
        private EagerLoadingExtension $eagerLoadingExtension,
        private OrderExtension $orderExtension,
        private PaginationExtension $paginationExtension
    ) { }

    /**
     * @throws JsonException
     */
    public function transform($object, string $to, array $context = []): iterable
    {
        $extensions = [
            $this->filterExtension,
            $this->filterEagerLoadingExtension,
            $this->eagerLoadingExtension,
            $this->orderExtension,
            $this->paginationExtension
        ];
        $this->validator->validate($object);

        $request = $this->requestStack->getCurrentRequest();
        if (is_null($request)) {
            throw new UnprocessableEntityHttpException('Cannot process empty request');
        }
        $filters = json_decode($request->getContent(), true, 512, JSON_THROW_ON_ERROR);

        // Remap id property filter to bypass issue : https://github.com/api-platform/api-platform/issues/343
        if (isset($filters['resourceId'])) {
            $filters['id'] = $filters['resourceId'];
            unset($filters['resourceId']);
        }

        // Pagination remapping
        $page = $request->get('page');
        if (is_numeric($page)) {
            $filters['page'] = $page;
        }
        $itemsPerPage = $request->get('itemsPerPage');
        if (is_numeric($itemsPerPage)) {
            $filters['itemsPerPage'] = $itemsPerPage;
        }

        // Set any other filters provided in the request
        $context = null === $filters ? $context : array_merge($context, ['filters' => $filters]);
        $queryNameGenerator = new QueryNameGenerator();
        $repository = $this->em->getRepository($context['resource_class']);
        if (!$repository instanceof GenericDataProviderRepositoryInterface) {
            throw new UnprocessableEntityHttpException('Given resource is not compatible with GenericDataProviderRepositoryInterface');
        }

        $queryBuilder = $repository->getQueryBuilderForDataProvider(
            $this->requestStack->getCurrentRequest()?->get(SharedConst::OPENAPI_ID_SCHOOL_PARAMETER_NAME)
        );

        if (is_null($queryBuilder)) {
            return [];
        }

        foreach ($extensions as $extension) {
            $extension->applyToCollection($queryBuilder, $queryNameGenerator, $context['resource_class'], $context['collection_operation_name'], $context);
            if ($extension instanceof QueryResultCollectionExtensionInterface && $extension->supportsResult($context['resource_class'], $context['collection_operation_name'], $context)) {
                return $extension->getResult($queryBuilder);
            }
        }

        return $queryBuilder->getQuery()->getResult();
    }

    public function supportsTransformation($data, string $to, array $context = []): bool
    {
        return isset($context['input']['class']) && SearchResourceInput::class === $context['input']['class'];
    }
}

So the only caveat of this approach are :

• There is a bug in how API Platform handle DTO with $id property in it. (but we can bypass that like showned in my exemple)
• This can take time to implement if you have a lot of filters

The pros are :

• This approach allows to add extra validation, this can help to solve a lot of additional use cases 😃
• DTO allow us to document and add a JSON exemple for the payload of the search endpoint in the documentation.
• DataTransformer are context aware and we can benefit from that without having to use the context builder as showned in my exemple
• The Output exemple is the documentation is synced with the exemple of the corresonding GET endpoint, so we keep the advantage of API Platform integrated automations.
• This kind of search endpoint is compatible with existing custom filters implemented for the GET endpoint of the entity (if you have one implemented).
• This approach use standard API Platform classes and processes so this will be easy to maintain in case of API Platform or Symfony upgrade.
• Dto seem’s to be more suitable than CustomControllers as stated in the API Platform socumentation here : https://api-platform.com/docs/core/controllers/#creating-custom-operations-and-controllers
  • Note: using custom controllers with API Platform is discouraged.
  • DTO are not discouraged so they are better.