question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Remove server-side REST batch data infrastructure

See original GitHub issue

With the legacy data API no longer used (see #2258), we should also remove the related server-side infrastructure for batch requests, and the data batch REST endpoint.


Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

  • Our batch endpoint data (defined in REST_Routes class) should be completely removed.
  • The Module::get_batch_data method should be entirely removed.
  • There is one usage of the above method that needs to continue to work, which is in the Analytics 4 GET:webdatastreams-batch REST datapoint. This usage should be replaced with a (much simpler) “manual” direct usage (nothing too abstracted) of the Google client library’s batch functionality. At a high-level:
    • Use with_client_defer so that requests from Module::create_data_request aren’t immediately executed but returned as request objects.
    • Use the createBatch method from the Google_Service instance to create a batch request.
    • Add the request objects to the batch.
    • Execute the batch and parse the results so that the overall response has the same format as now.
    • Restore original client defer mode.

Implementation Brief

  • Edit REST_Routes class:
    • Delete data endpoint declaration in the get_routes method.
    • Delete the error_to_response method.
  • Edit Module class:
    • Delete the get_batch_data method.
  • Update GET:webdatastreams-batch case in the Analytics_4::create_data_request method:
    • Grab the analyticsadmin service by calling $analyticsadmin = $this->get_service( 'analyticsadmin' ).
    • Create a batch request by calling $analyticsadmin->createBatch() method.
    • Loop through property IDs in the $data['propertyIDs'] list:
      • For each propertyID create web data streams request by calling $analyticsadmin->properties_webDataStreams->listPropertiesWebDataStreams( self::normalize_property_id( $data['propertyID'] ) )
      • Add web data stream request to the batch request using add method.
    • Return a closure that executes the batch request and returns the results of the execution.
  • Check the GET:webdatastreams-batch handler in the Analytics_4::parse_data_response method to make sure it returns the same results as it has been doing before.

P.S.: no need to call with_client_defer( true ) function explicitly, it will be done automatically for the GET:webdatastreams-batch endpoint.

Test Coverage

  • Remove ModuleTest::test_get_batch_data test.
  • Fix broken phpunit tests

Visual Regression Changes

  • N/A

QA Brief

Screen Shot 2021-07-26 at 4 57 20 PM

Changelog entry

  • Remove server-side REST batch data infrastructure.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
eugene-manuilovcommented, Jul 26, 2021

@ivankruchkoff no need to update datastores. What we need to do and what is missing in IB is that we need to unwrap results similar to how it was implemented in the legacy batch request method: https://github.com/google/site-kit-wp/blob/277034cb5213feb74fed30ad3e634aba8ce93874/includes/Core/Modules/Module.php#L350-L365

In our case, we need to add a new case for the 'GET:webdatastreams-batch' datapoint to the parse_data_response method that will loop through all batch results and compose the resulting array which consists of propertyIDs as keys and web data streams arrays as values. Web data streams need to be processed in the same way as we parse web data streams for the 'GET:webdatastreams' datapoint.

0reactions
asvinbcommented, Jul 30, 2021

QA: ✅

  • Disconnected Analytics module
  • Switched to develop branch
  • Proceeded to connect Analytics again with Devtools open
  • Copied request made to /wp-json/google-site-kit/v1/modules/analytics-4/data/webdatastreams-batch as a CURL command.
  • Response:
    {"281028198":[{"createTime":"2021-07-29T12:37:57.250Z","defaultUri":"https:\/\/competent-darkling.jurassic.ninja","displayName":"competent-darkling.jurassic.ninja","measurementId":"G-BKL2ZYKK0X","name":"properties\/281028198\/webDataStreams\/2723020063","updateTime":"2021-07-29T12:37:57.250Z","_id":"2723020063","_propertyID":"281028198"}]}
    
  • Switced to main branch.
  • Ran the same CURL command.
  • Response:
    {"281028198":[{"createTime":"2021-07-29T12:37:57.250Z","defaultUri":"https:\/\/competent-darkling.jurassic.ninja","displayName":"competent-darkling.jurassic.ninja","measurementId":"G-BKL2ZYKK0X","name":"properties\/281028198\/webDataStreams\/2723020063","updateTime":"2021-07-29T12:37:57.250Z","_id":"2723020063","_propertyID":"281028198"}]}
    
  • Both responses are the same.
Read more comments on GitHub >

github_iconTop Results From Across the Web

Patterns for handling batch operations in REST web services?
A simple RESTful pattern for batches is to make use of a collection resource. For example, to delete several messages at once. DELETE...
Read more >
Batch API | Google Cloud
taskGroups.tasks; REST Resource: v1.projects.locations.jobs ... An API to manage the running of batch jobs on Google Cloud Platform. REST ... Delete a Job....
Read more >
Web API implementation - Best practices for cloud applications
Learn about best practices for implementing a web API and publishing it to make it available to client applications.
Read more >
Using server-side encryption with AWS Key Management ...
You can use these KMS keys to protect your data in Amazon S3 buckets. When you use SSE-KMS encryption with an S3 bucket,...
Read more >
Server-side Tagging In Google Tag Manager - Simo Ahava
However, you are also removing all traces of what is actually done with the data bundled in the requests. Auditors will have a...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found