Expose gathering data state on page load

Feature Description

Until now, the concept of gathering data is something that we’ve calculated at runtime based on reporting data. In some cases, such as determining an initial layout it can be necessary to know whether a module is gathering data or not right away in order to avoid substantial changes as data is loaded.

While the goal is to make reporting data available right away on page load, we should do so in a way which keeps the fetching of such data async – that is, we should not be making API requests to query the gathering data state from the request serving the page itself.

Ideally this state would be available for the first page load once after module is connected, but if we can’t, it should be acceptable to have the gathering state immediately available for the next page load.

As part of this issue, it would be good to revisit the concept of gathering data itself which is generally a state that a module should only be in once – regardless of how long – for a given configured entity. E.g. If a module is connected and then we detect it is no longer gathering data, it shouldn’t be possible for that module to go back into a gathering data state again unless it’s connected entity (i.e. data source) changed. With that in mind, every eligible module would always start in a gathering state. As soon as we’re able to determine that the module is no longer in a gathering state, this would be persisted, after which we would no longer need to check. If the connection were later changed, the gathering state would be reset and the same process would repeat. This approach would be more efficient than the current implementation which only derives gathering state from requested report data (cached or freshly requested), so these queries are repeated every time the cache expires even after we detect the module is no longer gathering data.

This issue is relevant for the upcoming work for the Summary Widget (#5908) but not explicitly part of that epic.

---

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

• The “Gathering Data” state for Analytics and Search Console should be enhanced with a persistent server-side value which is available to the client on load
  • The server-side value should be implemented as a non-expiring Transient (googlesitekit_{$module_slug}_data_available), as the state itself is easily recalculatable
  • The stored value should be a simple flag that is set when data for the module is available (i.e. not-gathering)
    • There is no need to store a value when data is in a gathering state, this is only useful to know ahead of time that there already is data
  • The derivation of the gathering data state should still be determined on the client as it is today
• The module’s data availability transient should be deleted any time the connected entity changes (see below), the module is disconnected, or Site Kit is reset. Relevant entities are:
  • Search Console property
  • GA3 profile
  • GA4 measurement ID
• A new POST:data-available module datapoint should be added for modules that expose the state
  • This will be called to persist the available state. There is no particular parameter needed here because it should only set that data is available via the *_data_available transient
  • When data is gathering (or unknown), the transient should not exist
• The server-side data availability state should be selectable by a new isDataAvailableOnLoad selector which should always reflect the value on page load – it should not update during the page lifecycle under normal circumstances, i.e. if data is determined to be available
  • Since this value will be available on load, it should be set in the store’s initialState – it shouldn’t be received by a resolver. For testing purposes, it could have a respective action to receive it though if useful.
• isGatheringData selectors should receive corresponding state and resolvers. Selectors should be updated to return the value from state similar to other simple getters
  • A new action will be needed to receiveIsGatheringData(bool) (this can be used to simplify stories and tests for this state quite a bit but this should happen in a follow up issue)
  • In the new resolver:
    • If isDataAvailableOnLoad is true, the state should be received as false and do nothing else
    • If isDataAvailableOnLoad is false, it should select the same report it does today, and calculate the gathering state
      • If isGatheringData === true, the state should receive true and do nothing else
      • If isGatheringData === false, the state should receive false AND dispatch a fetch request to persist the data availability via the new module-specific REST route POST:modules/:identifier/data/data-available Note that this request should only be made when data was not available on load, and was then detected to be available

Note: this may introduce failures in E2E tests due to new/unhandled fetch requests when data is available.

Implementation Brief

IB For 5933

On PHP Side

• Create an interface Module_With_Data_Available_State in Google\Site_Kit\Core\Modules namespace.

  • It will have the following public methods:
    • setup_transients - sets the $transients instance variable.
    • is_data_available - returns whether the data_available transient is set.
    • set_data_available - sets the data_available transient.
    • delete_data_available - deletes the data_available transient.

• In Google\Site_Kit\Modules\Search_Console, Google\Site_Kit\Modules\Analytics and Google\Site_Kit\Modules\Analytics_4 class:

  • Add DATA_AVAILABLE_TRANSIENT constant with value defined in the AC.
  • Add $transients instance variable.
  • Add Module_With_Data_Available_State interface.
    • Implement setup_transients method.
      • It should set the $transients instance variable to a new instance of Google\Site_Kit\Core\Storage\Transients.
    • Implement is_data_available method.
      • It should return the result of calling get method on $transients instance variable with DATA_AVAILABLE_TRANSIENT constant as the first argument.
    • Implement set_data_available method.
      • It should call set method on $transients instance variable with DATA_AVAILABLE_TRANSIENT constant as the first argument and true as the second argument.
    • Implement delete_data_available method.
      • It should call delete method on $transients instance variable with DATA_AVAILABLE_TRANSIENT constant as the first argument.
  • In register method:
    • Call setup_transients method to setup the $transients instance variable.
    • Call is_data_available method to check if the data_available transient is set and append the information to the googlesitekitBaseData using googlesitekit_inline_base_data filter.
    • Add a googlesitekit_pre_save_settings_{module_slug} action to compare the current and new search console property/ga3 property/ga4 measurement id and call delete_data_available method if they are different.
  • In get_datapoint_definitions method:
    • Add POST:data-available definition.
  • In create_data_request and parse_data_response methods:
    • Call set_data_available method to set the data_available transient.

• In Google\Site_Kit\Modules\Analytics and Google\Site_Kit\Modules\Analytics_4 classes:

  • In addition to the things done in the above steps, also add the following:
  • In on_deactivation method:
    • Call delete_data_available method to delete the data_available transient.

On JS Side

• In assets/js/modules/search-console/datastore/report.js and assets/js/modules/analytics/datastore/report.js:
  • Add the serverside data available state from the global googleSitekitBaseData to the initialState object.
  • Add a new selector isDataAvailableOnLoad which returns the value of dataAvailableOnLoad from the state.
  • Update the isGatheringData selectors according to the details provided in the AC, notably:
    • It should have a state and a resolver. The resolver should do the following:
    • If isDataAvailableOnLoad is true, then isGatheringData should return false.
    • Otherwise, it should determine the gatheringDataState by similar means as it does currently.
      • If gatheringDataState is true, then isGatheringData should return true and do nothing else.
      • Otherwise, call the the newly added endpoint POST:data-available using API.set() and reurn false.

Test Coverage

• Add unit test for the newly added selectors.
• Update existing tests for isGatheringData to accommodate the changes.
• Fix any failing Unit or e2e tests.

QA Brief

Changelog entry

[eclarke1]

@kuasha420 could we please progress the IB here as it would be ideal to get this into Sprint 90 as the last issue remaining (other than #6180 which is just for the HaTS survey) cc @FlicHollis

[felixarntz]

ACs ✅