Publishing to and Subscribing from multiple Pub/Sub projects is extremely difficult

Is your feature request related to a problem? Please describe. We publish and subscribe to/from topics and subscriptions in other projects, and the libraries do not support this at all well, especially with the Pub/Sub emulator complication.

Please have a look at how we did it, to get a sense of how difficult it was to meet the basic requirement of being able to specify a project for a topic to publish to, or a subscription to pull from: https://github.com/ONSdigital/ssdc-rm-caseprocessor/pull/50

Describe the solution you’d like It would be ideal if there was simply an optional “project ID” parameter on the PubSubInboundChannelAdapter class constructor, in addition to the subscription name. For example:

new PubSubInboundChannelAdapter(pubSubTemplate, subscriptionName, projectId);

Likewise, it would be ideal if there was an optional “project name” parameter PubSubTemplate.publish() function. For example:

pubSubTemplate.publish(topic, message, projectId);

This would make it very trivial to be able to publish to a topic in a different project, or subscribe to a subscription in a different project.

Describe alternatives you’ve considered We have written a lot of config to work around this, as you can see here: https://github.com/ONSdigital/ssdc-rm-caseprocessor/pull/50

In my opinion, you shouldn’t have to write so much code just to specify the project.

Additional context There’s not a lot of documentation around this. I had to dig in the source code to arrive at the solution, which in fact is not complete in the PR I attached… we had to do extra stuff because the GcpPubSubProperties and PubsubTransportConfig behaved in a really unexpected way in production. We ended up bringing more and more of the GcpPubSubAutoConfiguration class into our code, to make it do what we need.

Our requirement is fairly simple: to be able to do integration tests with the pubsub emulator, to be able to run on our local machine with the pubsub emulator, and then to be able to deploy to GCP with very similar code… except we’ve ended up with a lot of code having to be in our “prod” codebase just for testing, which is really not ideal. It should be easier.

We appreciate the help with this. There’s not a lot of stuff in the docs or in the wider community about how to achieve what we need. I found just 1 (one) single stackoverflow post about this: https://stackoverflow.com/questions/61495154/how-to-subscribe-to-multiple-google-pubsub-projects-in-spring-gcp

[meltsufin]

@NickGrantONS I’m glad the suggestion worked well for you! I’m in favor of making improvements to the documentation, but I’m not sure about the proposal to significantly expand the public API of PubSubTemplate. This can potentially have the unintended consequence of making it more confusing for a lot of users. Note that the existing methods already point out the FQN support in the javadoc. We can probably do more in the reference documentation, however.

[NickGrantX]

@meltsufin we have changed our project to use fully qualified topics & subscriptions, when they are in another project, and it works very well. It’s cut down the complexity a whole lot: https://github.com/ONSdigital/ssdc-rm-caseprocessor/pull/56

However, I think the interface could be a lot clearer so that it’s obvious that project can be specified. The line in the documentation you referred me to was very easy to miss.

I’ve sent a pull request with a suggested improvement, which would allow developers to more easily understand that a project can be supplied as either ProjectSubscriptionName or ProjectTopicName.