Events no longer sent to application service, and no way to debug

Description

Copying from #5957 (comment)

I’m trying to write an appservice, and when I first started I was getting transaction PUTs. The request would fail (the AS I’m writing is very much WIP), and I’d see logs emitted by the scheduler indicating increased waiting periods for trying again. But now I’m in a state where it doesn’t even try, and I haven’t been able to figure out how to make it try. #1834 isn’t my issue as appservice_stream_position.event_ordering is the same as events.stream_ordering in my case.

After poking around in the DB on my own trying to make sense of it, I no longer have a row in application_service_state (similarly to https://github.com/matrix-org/synapse/issues/5957#issuecomment-546220041), even though synapse does still recognize it at startup. Restarting synapse hasn’t made any difference.

2022-06-27 00:16:34,137 - synapse.config.appservice - 90 - INFO - main - Loaded application service: ApplicationService: {'token': '<redacted>', 'url': 'http://localhost:5000', 'hs_token': '<redacted>', 'sender': '@neofarkas:cit.chat', 'namespaces': {'users': [Namespace(exclusive=True, regex=re.compile('neofarkas.*'))], 'aliases': [], 'rooms': []}, 'id': 'dc863c3be4a6b4eac0d881b2df244920', 'ip_range_whitelist': None, 'supports_ephemeral': False, 'msc3202_transaction_extensions': False, 'protocols': set(), 'rate_limited': True}

I haven’t yet even figured out a way to begin troubleshooting, really.

One update from the original comment: I left everything running for a couple days and after I came back to it there was one more attempted PUT to my AS, but it failed and there haven’t been any further attempts, and no logs indicate what the problem is.

Steps to reproduce

I have a homeserver that is configured mostly with defaults; no identity service is running, and I don’t even understand most of the commented-out elements in the configuration file.

I did create and configure a registration for my appservice, and the homeserver appears to be recognizing the registration YAML correctly.

Homeserver

cit.chat

Synapse Version

1.61.0

Installation Method

Debian packages from packages.matrix.org

Platform

Azure VM, Standard B1ms (1 vcpu, 2 GiB memory) Ubuntu 20.04

Synapse running under systemd. Current status:

● matrix-synapse.service - Synapse Matrix homeserver
     Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-06-28 20:02:08 UTC; 1 day 20h ago
   Main PID: 26621 (python)
      Tasks: 20 (limit: 2289)
     Memory: 353.3M
     CGroup: /system.slice/matrix-synapse.service
             └─26621 /opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matr>

Jun 28 20:02:05 cit-matrix-chatty matrix-synapse[26615]: Found file in config directory that does not end in '.yaml': '/etc/matrix-synapse/conf.d/slack-regis>
Jun 28 20:02:07 cit-matrix-chatty matrix-synapse[26621]: This server is configured to use 'matrix.org' as its trusted key server via the
Jun 28 20:02:07 cit-matrix-chatty matrix-synapse[26621]: 'trusted_key_servers' config option. 'matrix.org' is a good choice for a key
Jun 28 20:02:07 cit-matrix-chatty matrix-synapse[26621]: server since it is long-lived, stable and trusted. However, some admins may
Jun 28 20:02:07 cit-matrix-chatty matrix-synapse[26621]: wish to use another server for this purpose.
Jun 28 20:02:07 cit-matrix-chatty matrix-synapse[26621]: To suppress this warning and continue using 'matrix.org', admins should set
Jun 28 20:02:07 cit-matrix-chatty matrix-synapse[26621]: 'suppress_key_server_warning' to 'true' in homeserver.yaml.
Jun 28 20:02:07 cit-matrix-chatty matrix-synapse[26621]: --------------------------------------------------------------------------------
Jun 28 20:02:07 cit-matrix-chatty matrix-synapse[26621]: Config is missing macaroon_secret_key
Jun 28 20:02:08 cit-matrix-chatty systemd[1]: Started Synapse Matrix homeserver.

Relevant log output

2022-06-27 00:16:34,137 - synapse.config.appservice - 90 - INFO - main - Loaded application service: ApplicationService: {'token': '<redacted>', 'url': 'http://localhost:5000', 'hs_token': '<redacted>', 'sender': '@neofarkas:cit.chat', 'namespaces': {'users': [Namespace(exclusive=True, regex=re.compile('neofarkas.*'))], 'aliases': [], 'rooms': []}, 'id': 'dc863c3be4a6b4eac0d881b2df244920', 'ip_range_whitelist': None, 'supports_ephemeral': False, 'msc3202_transaction_extensions': False, 'protocols': set(), 'rate_limited': True}

Anything else that would be useful to know?

Important to remember that the appservice I’m working on is in development, so it’s expected that sometimes requests from the homeserver are going to fail.

[DMRobertson]

In short an AS will only get told about events:

• if the AS is interested in the event’s sender;
• if the event is a membership change for a user the AS is interested in; or
• if the event is in a room that the AS is interested in.

From the configuration posted in this issue’s description:

2022-06-27 00:16:34,137 - synapse.config.appservice - 90 - INFO - main -
 Loaded application service: ApplicationService: 
{
  'token': '<redacted>',
  'url': 'http://localhost:5000',
  'hs_token': '<redacted>',
  'sender': '@neofarkas:cit.chat',
  'namespaces': {
    'users': [
      Namespace(exclusive=True,
      regex=re.compile(
      'neofarkas.*'
      ))
    ],
    'aliases': [],
    'rooms': []
  },
  'id': 'dc863c3be4a6b4eac0d881b2df244920',
  'ip_range_whitelist': None,
  'supports_ephemeral': False,
  'msc3202_transaction_extensions': False,
  'protocols': set(),
  'rate_limited': True
}

This is an application service which is interested in

• Its sender user, @neofarkas:cit.chat
• Any user whose matrix ID matches the regex neofarkas.*. (It claims to have control over this namespace!)
• No rooms.

So I think you’re not seeing events sent to that application service, because your application service hasn’t registered an interest in seeing events for anyone whose ID doesn’t begin with neofarkas. Try changing the config so that e.g. your app service listens to all users on cit.chat with

namespaces:
  users:
    exclusive: false
    regex: ".*:cit.chat"

and let me know if that helps.

(The docs https://matrix-org.github.io/synapse/latest/application_services.html could probably be improved. It speaks of a group_id which I don’t recognise from the source code??!)

[DMRobertson]

Great, thanks for following up.