Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Push Notifications getting too many "504 Gateway Time-out" errors

See original GitHub issue

Summary

Since February, I’m getting A LOT of errors responses while sending push notifications using Expo Notifications Tokens in the server.

I’m getting exactly this error:

Error: Expo responded with an error with status code 504: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>

    at Expo.<anonymous> (/workspace/node_modules/expo-server-sdk/build/ExpoClient.js:234:30)
    at Generator.next (<anonymous>)
    at /workspace/node_modules/expo-server-sdk/build/ExpoClient.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/workspace/node_modules/expo-server-sdk/build/ExpoClient.js:4:12)
    at Expo.getTextResponseErrorAsync (/workspace/node_modules/expo-server-sdk/build/ExpoClient.js:233:16)
    at Expo.<anonymous> (/workspace/node_modules/expo-server-sdk/build/ExpoClient.js:222:35)
    at Generator.next (<anonymous>)
    at fulfilled (/workspace/node_modules/expo-server-sdk/build/ExpoClient.js:5:58)
    at runMicrotasks (<anonymous>)

And some times also this other one:

Error: Expo responded with an error with status code 504: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>

    at Expo.<anonymous> (/workspace/node_modules/expo-server-sdk/build/ExpoClient.js:234:30)
    at Generator.next (<anonymous>)
    at /workspace/node_modules/expo-server-sdk/build/ExpoClient.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/workspace/node_modules/expo-server-sdk/build/ExpoClient.js:4:12)
    at Expo.getTextResponseErrorAsync (/workspace/node_modules/expo-server-sdk/build/ExpoClient.js:233:16)
    at Expo.<anonymous> (/workspace/node_modules/expo-server-sdk/build/ExpoClient.js:222:35)
    at Generator.next (<anonymous>)
    at fulfilled (/workspace/node_modules/expo-server-sdk/build/ExpoClient.js:5:58)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

There is a huge blocking issue with this: If the response is an Error 504, I don’t get the Tickets, so I can’t know if the push notifications where actually sent or not.

I’m sending notifications using Chunks, and seems like when I get that error, SOME of the notifications in the bunch get send, and some others not, but it’s impossible to know it. So, what I have to do is to re-send the bunch again, resulting in duplicated push notifications (that as you may imagine, users are complaining a lot)

You may thing that a way to fix this problem is:

Send the notifications slowly, one by one instead of sending them in parallel,
Wait some exponential time after every bunch sent
Not sending notifications in “busy” times (as it hours XX:00)

I am already doing all that things but I still have the problem.

During the last 30 days I even implemented a change in the app to totally avoid sending more than 1 bunch of notifications every 3 minutes.

So, every 3 minutes I check if there are notifications to send, and I only select 20 or 50 of them, I create a chunk, and I send it. Then I repeat the process 3 minutes after… And I still have that problem. Only 1 call every 3 minutes with 50 tokens as much! That’s really really weird.

I don’t know what else I can do to reduce this problem.

If you implement an endpoint where I can check if the server is busy, I am totally willing to use it, in order to not add more notifications and wait some minutes.

I am even willing to pay for a premium server where I can ensure that it’s never busy and I know the notifications are not going to be duplicated.

What platform(s) does this occur on?

Not Applicable

Environment

Not applicable

Minimal reproducible example

Not applicable. I guess, but this is my code.

In my app I have some “scheduled” notifications. something like: “Send this notification in 30 hours from now”, that’s why I query notifications to the DB.

import { Expo } from 'expo-server-sdk'; // expo-server-sdk@3.6.0 right noe

const processScheduledNotifications = async (_req, res) => {
  try {
    const expo = new Expo();

    const NOW = Date.now();
    const tinutesRightNow = new Date().getMinutes();

    const notificationsInDB = await actionsPushNotifications.getBy({
      where: [
        { _state: 1 }, // notifications pendings to be sent.
        ['data.sendAt', '<', NOW + 1] // get notifications that needs to be send now.
      ],
      limit: tinutesRightNow > 20 ? 50 : 20, // between xx:00 and xx:20 I send only 20 notifications, and 50 after that.
    });

    if (notificationsInDB.length === 0) {
      return res.send({ status: 'ok' });
    }

    const messagesToExpo = [];

    // Just building the regular notification object
    notificationsInDB.forEach(noti => {
      const message = {
        to: noti.data.token,
        title: noti.data.title,
      };

      if (noti.data.body) {
        message.body = noti.data.body;
      }

      if (noti.data.data) {
        message.data = noti.data.data;
      }

      messagesToExpo.push(message);
    });

    // Creating the chunk, which is actually only gonig to be 1 chunk with 20 or 50 as max.
    const chunks = expo.chunkPushNotifications(messagesToExpo);

    // Here I'm going to save the Tickets.
    const expoResponses = [];

    // This for each doesn't makes sense. It's only 1 chunk... but... will refactor later.
    for (let chunk of chunks) {
      try {
        // THIS IS FAILING!
        const ticketChunk = await expo.sendPushNotificationsAsync(chunk);
        expoResponses.push(...ticketChunk);
      } catch (error) {
        // If fails, just create some fake tickets with data letting me know that they failed, and add that data to the DB object.
        expoResponses.push(
          ...new Array(chunk.length).fill({
            errorWhileSending: true,
            error: error.message,
          })
        );
      }
    }

    // Now, I will add all those tickets to each notification that I wanted to send
    const proms = [];
    expoResponses.forEach((expoResponse, i) => {
      // The `expoReponse` is the ticket from Expo (if doesn't fail), or the faked object created in the catch.
      const newNotiData = {
        expoResponse,
      };

      // If there as no error, change the `_state` to 2.
      if (!expoResponse.errorWhileSending) {
        newNotiData._state = 2;
      }

      // Just add the Ticket info in the DB
      proms.push(
        actionsPushNotifications.editById(notificationsInDB[i].id, newNotiData)
      );
    });

    await Promise.all(proms);

    return res.send({ status: 'ok' });
  } catch (error) {
    return res
      .status(500)
      .send({ status: 'error', message: 'Error sending notifications: ' + error });
  }
};

Issue Analytics

State:
Created a year ago
Reactions:4
Comments:21 (16 by maintainers)

Top GitHub Comments

2reactions

pganstercommented, Sep 23, 2022

We continuously have the same problem with the gateway timeouts and it seems to get worse over the last weeks. I don’t know if Expo even has this on their radar, as the status page doesn’t seem to report any problems with their push notifications broker.

In the last week our hourly job that sends push notifications failed 16 times with a 504 gateway timeout and just yesterday it has failed seven times.

All in all, the push notification service by Expo is a great concept, but it is now time that Expo offers a paid service with an uptime guarantee.

1reaction

BrodaNoelcommented, Oct 8, 2022

Well… This is causing it… 6 hours ago I got the same.