unexpected/unparsable response

When calling FirebaseMessaging.getInstance().send(message), sometimes it failes with strange exception. Sometimes no errors for a day, sometime hundreds or thousands errors. Average error rate was about 3% mid december but is much higher today.

i checked Firebase Cloud Messaging Api in Google Cloud Platform console and see increasing 404 answer for Api Method google.firebase.fcm.v1.FcmService.SendMessage

I tried to understand the code and i think the http answer is not as expected. Probably this is related to the firebase server infrastructure, but sdk should not fail on parsing the error and answer some server Exception instead?.

java.lang.IllegalArgumentException: key details at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parse(JsonParser.java:360) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parse(JsonParser.java:318) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseAndClose(JsonParser.java:162) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseAndClose(JsonParser.java:144) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.firebase.messaging.FirebaseMessagingClientImpl$MessagingErrorHandler.safeParse(FirebaseMessagingClientImpl.java:308) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessagingClientImpl$MessagingErrorHandler.createException(FirebaseMessagingClientImpl.java:292) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessagingClientImpl$MessagingErrorHandler.createException(FirebaseMessagingClientImpl.java:282) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.internal.AbstractHttpErrorHandler.handleHttpResponseException(AbstractHttpErrorHandler.java:57) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.internal.ErrorHandlingHttpClient.send(ErrorHandlingHttpClient.java:108) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.internal.ErrorHandlingHttpClient.sendAndParse(ErrorHandlingHttpClient.java:72) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessagingClientImpl.sendSingleRequest(FirebaseMessagingClientImpl.java:127) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessagingClientImpl.send(FirebaseMessagingClientImpl.java:113) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessaging$1.execute(FirebaseMessaging.java:135) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessaging$1.execute(FirebaseMessaging.java:132) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.internal.CallableOperation.call(CallableOperation.java:36) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessaging.send(FirebaseMessaging.java:102) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessaging.send(FirebaseMessaging.java:86) ~[firebase-admin-7.1.0.jar:?] at de.k2interactive.qeep.api.firebase.FirebaseAdminApi.sendMessage(FirebaseAdminApi.java:70) ~[classes/:?] at de.k2interactive.qeep.api.firebase.FirebaseSinkV1Impl.sendSingleMessageInt(FirebaseSinkV1Impl.java:130) ~[classes/:?] at de.k2interactive.qeep.api.notification.NotificationSink.pushFcmReactivation(NotificationSink.java:356) ~[classes/:?] at de.k2interactive.qeep.jobs.FcmReactivationJob.lambda$sendAll$0(FcmReactivationJob.java:89) ~[qeep-jobs-55.jar:?] at java.lang.Thread.run(Unknown Source) [?:?] Caused by: java.lang.IllegalArgumentException: key details, field private java.util.Map com.google.firebase.messaging.internal.MessagingServiceErrorResponse.error at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parse(JsonParser.java:451) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:787) ~[google-http-client-1.38.0.jar:1.38.0] ... 22 more Caused by: java.lang.IllegalArgumentException: key details, field private java.util.Map com.google.firebase.messaging.internal.MessagingServiceErrorResponse.error at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseMap(JsonParser.java:679) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:783) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parse(JsonParser.java:451) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:787) ~[google-http-client-1.38.0.jar:1.38.0] ... 22 more Caused by: java.lang.IllegalArgumentException: expected collection or array type but got class java.lang.Object at com.google.common.base.Preconditions.checkArgument(Preconditions.java:164) ~[guava-29.0-android.jar:?] at com.google.api.client.util.Preconditions.checkArgument(Preconditions.java:67) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:724) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseMap(JsonParser.java:679) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:783) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parse(JsonParser.java:451) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:787) ~[google-http-client-1.38.0.jar:1.38.0] ... 22 more

[qeepcologne]

I retry the request for the last 10 errors with postman by hand. All of them produce the error i posted above. Then i called the backend send method via jconsole for the same fcm token and i got correct FirebaseMessagingException with MessagingErrorCode.UNREGISTERED in ex.getMessagingErrorCode(). In summary: i cannot reproduce the problem on our staging (with the tokens and content from live / we use the same firebase account for live&staging). Number of errors on live is increasing, but that’s logical, because we do not remove the unregistered tokens because of the error above and send more messages to these users.

After more investigation i found out, that error went away after server restart (and coming back after some hours). So i think it is related to http-client and or internal connection pool or connection reuse in the sdk. That is also explains, why i could never reproduce it on staging or with postman.

[roma2341]

We have the same error but with Gmail: Caused by: java.lang.IllegalArgumentException: expected collection or array type but got java.util.List<com.google.api.services.calendar.model.CalendarListEntry> And we can’t load emails until our app restarted. Problem appears ~ every two weeks…