Calling all contributors - Issues and changes for gmail.js going forward

Hey everyone.

I’m seeing some troubles up ahead. Action is required.

Executive summary

There’s issues ahead and I don’t foresee any way to resolve them in a fully backwards compatible way.

I have a proposal for how we go about dealing with these issues, but they mean taking gmail.js in a slightly new direction and I’d like some input on those suggestions.

Context - the new Gmail, the new IDs

Google recently unveiled their “new Gmail” which was more than a GUI-change, it also includes a new data-layer and XHR-format.

What seemed obvious from the first second of testing gmail.js on the new gmail was that this was a fundamentally breaking change.

Our strategy so far has been to try to patch up the things we see which are reported broken, and try to keep the existing gmail.js API-surface unchanged.

In the process of introducing a new data-later, Google has also introduced some new ID-formats: One for threads and one for emails (whereas these used to be the same ID and same format for both in “old” Gmail).

We’ve tried to the best of our efforts to make gmail.js not just a library, but also a forward compatibility-shim which shields the extension writer from having to deal with “old” or “new” Gmail. If that can be done, that’s absolutely marvellous and something we should strive for.

But I’m starting to have my doubts… But why?

A reality check: the current gmail.js codebase will stop working early 2019

The current gmail.js codebase is 100% tied to legacy IDs, using the old Gmail XHR endpoints.

In some internal/classified documentation provided to various extension-writers, Google has informed us that these API endpoints (and thus any support for legacy IDs) will stop working in “early 2019”.

That means that if we keep the codebase as is and do nothing, all gmail.js-based code may start failing 4 months from now.

Ladies and gentlemen: The clock is ticking!

The XHR problem

The biggest and most obvious problem we have right now is that: We’re still having issues trying to perform XHR-requests to obtain Gmail-data in new Gmail.

If we cannot use XHR to obtain this data, we will have to resort to fallback methods (either DOM or OAuth).

This particularly impacts api.get.email_data() (and all derived/dependant functions).

While there’s lots of the info provided by that function can now be provided reasonably reliable through the DOM (which only works if the email is visible), there’s absolutely no way we can make this function return the same data, in the same format, with the same granularity and same guarantees once support for legacy IDs is dropped.

So if we can’t “fix” that function in a compatible way… We’ll have to make a new function whose return-data can be guaranteed, even after legacy IDs (and possible XHR requests) are shut down.

At first I thought about just adding a new api.dom.email_data() function and calling it a day…

But then I started thinking about it a bit more deeply, and also considered our other big problem.

The function/ID ambiguity problem

We have lots of functions with names such as “email_data”. We have lots of functions accepting a simple “ID”-parameter.

It’s very ambiguous what these functions except and provide. Will they provide email-data? Does this function required a thread-ID or an email-ID?

In the past we didn’t have to make that distinction. But now we must.

Now we must be explicit about “I want this info about this thread” and “I want that info about this email”. And then I think it makes sense that our functions are explicit about that too: both in naming and return-types.

But making these changes to a published API is not possible in a compatible manner.

And finally: The reliability problem

Right now, and especially after 2019 enters, as an extension-developer, I can use gmail.js in my extension, and at runtime all my API calls will look good. With Typescript I can even verify that I’m using the published API 100% correctly at compile-time!

But while the “build” is good, while the extension initialization is good… There’s actually no way for me to know if this code will actually work on new Gmail or not without testing.

With current Gmail.js there is no way to make a guaranteed-to-work extension for new Gmail.

You have to use the API as provided, hope it works and invest heavily in manual testing. Is that really considered acceptable in 2018?

Basically the API is telling extension-developers “yes, this will be fine”, when it may in fact NOT be so. The API is lying.

Pulling it all together: the proposal

So we know now that:

• We’re living on borrowed time. Action required.
• We may lose some functionality (specifically anything XHR-related), and we there is a chance we won’t be able to make up for it.
• Keeping the current API-surface backwards compatible is not realistic. Trying to do so, will at best lie to extension-authors about the what they can expect returned and when.

In response to that, my proposal is pretty simple:

While we keep the old Gmail-class as is (no need to create havoc!), going forward we create a new GmailJs-class (Gmail2?), a new API, in which we only expose functionality we know works with new Gmail.

That is, if you as an extension author use Gmail2 and try to use functionality not supported by new Gmail, your code will either not compile (TypeScript) or cause runtime failures because the methods are not there.

Basically, the new API should not lie.

For the time being this class can simply wrap up call to the existing Gmail-class we have, but since this is new code we’re writing, we can use this occasion to sort out other issues too:

• The thread/email ambiguity problem. We can now create explicit functions for threads and emails. We can enforce correct ID-format without breaking existing code.
• We can modernize the codebase: Upgrade to ES2015 for all new code. This will give us proper classes, and a structure tooling can understand and support.

Other things which are tempting to at the same time (and really, has to be done at the same time since it affects the API surface):

• Replace callbacks with promises where it makes sense.
• Ditch the jQuery-dependency. It’s 2018 and queryMediaSelectorAll works really nice. (And with TypeScript, handling DOM-objects is actually better supported than jQuery).
• Possibly base the new class (classes?) on Typescript, unless there’s any major objections.
• Create new, smaller classes instead of a new huge monolith.

Phew! That was a lot of stuff

Yeah it was. And I’ll avoid going into more details for now, but that’s roughly it.

What I’m interested in knowing is what you guys think about this proposal. Does it seem reasonable? Or are there things you object to?

cc: All major and recent contributors. @KartikTalwar, @shashikiran-im, @mheinzler , @hankypanky666, @phyxolog, @SuperPaintman, @brentkelly, @madhugb, @Navega, @michi88, @wearhere, @GeneralZero, @avzabr, @neomarbassani, @Capo93

[josteink]

A final FYI, I’ve now messed around with the new code in a production-extension and found it somewhat unergonomic, and needlessly hard to port in places.

As a result I’ve made some breaking changes w.r.t. the shape of data returned from api.new.get.email_data() to make migration easier.

These changes can be found in the latest official package (0.9.0), and at this point I’m going to consider the API stable unless someone objects .

If you’re unsure if code you wrote for 0.8.0 still works, I highly advise using TypeScript as this should catch all breaking errors.

[josteink]

And merged.

The main readme/docs needs to be updated, but for those who wants to give this a try without fetching git feature-branches, this is now published as version 0.8.0.

Closing issue as main problem seems to be solved.

To everyone who contributed and provided feedback: Thanks. It’s appreciated!