Cache check_ is_ active_ rm

rm = ResourceManager() Spend a lot of time on check_ is_ active_ rm function , could i cache it ？

[dimon222]

@lxorc thanks for this information. Indeed, it sounds plausible to achieve it when cluster page takes long to load. I still have to review available endpoints, but if there’s one that can play the role of active mode health check, it’s good idea to consider replacement. That is also in top of above suggestions (optimization-wise)

[dimon222]

I had some thinking about this question myself few years ago, but here’s why I still haven’t done so:

• Hadoop is considered cluster nature environment. That means that to achieve HA on clientside when running against direct namenodes you need to hit multiple and be able to fallback easily in chances when one or another gets down. If you cache the result, then you risk getting stuck on same broken node for n-time. We could consider short living cache for few minutes, but it might still breach the concept of HA.

Now, enterprise distributions of Hadoop usually include Knox gateway that typically deals with HA concept on its own. If you have single direct Knox url you, however, don’t need to check if cluster is active at all, because it always considered so.

So in my opinion what we could consider:

1. Knox mode - don’t check active RM. Its always same single URL and it should be alive by design
2. Non-knox mode - as is what we have today - default