Create a setup for multiple pods

The server can be run in a setup where there are multiple pods.

The current line of thought is to give every pod their own store, e.g., one file store per pod. While there might be (limited) memory overhead, it keeps things nicely separated (also good from a security perspective), and leaves the option to have entirely different backend configurations per pod.

This means we will need a switching resource store on top (#59), which then delegates to the pod.

Such a configuration might or might not be fully instantiated by Components.js. We need to keep in mind that, at some point, pods will be added and removed and runtime. If we stick to Components.js, we might need a convenient JSON-LD frame for writing pod configurations. Then every pod is just simply a JSON file on disk, and we have one other JSON-LD file that is an index for them. We’d still need a mechanism to swap them in and out at runtime, but that could be nice.

Alternatively, if not using Components.js for that part (for instance, if we rather have a database table with users), we could proceed as follows:

• have a main Components.js configuration, which instantiates an opaque PodResourceStore
• have one or more template Components.js configurations for pods, which contain variables
• then PodResourceStore calls Components.js n times to instantiate the templates with different values for the variables in the templates

Note that both approaches are actually not mutually exclusive: we will likely need a PodResourceStore anyway that can instantiate pods via Components.js at runtime; the only difference is how the configuration is written (to a JSON-LD file or as a database row). So looks like we have our mechanism right there.

[joachimvh]

Currently looking a bit into this. We could use an Initializer to generate all the ResourceStores once the server starts, which reads the pods from a KeyValueStorage with the keys being the pod names and the values everything that is needed for configuration. The backend of the storage could be a simple JSON file initially. The configuration values could be a path/identifier (to be seen) to the template config + values for the variables in the template. But the value could potentially also be a complete configuration or perhaps even just an URL to the config.

For routing we already have a store for that which mostly works if the regex stays very simple which is the case here. I was thinking of giving it an extra interface Register (or something like that). The Initializer would use a function of that interface to register all the pods on startup with the routing store so it links their pod paths to the correct store. Alternatively we could also give another KeyValueStorage as input parameter with values being the actual ResourceStores. The initializer would then have to fill up that storage.

When creating a new pod the data being sent (to /pods) could then also include the name of the template + some values for the variables (with potential future extensions to sending full configs or links to configs). Some other variables probably would have to be filled in by the PodManager such as the exact file path in case a file backend gets chosen (since an agent can’t know what the internal file system looks like) and certain defaults if no value was given there. The same class as used in the Initializer above could then be used to instantiate the pod and register it to the routing resource store. One additional step would be needed to register it to the JSON storage used by the initializer. After that the pod could be filled up as already happens now.

All this combined sounds quite doable tbh. Not thinking about multithreading as things get a lot more interesting there. Same for destroying and cleaning up pods.

[joachimvh]

Or perhaps the intializer just reads the config file from store, and then call .loadConfig on the PodManager.

Yea I agree, that’s what I meant with the initial KeyValueStorage. It would just read out the values there and then send them somewhere to be instantiated into an actual store and linked to the correct route in the routing store.

I think having a folder with all configs somewhere and automatically letting them instantiate by Components.js without any code interference might make things too complex. Both in storing the configs and linking them to the correct paths in the routing store.