Revamp Sparkler Config - Type checking + Validation + *-site.yaml overriding
See original GitHub issueThere are two FIXME: in configuration:
First, support loading sparkler-defaults.yaml
and sparkler-site.yaml
. The common practice is *-default.yaml
provides default and recommended values from developers. The *-site.yaml
should beused by users to customize or to override the defaults. Right now sparkler-default.yaml
is supported and sparkler-site.yaml
is not yet supported.
Second, towards the stability of system and fail early when errors are introduced by humans. Right now YAML configuration is transformed to JSON and internally it is treated as JSON. The system wouldn’t know ahead of time if config is invalid. There is no type check for values, no schema validation etc. This could introduce errors to system and crash in later stages when the input config is invalid. As you see, except the config module, the rest of system is in staticly typed and strictly enforced. I think it will be nice to fail early when the config is invalid.
Issue Analytics
- State:
- Created 7 years ago
- Comments:6 (6 by maintainers)
@thammegowda @karanjeets Let me know if you find things that can be done better. https://github.com/SHASHANK-PRO-05/sparkler.git. I am doing development in “dev” branch. Also, I am creating reusable annotations in utils package. Do check that out also
@SHASHANK-PRO-05 Sounds good to me.
I will be excited to see these changes coming! Send me a link to your repository and your branch, I will check them and provide early feedback as quickly as possible
CC @karanjeets Check this out.