Clarification on tags justification, e.g. AWS cloudformation's shorthand private tags

First off, I really like this library, and the design choices you’ve made, so thanks!

I was looking at the removed features, and it lists explicit tags as being a form of syntax typing, which is absolutely bad when it’s defined by the schema, yes! But tags don’t have to be used that way, they can be used as a reserved syntax for alternate ways to provide a value, in particular AWS Cloudformation uses them as a short-hand for their “function” syntax:

Without tag shorthand (or flow):

Parameters:
  HostedZoneName: ...
  RecordName: ...
  RecordComment: ...
  ...

Resources:
  LoadBalancer: ...

  RecordSet:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneName:
        Fn::Sub: '${HostedZoneName}.'
      Comment:
        Ref: RecordComment
      Name:
        Fn::Sub: '${RecordName}.${HostedZoneName}.'
      Type: A
      AliasTarget:
        DNSName:
          Fn::GetAtt:
          - LoadBalancer
          - DNSName
        HostedZoneId:
          Fn::GetAtt:
          - LoadBalancer
          - CanonicalHostedZoneNameID

With tag shorthands:

  RecordSet:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneName: !Sub '${HostedZoneName}.'
      Comment: !Ref RecordComment
      Name: !Sub '${RecordName}.${HostedZoneName}.'
      Type: A
      AliasTarget:
        DNSName: !GetAtt LoadBalancer.DNSName
        HostedZoneId: !GetAtt LoadBalancer.CanonicalHostedZoneNameID

Embedding a sub-syntax in strings like suggested in #20 here is a bad idea, as the transformation is generic across the whole document (even if there are places where it’s not valid), and there are plenty of values (like Comment) that permit arbitrary values; so I think AWS has the right long-hand syntax, but as you can see it quickly gets unwieldy, so the private tags are very heavily used. In this sense, tags are used as an already reserved syntax that can safely escape an embedded string syntax (rather than adding another level of escaping).

As another example that is closer to the original justification for removal, you can also (and it is probably the original intention of tags) use tags to provide types better syntax and lower (user) implementation cost where it’s not directly providable by the schema, for example:

shape:
  fill: !LinearGradient
    from: 0 10
    to: 30 10
    stops: red 0.0, blue 0.2, green 1.0
  path:
  - !Move 0 0
  - !Arc 20 0 20 20
  - !Line 20 0

That said, I’m perfectly OK with strictyaml not supporting tags for implementation or compatibility complexity, or other such reasons, but the justifications only talk about using them for syntax typing, and then only for yaml built-in types, which is insufficient for me to remove this feature on its own.

To be clear, just updating the docs would be fine, though I won’t refuse adding tags support 😇

[witten]

Piggybacking on this issue, although perhaps I should open a new one… Here’s another potentially valid use of tags: YAML file includes. Here’s an actual example from the wild:

retention:
    !include /etc/borgmatic/common_retention.yaml

The idea is that a common YAML fragment gets dynamically included into the YAML document in question at runtime. The main rationale is reuse, so as to avoid having to repeat common configuration in multiple documents. (Usable by non-programmers? Perhaps, although it is admittedly a little advanced.)

But wait, I hear you say, can’t you do that outside of strictyaml, and then still use strictyaml for the other aspects of YAML parsing and validation? You can, but not without problems. Two alternative approaches I can think of:

1. Prior to feeding the YAML document to strictyaml, pre-process it (e.g. with raw ruamel.yaml) to inline all includes and produce a single document. This works, but then any line numbers in strictyaml error messages are completely bogus in relation to the source YAML files.
2. Or, escape the include tags (as suggested by a comment above), and then after strictyaml parses and validates the YAML document, post-process the include directives at the application level. This doesn’t really work well though, because then you give up strictyaml schema validation on any part of the YAML document that’s pulled in by an include. And in fact, schema validation may simply not work if a required part of the document is hidden behind an include tag that strictyaml doesn’t understand.

To be clear, I’m not making a feature request here for file include functionality in strictyaml (although that’d be pretty great). Rather, I’m making the case that support for custom tags in strictyaml would be pretty darn useful — even necessary for some use cases.

[witten]

It just seems like a lot of work — three passes, and fair amount of complexity to make the schemas separable at runtime — to do something that could in theory be done in one pass. But I’ll give it a shot anyway. 😃

For comparison, my current non-strictyaml code does one pass to load, and then does validation on the resulting data structure in memory. (I do realize that performance is not one of strictyaml’s main goals.)