obj.UpdatedRaw ends in Z but obj.Updated.Value.Kind is Local

Environment details

• OS: macOS
• .NET version: 7.0.203
• Package name and version: Google.Cloud.Storage.V1 4.2.0

Steps to reproduce

1. Under debugger, get an object from a Google Storage bucket and
2. Inspect Updated.Value.Kind and UpdatedRaw

Comments

I’m confused about date time handling in the SDK in general. This might be exacerbated by my broader confusion with date times in .NET.

I am using DateTimeOffset for most representations of moments in time. I see that your SDK uses a DateTime? for the updated date on the metadata object.

I assume this is because, “server side” within GCP Storage, you’re stamping objects with a Z-time moment, since it makes little sense to record the server’s particular offset from UTC and could be strange to see for us consumers, and perhaps helps with region synch.

Anyway, this means your wire format is an RFC 3339 string and then there’s a conversion to a DateTime? via a utility. This is interesting in itself, because you eschew the new DateTimeOffset format and I’m not sure why.

Perhaps it’s because a normal DateTime does the trick for UTC, but here’s the confusing part: you don’t specify the Kind as DateTimeKind.Utc but as Local. The string representation ends with Z, so I think it is UTC.

Is this a bug?

[jskeet]

The 1.61.x libraries will now have DateTimeOffset-based properties. Please let us know if you run into any difficulties, but it should all be okay now.

[jskeet]

Okay, so it’s worth being clear about the various issues we have with DateTime:

• There’s conversion at JSON serialization/deserialization time for properties of type DateTime. This may be used by auth code, but I don’t think it’s used by the “date-time” type for Discovery (which is described below). Our Json.NET serializer has a custom converter for converting DateTime to string

• There’s conversion on property access, which happens with the “date-time” type (and “date”) in Discovery docs - which is the case for things like StorageBucket.Updated. For those properties, we generate two properties - the “Raw” property which is populated by Json.NET, and the suffix-less property which uses the “Raw” property. For example:

  /// <summary>The modification time of the bucket in RFC 3339 format.</summary>
  [Newtonsoft.Json.JsonPropertyAttribute("updated")]
  public virtual string UpdatedRaw { get; set; }
  
  /// <summary><seealso cref="System.DateTime"/> representation of <see cref="UpdatedRaw"/>.</summary>
  [Newtonsoft.Json.JsonIgnoreAttribute]
  public virtual System.Nullable<System.DateTime> Updated
  {
      get => Google.Apis.Util.Utilities.GetDateTimeFromString(UpdatedRaw);
      set => UpdatedRaw = Google.Apis.Util.Utilities.GetStringFromDateTime(value);
  }
  

• There’s the handling of the “google-datetime” type (see #2040), along with “google-duration” and “google-fieldmask”. This is not only painful in terms of generating properties of type object, but also because we’re just relying on Json.NET recognizing “hey, this looks like it could be a DateTime” and doing the conversion speculatively.

My top priority is not to break existing code, but adding a warning (via deprecation) is probably acceptable - especially if we can provide a smooth migration path before that. (It will break users who have warnings as errors, of course.)

This issue is focusing on the second of these bullet points, and that’s where I want to focus the work as well - but with an understanding that we should look at what future improvements to the first and last bullet points might include as well; if we can design an eventual solution that is reasonably consistent, that would be great.

The second bullet point covers multiple problems, in fact:

• The conversion to local time, as you’ve noted
• The fact that parsing just uses DateTime.TryParse, which will use the default culture’s default calendar system
• The truncation of any “set” properties to millisecond precision is probably okay in reality, given the Discovery format documentation, but it would be good to document that.

It’s also worth being aware we can only allocate somewhat-limited time to this - the REST-based libraries are “mostly in maintenance mode”. But it’s been something that’s been niggling me for a long time too, so I do want to make progress with any low-hanging fruit.

The fact that we’re keeping the raw string value is really helpful in terms of migration. I’m going to write up an internal design doc with more details, but my current thinking is:

• Introduce a new property of type DateTimeOffset? for each of the resource properties:
  • Naming TBD, but possibly a suffix of “DateTimeOffset”, so we’d have UpdatedDateTimeOffset - which is ugly, but at least it’s unambiguous and unlikely to collide with anything else.
  • This will be like the existing DateTime? property, in terms of just updating the state of the Raw property. Note that it will only ever store UTC. This means if you set the property to a value with an offset of 2 hours from UTC, but then fetch the property again, you won’t get the same value - you’ll get the same instant in time, but in UTC instead of the offset being maintained. While I’d love to use my Noda Time project instead and Instant, that’s probably a step too far… (Note that the value will also be truncated to millisecond precision.)
• Update just the generated documentation for each DateTime? property to recommend the use of the new property instead.
• After that’s been generated for a while (possibly even a year or more, to allow lots of time for migration), generate [Obsolete("Use the other property")] for each DateTime? property; this could potentially break users who have warnings as errors, but I generally regard that as an acceptable kind of breakage within a major version. (Users are effectively opting into being broken more easily, presumably in order to fix things more eagerly.)

One option to consider: we could keep track of properties we’re already generating and for new properties (including any new APIs), only generate the DateTimeOffset? code. We’re already stateful for enum values, so this would just be an expansion to this. If we go down this route, we could skip the suffix on the DateTimeOffset? property… but that would lead to inconsistency.

Feedback on that plan would be welcome. In the meantime, you might want to consider using the “raw” property instead in your consuming code.