Use Markdown to represent rich text

The Problem

Currently, there are some oddities with the way the API represents rich text fields, such as spell descriptions. The present strategy is to separate paragraphs into their own strings, as items in an array. This is practical for rendering - for each over the array, and produce a <p> tag for each string (or your stack’s equivalent); however, this approach is severely limiting when it comes to complex mark-up, such as tables, subheadings, etc.

Compliance with this approach leads to some interesting formatting in the API’s data. For example, the spell scrying has two tables, in the source:

Whereas the format given in the API is distant from the source, and carries no information about emphasis, where tables begin and end, and treats table rows as distinct paragraphs:

{
  "index": "scrying",
  "name": "Scrying",
  "desc": [
    "You can see and hear a particular creature you choose that is on the same plane of existence as you. The target must make a wisdom saving throw, which is modified by how well you know the target and the sort of physical connection you have to it. If a target knows you're casting this spell, it can fail the saving throw voluntarily if it wants to be observed.",
    "Knowledge & Save Modifier",
    "Secondhand (you have heard of the target) +5",
    "Firsthand (you have met the target) +0",
    "Familiar (you know the target well) -5",
    "Connection & Save Modifier",
    "Likeness or picture -2",
    "Possession or garment -4",
    "Body part, lock of hair, bit of nail, or the like -10",
    "On a successful save, the target isn't affected, and you can't use this spell against it again for 24 hours.",
    "On a failed save, the spell creates an invisible sensor within 10 feet of the target. You can see and hear through the sensor as if you were there. The sensor moves with the target, remaining within 10 feet of it for the duration. A creature that can see invisible objects sees the sensor as a luminous orb about the size of your fist.",
    "Instead of targeting a creature, you can choose a location you have seen before as the target of this spell. When you do, the sensor appears at that location and doesn't move."
  ],
  "...": "..."
}

Proposed Solution: Markdown

If we replace arrays of strings with single strings that follow a markdown standard, we’ll be able to represent entities like tables with far greater ease. We would have to choose a single markdown flavour, for consistency. GitHub Flavoured Markdown (GFM) is a solid choice; it is used widely, we already write our PRs and Issues with it, it is compatible with the majority of markdown parsers across various languages, and encompasses most (if not all) of the mark-up required to faithfully replicate the formatting in the SRD.

Pros:

• We can use tables, headings, etc.
• Markdown is an industry standard & well supported, so data-receivers should be able to process the data easily, and contributors to the repo will likely already have some level of comfort with the syntax.
• Markdown maintains some level of human-readability, even if unrendered.

Cons

• The source may be less readable, as raw newlines aren’t acceptable in JSON strings. This is fairly minor, as markdown can be edited externally, if necessary, and copied in to a JSON string.

[ogregoire]

The issue is that we are making assumptions about how a developer wants to display the data. Including markdown that turns the data into tables assumes the developer wants the table, not just the data. For instance, someone using the api to make a data driven application may want those data in a drop down instead (generally, not specifically for the example provided), now they are left with trying to strip out the markdown.

We are definitely trying to make all the data themselves in structured data, but we can’t yet. But the text is part of the SRD, and we write it down as part of the desc field. So we basically have two things here: the textual representation and the extracted data.

We should definitely keep the textual representation as is, and if we want, we improve its representation. As @fergcb said, we are not speaking about the data itself, we are speaking about the textual representation. So this issue is spot-on and suggest the most appropriate fix. Writing the data in markdown form doesn’t invalidate the data we’ve extracted so far.

To be clear, we should have both the data and a textual representation. We come from the text and are migrating towards the data, piece by piece. Transforming the text into markdown doesn’t invalidate the fact that at some point in the future, we want to have a data form of the text.

[ogregoire]

My take is “don’t reinvent the wheel”.

Markdown is popular and well known and works well in a variety of different contexts.

desc is just a text field. There’s no reason to have it split on lines. If anything, it’s just a faulty design to do so. So there should be if not linebreaks at least \n in it.

If we want to reflect the metadata that is the structure of the description, the text field should contain some markup language. Given the popularity of markdown as well as the readability of it, I strongly suggest using it.

Any attempt to make our own markup language is flawed. @scarabdesign’s idea introduces a new level of complexity both for the user and the writer, and overall makes everything less readable.