Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Flatten viz.json layers

See original GitHub issue

Background

viz.json files contain information about a visualization created by the CARTO Builder. deep-insight.js/cartodb.js use these files to render beautiful dashboards and maps 😃

In previous versions of the editor (before The Builder was born) there are/were two different flavours of viz.json files:

  1. viz.json for public maps.
  2. viz.json for maps that are password protected or have any private dataset.

Let’s see a couple of examples of viz.json files for the same map (4 layers: 1 basemap, 1 CartoDB layer, 1 torque layer and the layer with labels) with different privacy settings (showing only the part related to layers):

1. viz.json format for public maps:

{
  ...
  "layers": [
    {
      "options": {
        "default": "true",
        "url": "http://{s}.basemaps.cartocdn.com/light_nolabels/{z}/{x}/{y}.png",
        "subdomains": "abcd",
        "minZoom": "0",
        "maxZoom": "18",
        "name": "Positron",
        "className": "positron_rainbow_labels",
        "attribution": "&copy; <a href=\"http://www.openstreetmap.org/copyright\">OpenStreetMap</a> contributors &copy; <a href=\"https://carto.com/attributions\">CARTO</a>",
        "labels": {
          "url": "http://{s}.basemaps.cartocdn.com/light_only_labels/{z}/{x}/{y}.png"
        },
        "urlTemplate": "http://{s}.basemaps.cartocdn.com/light_nolabels/{z}/{x}/{y}.png",
        "type": "Tiled"
      },
      "infowindow": null,
      "tooltip": null,
      "id": "9b9a0f4e-b486-4281-abcf-7fde4e26408d",
      "order": 0,
      "type": "tiled"
    },
    {
      "type": "layergroup",
      "options": {
        "user_name": "pabloalonso",
        "maps_api_template": "https://{user}.carto.com:443",
        "sql_api_template": "https://{user}.carto.com:443",
        "filter": "mapnik",
        "layer_definition": {
          "stat_tag": "67478336-48e4-11e6-ad50-0ecd1babdde5",
          "version": "3.0.0",
          "layers": [
            {
              "id": "7713296c-ccb7-4047-a736-5fb272d9f8ac",
              "type": "CartoDB",
              "infowindow": { ... },
              "tooltip": { ... },
              "legend": null,
              "order": 1,
              "visible": true,
              "options": {
                "layer_name": "airbnb_madrid_oct_2015_listings_with_closest_metro",
                "cartocss": "// points\n [mapnik-geometry-type=point] {\n    marker-fill: #FF6600;\n    marker-opacity: 1;\n    marker-width: 12;\n    marker-line-color: white;\n    marker-line-width: 3;\n    marker-line-opacity: 0.9;\n    marker-placement: point;\n    marker-type: ellipse;marker-allow-overlap: true;\n  }\n\n //lines\n [mapnik-geometry-type=linestring] {\n    line-color: #FF6600; \n    line-width: 2; \n    line-opacity: 0.7;\n  }\n\n //polygons\n [mapnik-geometry-type=polygon] {\n    polygon-fill:#FF6600;\n    polygon-opacity: 0.7;\n    line-opacity:1;\n    line-color: #FFFFFF;\n   }",
                "cartocss_version": "2.1.1",
                "interactivity": "cartodb_id",
                "source": "c1"
              }
            }
          ]
        },
        "attribution": ""
      }
    },
    {
      "id": "71b75edb-8140-4640-a266-4a13bc84d994",
      "type": "torque",
      "order": 2,
      "legend": null,
      "options": {
        "stat_tag": "67478336-48e4-11e6-ad50-0ecd1babdde5",
        "maps_api_template": "https://{user}.carto.com:443",
        "sql_api_template": "https://{user}.carto.com:443",
        "visible": true,
        "table_name": "airbnb_madrid_oct_2015_listings_with_closest_metro"
      },
      "cartocss": "Map {\n  -torque-frame-count: 256;\n  -torque-animation-duration: 30;\n  -torque-time-attribute: \"cartodb_id\";\n  -torque-aggregation-function: \"count(1)\";\n  -torque-resolution: 2;\n  -torque-data-aggregation: linear;\n}#layer {\n  marker-width: 7;\n  marker-fill: #FFB927;\n  marker-fill-opacity: 0.9;\n  marker-allow-overlap: true;\n  marker-line-width: 1.5;\n  marker-line-color: #ffda8d;\n  marker-line-opacity: 1;\n}\n#layer[frame-offset=1] {\nmarker-width: 9;\nmarker-fill-opacity: 0.5;\n}\n#layer[frame-offset=2] {\nmarker-width: 11;\nmarker-fill-opacity: 0.25;\n}",
      "cartocss_version": "2.1.1",
      "sql": "SELECT * FROM airbnb_madrid_oct_2015_listings_with_closest_metro",
      "source": "a0"
    },
    {
      "options": {
        "default": "true",
        "url": "http://{s}.basemaps.cartocdn.com/light_only_labels/{z}/{x}/{y}.png",
        "subdomains": "abcd",
        "minZoom": "0",
        "maxZoom": "18",
        "attribution": "&copy; <a href=\"http://www.openstreetmap.org/copyright\">OpenStreetMap</a> contributors &copy; <a href=\"https://carto.com/attributions\">CARTO</a>",
        "urlTemplate": "http://{s}.basemaps.cartocdn.com/light_only_labels/{z}/{x}/{y}.png",
        "type": "Tiled",
        "name": "Positron Labels"
      },
      "infowindow": null,
      "tooltip": null,
      "id": "e2e7eac2-f9a0-4f3b-b97c-a377c812c3d6",
      "order": 3,
      "type": "tiled"
    }
  ],
  "analyses": [
    {
      "id": "c1",
      "type": "buffer",
      "params": {
        "source": {
          "id": "c0",
          "type": "source",
          "params": {
            "query": "SELECT * FROM airbnb_madrid_oct_2015_listings_with_closest_metro LIMIT 10"
          },
          "options": {
            "table_name": "airbnb_madrid_oct_2015_listings_with_closest_metro"
          }
        },
        "radius": 1000,
        "dissolved": false
      },
      "options": {
        "distance": "kilometers"
      }
    },
    {
      "id": "a0",
      "type": "source",
      "params": {
        "query": "SELECT * FROM twitter_t3chfest_reduced"
      },
      "options": {
        "table_name": "twitter_t3chfest_reduced"
      }
    }
  ],
  ...
}

2. viz.json format for protected maps/maps with private datasets:

{
  ...
  "layers": [
    {
      "options": {
        "default": "true",
        "url": "http://{s}.basemaps.cartocdn.com/light_nolabels/{z}/{x}/{y}.png",
        "subdomains": "abcd",
        "minZoom": "0",
        "maxZoom": "18",
        "name": "Positron",
        "className": "positron_rainbow_labels",
        "attribution": "&copy; <a href=\"http://www.openstreetmap.org/copyright\">OpenStreetMap</a> contributors &copy; <a href=\"https://carto.com/attributions\">CARTO</a>",
        "labels": {
          "url": "http://{s}.basemaps.cartocdn.com/light_only_labels/{z}/{x}/{y}.png"
        },
        "urlTemplate": "http://{s}.basemaps.cartocdn.com/light_nolabels/{z}/{x}/{y}.png",
        "type": "Tiled"
      },
      "infowindow": null,
      "tooltip": null,
      "id": "9b9a0f4e-b486-4281-abcf-7fde4e26408d",
      "order": 0,
      "type": "tiled"
    },
    {
      "type": "namedmap",
      "order": 1,
      "options": {
        "type": "namedmap",
        "user_name": "pabloalonso",
        "maps_api_template": "https://{user}.carto.com:443",
        "sql_api_template": "https://{user}.carto.com:443",
        "filter": "mapnik",
        "named_map": {
          "name": "tpl_67478336_48e4_11e6_ad50_0ecd1babdde5",
          "stat_tag": "67478336-48e4-11e6-ad50-0ecd1babdde5",
          "params": {
            "layer0": 1
          },
          "layers": [
            {
              "id": "7713296c-ccb7-4047-a736-5fb272d9f8ac",
              "layer_name": "airbnb_madrid_oct_2015_listings_with_closest_metro",
              "interactivity": "cartodb_id",
              "visible": true,
              "options": {
                "source": "c1"
              },
              "infowindow": { ... },
              "tooltip": { ... }
            }
          ]
        },
        "attribution": ""
      }
    },
    {
      "id": "71b75edb-8140-4640-a266-4a13bc84d994",
      "type": "torque",
      "order": 2,
      "legend": null,
      "options": {
        "stat_tag": "67478336-48e4-11e6-ad50-0ecd1babdde5",
        "maps_api_template": "https://{user}.carto.com:443",
        "sql_api_template": "https://{user}.carto.com:443",
        "visible": true,
        "table_name": "airbnb_madrid_oct_2015_listings_with_closest_metro",
        "named_map": {
          "name": "tpl_67478336_48e4_11e6_ad50_0ecd1babdde5",
          "layer_index": 1,
          "params": {
            "layer0": 1
          }
        }
      },
      "cartocss": "Map {\n  -torque-frame-count: 256;\n  -torque-animation-duration: 30;\n  -torque-time-attribute: \"cartodb_id\";\n  -torque-aggregation-function: \"count(1)\";\n  -torque-resolution: 2;\n  -torque-data-aggregation: linear;\n}#layer {\n  marker-width: 7;\n  marker-fill: #FFB927;\n  marker-fill-opacity: 0.9;\n  marker-allow-overlap: true;\n  marker-line-width: 1.5;\n  marker-line-color: #ffda8d;\n  marker-line-opacity: 1;\n}\n#layer[frame-offset=1] {\nmarker-width: 9;\nmarker-fill-opacity: 0.5;\n}\n#layer[frame-offset=2] {\nmarker-width: 11;\nmarker-fill-opacity: 0.25;\n}",
      "cartocss_version": "2.1.1",
      "sql": "SELECT * FROM airbnb_madrid_oct_2015_listings_with_closest_metro",
      "source": "a0"
    },
    {
      "options": {
        "default": "true",
        "url": "http://{s}.basemaps.cartocdn.com/light_only_labels/{z}/{x}/{y}.png",
        "subdomains": "abcd",
        "minZoom": "0",
        "maxZoom": "18",
        "attribution": "&copy; <a href=\"http://www.openstreetmap.org/copyright\">OpenStreetMap</a> contributors &copy; <a href=\"https://carto.com/attributions\">CARTO</a>",
        "urlTemplate": "http://{s}.basemaps.cartocdn.com/light_only_labels/{z}/{x}/{y}.png",
        "type": "Tiled",
        "name": "Positron Labels"
      },
      "infowindow": null,
      "tooltip": null,
      "id": "e2e7eac2-f9a0-4f3b-b97c-a377c812c3d6",
      "order": 3,
      "type": "tiled"
    }
  ],
  "analyses": [
    {
      "id": "c1",
      "type": "buffer",
      "params": {
        "source": {
          "id": "c0",
          "type": "source",
          "options": {
            "table_name": "airbnb_madrid_oct_2015_listings_with_closest_metro"
          }
        },
        "radius": 1000,
        "dissolved": false
      },
      "options": {
        "distance": "kilometers"
      }
    },
    {
      "id": "a0",
      "type": "source",
      "options": {
        "table_name": "twitter_t3chfest_reduced"
      }
    }
  ],

  ...

}

The information that was used to instantiate the map in both cases, was extracted to a new section in the viz.json, so it can be safely removed from the layergroup/namedmap layer itself:

"datasource": {
  "user_name": "chuck",
  "maps_api_template": "https://{user}.carto.com:443",
  "stat_tag": "67478336-48e4-11e6-ad50-0ecd1babdde5",
  "template_name": "tpl_67478336_48e4_11e6_ad50_0ecd1babdde5"
},

CartoDB.js looks for datasource.template_name to determine if the map should be instantiated via an anonymous or named map.

In 1. CartoDB layers are grouped into a layergroup layer. As we can see, each (sub) layer within this layer has a CartoDB type and exposes some information that is somehow related to the schema of the associated dataset (eg: sql, cartocss, infowindow and tooltip field names, etc). eg:

{
  "id": "7713296c-ccb7-4047-a736-5fb272d9f8ac",
  "type": "CartoDB",
  "infowindow": {
    "template_name": "none", // TODO: This could be removed from viz.json
    "fields": [],
    "maxHeight": 180,
    "template": "",
    "alternative_names": {},
    "width": 226,
    "headerColor": { // TODO: This could be removed from viz.json
      "color": {
        "fixed": "#35AAE5",
        "opacity": 1
      }
    }
  },
  "tooltip": {
    "fields": [],
    "template_name": "none",
    "template": "",
    "template_type": "mustache"
  },
  "legend": null,
  "order": 1,
  "visible": true,
  "options": {
    "layer_name": "airbnb_madrid_oct_2015_listings_with_closest_metro",
    "cartocss": "...",
    "cartocss_version": "2.1.1",
    "interactivity": "cartodb_id",
    "source": "c1"
  }
}

Also, in 1. some analyses items include a query param with the SQL for that node.

In 2. There’s a namedmap layer whose (sub)layers don’t have a type and don’t include any SQL or CartoCSS options. eg:

{
  "id": "7713296c-ccb7-4047-a736-5fb272d9f8ac",
  "layer_name": "airbnb_madrid_oct_2015_listings_with_closest_metro",
  "interactivity": "cartodb_id",
  "visible": true,
  "options": {
    "source": "c1"
  },
  "infowindow": {
    "template_name": "infowindow_light",
    "fields": [
      {
        "name": "require_guest_phone_verification",
        "title": true,
        "position": null
      },
      {
        "name": "require_guest_profile_picture",
        "title": true,
        "position": null
      },
      {
        "name": "instant_bookable",
        "title": true,
        "position": null
      }
    ],
    "maxHeight": 180,
    "template": "<div class=\"CDB-infowindow CDB-infowindow--light js-infowindow\">...</div>\n",
    "alternative_names": {},
    "width": 226,
    "headerColor": {
      "color": {
        "fixed": "#35AAE5",
        "opacity": 1
      }
    },
    "template_type": "mustache"
  },
  "tooltip": {
    "fields": [
      {
        "name": "require_guest_phone_verification",
        "title": true,
        "position": null
      },
      {
        "name": "require_guest_profile_picture",
        "title": true,
        "position": null
      },
      {
        "name": "instant_bookable",
        "title": true,
        "position": null
      }
    ],
    "template_name": "tooltip_dark",
    "template": "<div class=\"CDB-Tooltip CDB-Tooltip--isDark\">...</div>\n",
    "template_type": "mustache"
  }
}

Queries are removed from analyses in the viz.json in 2.

Goals / scope

The goal of this issue is to remove the concept of layergroup and namedmap layers from the viz.json. Instead of grouping CartoDB layers into one, all layers should be rendered at the same level.

Proposed changes

There’ll still be two different viz.json “flavours”:

  1. The one used by the builder, where data (layers and analyses) is unfiltered.
  2. The one used by embeds/public maps, where data (layers and analyses) is “filtered” (no sql, cartocss).

Layers that lived inside of layergroup and namedmap layers, will now be “first class citizens”, and have the following format:

{
  "id": "7713296c-ccb7-4047-a736-5fb272d9f8ac",
  "type": "CartoDB",
  "visible": true,
  "options": {
    "source": "c1", // 
    "sql": "SELECT foo FROM bar;", // (*)
    "cartocss": ... // (*)
    "cartocss_version": "2.1.1" (*)
    "layer_name": "something",
    "attribution": "© Some custom attribution"
  },
  "infowindow": { ... },
  "tooltip": { ... }
}

(*) Only present if type of viz.json is unfiltered…

Notice that each “CartoDB” layer should include any custom attribution present in the dataset.

cc: @javisantana @rochoa @xavijam @CartoDB/frontend @CartoDB/backend. I’d love your input/feedback/ideas about this! Thx

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Comments:13 (13 by maintainers)

github_iconTop GitHub Comments

1reaction
alonsogarciapablocommented, Jul 26, 2016

I’d say yes cause they are not being used by CartoDB.js v4. .name has been moved to the datasource section.

1reaction
matallocommented, Jul 22, 2016

I’m sorry I won’t be of much help but probably more hassle. I can tell as a newcomer what are the main things it’s hard for me to get (but probably that’s an offtopic, for another PR)

why do we have tooltip, and infowindow at a higher level, and why not inside the options? why do we have those options in the first place? why there’s a kind/type you refer earlier (we deal with this differently in many places of the codes, with transformations and such)

sorry if I should rise these concerns elsewhere, but I’m hitting them again and again, first with the infowindows, now with the layers (as layer in the map, and user layer).

Read more comments on GitHub >

github_iconTop Results From Across the Web

Support viz.json files with flat layers · Issue #829 · CartoDB/carto.js ...
New viz.json format is implemented in CartoDB/cartodb#9118. cc: @rochoa @javisantana. ... Support viz.json files with flat layers #829.
Read more >
Flatten objects in JSON documents | Apache Pinot Recipes
How to flatten objects in JSON documents. In this recipe we'll learn how to flatten nested objects when ingesting JSON documents into Apache...
Read more >
Generically Flatten Json using c# - Stack Overflow
Let's say you had a generic way to transform the above JSON hierarchy to a data table. What would be the columns and...
Read more >
Flattening JSON objects in Python - Towards Data Science
Pandas provides a nice utility function json_normalize for flattening semi-structured JSON objects. The array was not flattened.
Read more >
Flatten Transform — Altair 4.2.0 documentation
The flatten transform can be used to extract the contents of arrays from data entries. This will not generally be useful for well-structured...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Delete Animated option and add a new Aggregation method

Next page

Write explanations for intact and dissolved