pydeck: Extend the support of JavaScript syntaxes that can be used in pydeck.Layer

Target Use case

Extending the support of JavaScript syntaxes that can be used in pydeck.Layer’s argument will help users simplify their code. Users don’t need to make additional columns that are not essentially related to their analysis but are just required for visualization.

Proposed feature

Support JavaScript syntaxes such as ** and Math object among others so that they can be used in pydeck.Layer’s argument. Let me explain the motivation behind this request.

Suppose we have the following pydeck visualization (based on this example in the documentation). As you can see in the keyword argument get_fill_color="entries % 2 === 1 ? [255, 140, 0] : [0, 140, 255]", strings containing a JavaScript expression can be passed. Modulo, strict equality, and ternary operators are obviously accepted.

import pydeck as pdk
import pandas as pd

SCATTERPLOT_LAYER_DATA = "https://raw.githubusercontent.com/visgl/deck.gl-data/master/website/bart-stations.json"
df = pd.read_json(SCATTERPLOT_LAYER_DATA)

# Use pandas to calculate additional data
df["exits_radius"] = df["exits"].pow(0.5)

# Define a layer to display on a map
layer = pdk.Layer(
    "ScatterplotLayer",
    df,
    radius_scale=6,
    get_position="coordinates",
    get_radius="exits_radius",
    get_fill_color="entries % 2 === 1 ? [255, 140, 0] : [0, 140, 255]",
)

# Set the viewport location
view_state = pdk.ViewState(latitude=37.7749295, longitude=-122.4194155, zoom=10, bearing=0, pitch=0)

# Render
r = pdk.Deck(layers=[layer], initial_view_state=view_state)
r.show()

If the following change is applied in order to calculate the radius on the JavaScript end, it won’t work and claims “Error displaying widget”. Exponentiation operator **, which was introduced in ES2016, results in the same error as well.

@@ -4,16 +4,13 @@
 SCATTERPLOT_LAYER_DATA = "https://raw.githubusercontent.com/visgl/deck.gl-data/master/website/bart-stations.json"
 df = pd.read_json(SCATTERPLOT_LAYER_DATA)

-# Use pandas to calculate additional data
-df["exits_radius"] = df["exits"].pow(0.5)
-
 # Define a layer to display on a map
 layer = pdk.Layer(
     "ScatterplotLayer",
     df,
     radius_scale=6,
     get_position="coordinates",
-    get_radius="exits_radius",
+    get_radius="Math.sqrt(exits)",
     get_fill_color="entries % 2 === 1 ? [255, 140, 0] : [0, 140, 255]",
 )

I’d be happy to have these operator and built-in objects available in this context.

To Do List

• Add label and assign to milestone
• Coding
• Doc update
• What’s new update
• Test

[ibgreen]

I’ll see if there is any room I can contribute to these 3 packages after familiarizing myself with their codebases.

@yudai-nkt that sounds good, adding support for the exponentiation operator should be done upstream.

However, in the mean-time, if you added support for whitelisting the Math functions, you could always call Math.pow().

To help you get started, maybe you could adapt this pseudo-code, add it to parse-expression-string and make it work?

Note that we’d also want to see a few test cases and some doc updates before landing.

const ALLOWED_FUNCTION_MAP = Math;

function checkASTNode(node, parameters, options) {
  switch (node.type) {
    case 'CallExpression':
      // Only top level functions may be called - never on an object
      const functionCallValid = node.callee.type === 'Identifier' && node.callee.name in ALLOWED_FUNCTION_MAP;
      if (!functionCallValid) {
        let message = `Invalid function call`;
        if (node.callee && node.callee.name) {
           message += ` ${node.callee.name}`;
        }
        throw new Error(message);  
      }
      break;

    case 'ThisExpression':
      throw new Error(`Invalid identifier "this"`);

    case 'MemberExpression':
      throw new Error(`Member function calls not permitted`);

    case 'Identifier':
      const isValid = node.name in parameters;
      if (!isValid) {
        failIdentifierValidation(node.name, parameters, options);
      }
      break;

    default:
  }
}

[ibgreen]

@yudai-nkt Agree that expanding the supported function syntax makes a lot of sense.

Function calling is actually supported under the hood, but there were security concerns with letting the remote app invoke arbitrary javascript functions so it was disabled - permitted functions (e.g Math. ...) should be controlled with a whitelist.

The relevant code is in the @deck.gl/json module: https://github.com/visgl/deck.gl/blob/master/modules/json/src/helpers/parse-expression-string.js

Documentation is here: https://deck.gl/docs/api-reference/json/conversion-reference

@ajduberstein