Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Improved markup, CSS, JS tokenization

See original GitHub issue

Hi! First, thanks of all for this awesome project! It’s affected so many projects around the world. Incredible work!

Motivation

I’m trying to recreate the default VS Code syntax highlighting with Prism.js (via prism-react-renderer) and I’m running into limitations of Prism.js.

Description

I would like to be able to highlight certain tokens that are not granular enough in HTML, CSS and JS.

Would it be something that the Prism.js team is interested in to have more granular tokenization of HTML, CSS and JS? If possible, also with backwards compatibility?

Some first quick examples of things that could be improved across HTML, CSS and JS (compared against VS Code 1.33.1 default dark mode):

HTML: DOCTYPE

HTML: Treating quotes and equals differently than normal punctuation

CSS: URL values

CSS: Arguments in selectors

CSS: Keywords - ex. generic font family names, colors (maybe in CSS-Extra?)

JS: Regex sequences, regex escaping

The code used:

<!DOCTYPE html>

<html lang="en">
  <head>
    <script>
      window.console && console.log('foo');
      function initHighlight(block, cls) {
        try {
          if (cls.search(/\bno\-highlight\b/) != -1)
            return process(block, true, 0x0f) + ` class="${cls}"`;
        } catch (e) {
          /* handle exception */
        }
        for (var i = 0 / 2; i < classes.length; i++) {
          if (checkCondition(classes[i]) === undefined)
            console.log('undefined');
        }
      }
    </script>
    <title>Prism</title>
    <style>
      @font-face {
        src: url(https://lea.verou.me/logo.otf);
        font-family: 'LeaVerou';
      }

      #features li:nth-child(odd),
      footer p {
        font: 100% Consolas, 'Andale Mono', monospace;
        background-image: linear-gradient(
          45deg,
          transparent 34%,
          white 34%,
          white 66%,
          transparent 66%
        );
        text-shadow: 0 1px white;
      }
    </style>
  </head>
</html>

Issue Analytics

State:
Created 4 years ago
Comments:10 (10 by maintainers)

Top GitHub Comments

1reaction

RunDevelopmentcommented, Apr 28, 2019

Regarding “HTML: Treating quotes and equals differently than normal punctuation”:

Another CSS-based solution is to use the fact that the = character will always be the first token inside a attr-value token. So the following to uniquely identify every = character:

.token.attr-value > .token.punctuation:first-child {
  /* styles */
}

1reaction

RunDevelopmentcommented, Apr 26, 2019

Regarding the quotes:

I have 2 possible solutions: 1. Add a hook which gives the first = a special class. Change the style of attr-value to look the same as strings and overwrite the color for the = using the special class.

Prism.hooks.add('wrap', function (env) {
	if (env.type === 'punctuation' && env.content === '=') {
		env.classes.push('special-punctuation-name');
	}
});

Not tested but should work. This will also affect the = signs of all other languages but that shouldn’t be too much of a problem.

The second solution is to modify the markup language definition:

'attr-value': {
	pattern: /=\s*(?:"[^"]*"|'[^']*'|[^\s'">=]+)/i,
	inside: {
		'punctuation': /^=/,
		'string': {
			pattern: /(\s*)[\s\S]+/,
			lookbehind: true
		}
	}
}

I would careful with modifying the language definition because my other languages rely on markup, so this might cause problems.