NullReferenceException on an early EOS

I found this situation where the lexer appears to recognize an EOS before the stream is actually done. This is happening on a Lucene-like query parser we are putting together.

This query runs fine:

(java AND dotnet)

But this one fails

(java AND dotnet )
                ^

Somehow, that space triggers an EOS before reading the ). I’ve seen the EOS while debugging, but the error is an NRE:

ParserTests.EarlyEos.EarlyEosTests.Early_EOS_NRE

System.NullReferenceException: Object reference not set to an instance of an object.

System.NullReferenceException
Object reference not set to an instance of an object.
   at sly.parser.generator.visitor.EBNFSyntaxTreeVisitor`2.Visit(SyntaxNode`1 node, Object context) in csly/sly/parser/generator/visitor/EBNFSyntaxTreeVisitor.cs:line 91
   at sly.parser.generator.visitor.EBNFSyntaxTreeVisitor`2.Visit(ISyntaxNode`1 n, Object context) in csly/sly/parser/generator/visitor/EBNFSyntaxTreeVisitor.cs:line 30
   at sly.parser.generator.visitor.EBNFSyntaxTreeVisitor`2.Visit(SyntaxNode`1 node, Object context) in csly/sly/parser/generator/visitor/EBNFSyntaxTreeVisitor.cs:line 89
   at sly.parser.generator.visitor.EBNFSyntaxTreeVisitor`2.Visit(ISyntaxNode`1 n, Object context) in csly/sly/parser/generator/visitor/EBNFSyntaxTreeVisitor.cs:line 30
   at sly.parser.generator.visitor.EBNFSyntaxTreeVisitor`2.Visit(SyntaxNode`1 node, Object context) in csly/sly/parser/generator/visitor/EBNFSyntaxTreeVisitor.cs:line 89
   at sly.parser.generator.visitor.EBNFSyntaxTreeVisitor`2.Visit(ISyntaxNode`1 n, Object context) in csly/sly/parser/generator/visitor/EBNFSyntaxTreeVisitor.cs:line 30
   at sly.parser.generator.visitor.EBNFSyntaxTreeVisitor`2.Visit(SyntaxNode`1 node, Object context) in csly/sly/parser/generator/visitor/EBNFSyntaxTreeVisitor.cs:line 89
   at sly.parser.generator.visitor.EBNFSyntaxTreeVisitor`2.Visit(ISyntaxNode`1 n, Object context) in csly/sly/parser/generator/visitor/EBNFSyntaxTreeVisitor.cs:line 30
   at sly.parser.generator.visitor.EBNFSyntaxTreeVisitor`2.Visit(SyntaxNode`1 node, Object context) in csly/sly/parser/generator/visitor/EBNFSyntaxTreeVisitor.cs:line 89
   at sly.parser.generator.visitor.EBNFSyntaxTreeVisitor`2.Visit(ISyntaxNode`1 n, Object context) in csly/sly/parser/generator/visitor/EBNFSyntaxTreeVisitor.cs:line 30
   at sly.parser.generator.visitor.EBNFSyntaxTreeVisitor`2.Visit(SyntaxNode`1 node, Object context) in csly/sly/parser/generator/visitor/EBNFSyntaxTreeVisitor.cs:line 89
   at sly.parser.generator.visitor.EBNFSyntaxTreeVisitor`2.Visit(ISyntaxNode`1 n, Object context) in csly/sly/parser/generator/visitor/EBNFSyntaxTreeVisitor.cs:line 30
   at sly.parser.generator.visitor.SyntaxTreeVisitor`2.VisitSyntaxTree(ISyntaxNode`1 root, Object context) in csly/sly/parser/generator/visitor/SyntaxTreeVisitor.cs:line 147
   at sly.parser.Parser`2.ParseWithContext(IList`1 tokens, Object parsingContext, String startingNonTerminal) in csly/sly/parser/parser/Parser.cs:line 103
   at sly.parser.Parser`2.ParseWithContext(String source, Object context, String startingNonTerminal) in csly/sly/parser/parser/Parser.cs:line 85
   at sly.parser.Parser`2.Parse(String source, String startingNonTerminal) in csly/sly/parser/parser/Parser.cs:line 54
   at ParserTests.EarlyEos.EarlyEosTests.Parse(String query) in csly/ParserTests/EarlyEos/EarlyEosTests.cs:line 40
   at ParserTests.EarlyEos.EarlyEosTests.Early_EOS_NRE() in csly/ParserTests/EarlyEos/EarlyEosTests.cs:line 13

Can anyone help me figure out why the lexer is getting that EOS?

I’ve created unit tests to reproduce the issue on my fork/branch: https://github.com/mauricio-morales/csly/tree/issue/223-early-eos

[b3b00]

I released v2.6.4 including fixes for your issues along with your regex lexer improvment PR. Tanks for both

[b3b00]

I found it, First you’ve wanted to manage whitespaces in your grammar which is not an efficient way as it clutters your grammar with useless SPACE tokens (this is the general case when using parser generators). You’d better let CSLY regex lexer skip them during lexing phase. To make CSLY regex lexer skip these tokens you have to mark your SPACE lexeme asskippable :

[Lexeme(@"\s+",isSkippable:true)] // note the optional isSkippable parameter
        SPACE,

Then you can remove all references to SPACE in your grammar :

using System;
using sly.lexer;
using sly.parser.generator;

namespace ParserTests.Issue223_EarlyEos
{
    public class EarlyEosParser
    {
        [Production("primary_expression: VALUE COLON primary_expression")]
        public FieldValueExpression PrimaryExpression(
            Token<EarlyEosToken> fieldNameToken,
            Token<EarlyEosToken> colonToken,
            Expression expression)
        {
            if (Enum.TryParse<Fields>(fieldNameToken.Value, true, out var fieldName))
            {
                return new FieldValueExpression(
                    fieldName,
                    expression
                );
            }

            return new FieldValueExpression(
                null,
                new ValueExpression($"{fieldNameToken.Value}:{expression}")
            );
        }

        [Production("primary_expression: LPAREN  expression  RPAREN")]
        public GroupExpression PrimaryExpression(
            Token<EarlyEosToken> lParenToken,
            Expression expression,            
            Token<EarlyEosToken> rParenToken
            )
        {
            return new GroupExpression(expression);
        }

        [Production("primary_expression: INT")]
        [Production("primary_expression: DOUBLE")]
        [Production("primary_expression: VALUE")]
        [Production("primary_expression: QUOTED_VALUE")]
        public ValueExpression PrimaryExpression(Token<EarlyEosToken> valueToken)
        {
            switch (valueToken.TokenID)
            {
                case EarlyEosToken.INT:
                case EarlyEosToken.DOUBLE:
                case EarlyEosToken.VALUE:
                    return new ValueExpression(valueToken.Value);
                case EarlyEosToken.QUOTED_VALUE:
                    return new ValueExpression(
                        valueToken.Value
                            .TrimStart('\'')
                            .TrimStart('"')
                            .TrimEnd('\'')
                            .TrimEnd('"')
                    );
                default:
                    throw new Exception($"Invalid value token {valueToken.TokenID}");
            }
        }

        [Production("secondary_expression: primary_expression")]
        public Expression SecondaryExpression(Expression valueExpression) => valueExpression;

        [Production("secondary_expression: NOT  secondary_expression")]
        [Production("secondary_expression: PLUS secondary_expression")]
        [Production("secondary_expression: MINUS secondary_expression")]
        public Expression SecondaryExpression(
            Token<EarlyEosToken> unaryOperatorToken,
            Expression unaryExpression)
        {
            var unaryOperator = Enum.Parse<UnaryOperator>(unaryOperatorToken.Value, true);
            return new UnaryExpression(unaryOperator, unaryExpression);
        }

        [Production("tertiary_expression: secondary_expression")]
        public Expression TertiaryExpression(Expression valueExpression) => valueExpression;

        [Production("tertiary_expression: secondary_expression  tertiary_expression")]
        public BinaryExpression TertiaryExpression(
            Expression leftExpression,
            Expression rightExpression)
        {
            return new BinaryExpression(leftExpression, BinaryOperator.AND, rightExpression);
        }

        [Production("tertiary_expression: secondary_expression  AND  tertiary_expression")]
        [Production("tertiary_expression: secondary_expression  OR  tertiary_expression")]
        public BinaryExpression BinaryExpression(
            Expression leftExpression,
            Token<EarlyEosToken> binaryOperatorToken,
            Expression rightExpression)
        {
            var binaryOperator = Enum.Parse<BinaryOperator>(binaryOperatorToken.Value, true);
            return new BinaryExpression(leftExpression, binaryOperator, rightExpression);
        }

        [Production("expression: tertiary_expression")]
        public Expression Expression(Expression valueExpression) => valueExpression;
    }}

Tell me if this answer is good for you