question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Invalid binary aggregation optimization regression

See original GitHub issue

After upgrading from 2.6(.4) to 2.7(.2), a predicate interpretation/optimization problem occours when condition is in conjunctive normal form, and extra condition added in nested reverse form.

We use boolean constants to control how the predicate would behave under certain data driven scenarios. Constants are translated into SQL “1 = 1” or “1 = 0” accordingly, which is fine. The problem occours for example in cases like:

var isEmptyX = true, isEmptyY = false;
var includeZ = true, includeW = true;

(isEmptyX || x.Prop1 == [...]) && (isEmptyY || x.Prop2 == [...]) && ((includeZ && x.Field == EnumVal1) || (includeW && x.Field == EnumVal2))

The last nested past of this expression in 2.7.2 will translate into ...AND (1 == 1 AND 1 == 0 OR 1 == 1 AND Field == EnumVal2). If I switch the EnumVal1 equality check with the part EnumVal2 to be the last in the predicate, then it will generate ... AND Field == Enumval1 accordingly. Inspecting the Expression on query object it seems that parentheses dropped internally as well. This seems to be some kind of too optimistic optimization that replaces the other field access expression with constant boolean for no reason.

Setting used: Configuration.Linq.UseBinaryAggregateExpression = true

(this setting is a must for use, because some sub-system generates huge (1000+) parameter-based predicates that would cause stack overflow otherwise)

Thanks in advance.

Environment details

linq2db version: 2.7.2 Database Server: SQL Server 2017 Database Provider: SqlClient Operating system: Window 10 1804 .NET Framework: 4.7.2

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
sdanylivcommented, Jun 6, 2019

@detoxhby, thanks. Wait for quick fix and bug-fix release.

0reactions
detoxhbycommented, Jun 6, 2019

It generates LinqService pair tests, this could be excluded (but generates good query though…).

using NUnit.Framework;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using LinqToDB;
using LinqToDB.Common;
using Tests.Model;

namespace Tests.UserTests
{
	[TestFixture]
	public class Issue1750Tests : TestBase
	{
		[Test]
		public void Issue1750Test([DataSources(ProviderName.SqlServer2017)] string context)
		{
			var ubae = Configuration.Linq.UseBinaryAggregateExpression;

			try
			{
				Configuration.Linq.UseBinaryAggregateExpression = true;

				using (var db = GetDataContext(context))
				{
					// Arrange
					var childrenIn = (int[])null;
					var grandChildIn = GrandChild.Select(x => x.ParentID);					
					var includeX = true;
					var includeY = true;

					// Act
					var query = from p in db.Parent
								where
								(childrenIn == null || childrenIn.Contains(p.ParentID)) && (grandChildIn == null || grandChildIn.Contains(p.ParentID)) &&
								// this should be "(1 = 1 AND Value1 = 1) OR (1 = 1 AND Value1 = 2)"
								// or with proper normalization it should be "(Value1 = 1 OR Value1 = 2)"
								((includeX && p.Value1 == 1) || (includeY && p.Value1 == 2))
								select p;

					var querySQLText = query.ToString();

					// Assert
					Assert.False(
						querySQLText.TrimEnd().EndsWith("(1 = 1 AND [p].[Value1] = 2)", StringComparison.OrdinalIgnoreCase),
						$"Invalid predicate normalization: {querySQLText}"
					);
				}
			}
			finally
			{
				Configuration.Linq.UseBinaryAggregateExpression = ubae;
			}
		}
	}
}
Read more comments on GitHub >

github_iconTop Results From Across the Web

Regression — Why Mean Square Error?
In this article I'll be going through some basic Loss Functions used in Regression Algorithms and why exactly are they that way. Let's...
Read more >
PROC GENMOD in SAS Invalid Reference Value for ...
I'm trying to model a binary outcome ( p1ODD ) on binary predictor variables ( c1kdscc3 , c1kdscc4 and c1kdscc5 ). When I...
Read more >
Avoiding Invalid Instruments and Coping with Weak ...
discusses classic strategies for avoiding invalid instruments (instruments themselves correlated with the regression's disturbances), and describes recently ...
Read more >
Optimization metrics: DataRobot docs
Provides a complete reference to the optimization metrics DataRobot employs during the modeling process. ... Regression, binary classification.
Read more >
4. Regression and Prediction - Practical Statistics for Data ...
Regression and Prediction Perhaps the most common goal in statistics is to answer ... Analysis of data in an aggregated form such that...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found