Searching with scope "sub" does not include baseDN on Samba 4.7.6

Hi, we’ve found an issue (on current branch next) with subtree-scoped searches against Samba 4.7.6 LDAP not returning the base DN when they should.

Assuming baseDN = 'CN=Users,DC=ad,DC=example,DC=org' and a DIT with only the Users container (no objects underneath it):

• ldapsearch -b "cn=Users,dc=ad,dc=example,dc=org" -s sub '(objectClass=container)' ✔️ works - returns exactly 1 entry (the Users container)
• client.search(baseDN, { scope: 'base', filter: new EqualityFilter({ attribute: 'objectClass', value: 'container' }) }) ✔️ works - returns exactly 1 entry
• client.search(baseDN, { scope: 'sub', filter: new EqualityFilter({ attribute: 'objectClass', value: 'container' }) }) ❌ does not work - returns exactly 0 entries - this should be impossible!

What is more concerning is that it seems that it behaves differently on Samba 4.11 (not sure why, but probably because of some opaque internal LDB handling changes).

We initially thought it was caused by #602, but even after fixing the race condition and using our fork of the library, the issue remained. The queries looked legit, but we decided to compare the generated BER on the wire with the actual messages sent by ldapsearch.

There’s only 1 difference: whitespace in DNs:

• ldapsearch sends the DNs as-is, without processing: CN=Users,DC=ad,DC=example,DC=org
• ldapjs re-formats DNs like this: cn=Users, dc=ad, dc=example, dc=org - with spaces as separators!

What’s strange is that RFC 4514 (String Representation of Distinguished Names) does not seem to imply spaces are valid RDN separators. In fact, the only separator it defines is , (comma, or U+002C).

Why it works with scope: 'base' is beyond me, but when we tested ldapsearch with a re-formatted baseDN with spaces inserted with scope sub, we got the same results as with ldapjs (0 results). This means the spaces really are extraneous.

This looks related to #151 and the defaults it established. I’m really surprised it hasn’t caused a bigger number of issues till now.

I wonder about @mcavage’s comments in #80 on “old LDAP clients” - this implies that “new LDAP clients” (what about servers?) should handle spaces in DNs gracefully, right? It seems non-obvious to me since I can’t find an explanation for this special handling of spaces in the RFCs, though human users could of course imply that whitespaces are insignificant (which they are apparently not, to this Samba instance).

For now, I’ll get a fix ready that changes the DN serialization in SearchRequest messages to not use spaces. More generally, it seems like a long-standing mistake to add spaces by default if RFC 4514 does not define them as separators. At this point, a change to the default value produced by DN.toString() will likely break people’s systems, though, so this “precision” fix that only targets SearchRequest seems like a more viable option.

[rkaw92]

Aha! I may have found the root cause for the confusion around spaces and quotes:

• RFC 1779, released in 1995, allows <spaced-separator>, which, in their BNF, is optional space, a comma or a semicolon, and another optional space
• RFC 2253, from Netscape in 1997, says only the canonical form (comma, no spaces) must be produced, but the old form from LDAPv2 must be understood by implementations for backwards compatibility. Also, it allows quoting string values.
• RFC 4514, from 2006, only mentions commas, no spaces around them. Also, no mention of quotes quoting as a means of escaping special chars is to be seen anywhere.

Now, the LDAP RFC (4511, “The Protocol”, which obsoletes old LDAP standards) only mentions RFC 4514 as the format of DNs to use on the wire.

This means:

• LDAPv3-compliant servers should understand both the “original” LDAPv3 syntax with quoting and the new RFC
• If we aim to fully adopt the “new” RFC (which 14 years old, so not exactly bleeding-edge), we should fix the package to always generate DNs without quoting and according to the up-to-date ABNF which is, thankfully, simpler than the old formats

For now, I’ll just do a targeted fix that addresses the issue at hand (no spaces between commas in SearchRequest) and preserves the previous behaviour w.r.t. quotes/special chars, but it looks like we should review DN handling in general: spacing, quoting and escaping of special characters.

[jsumners]

next will be a semver major. I am okay with breakages for spec compliance for sure.

I am likely to just cut next as a final release in the next few days.