[proposal] Refactor/Improve LDAP implementation to be standards compliant

Description

Hi all. I would like to contribute to this project by correctly implementing LDAP so that the two common LDAP Directories can be utilized, Active Directory and OpenLDAP. I write this proposal to start a discussion prior to commencing any coding and for your input.

The current implementation enforces restrictions upon it’s usage due to it’s design which prevents the end user from fully utilising the ldap protocol as it was intended.

Example:

• hard coded filter restriction enforces that the ldap search filter ends with ={0}

• user groups only out of the box for Active Directory

• OpenLDAP (a V3 compliant LDAP Server) is unable to use LDAP groups

• for an LDAP Directory to be able to use LDAP Groups, they must implement a non-standards complaint schema that may have further unintended consequences.

Proposal

Refactor the ldap implementation in frappe so that it works as one would intend/assume. I.e.

• user customisable LDAP Filters

• distinguish the LDAP Directory being used so that the correct logic is applied to utilize LDAP

• LDAP Groups work out of the box for a standards compliant LDAP Directory as well as Active Directory.

Even though there are many LDAP Directories and it would be beyond the scope of this proposal to suggest that all of them could be implemented at once. This proposal is to ensure the common LDAP Directories can be used, being Active Directory and OpenLDAP. The changes to be made are simple enough to implement and would enable LDAP standards complaint directories to be fully implemented.

end state: frappe be able to use both Active Directory and OpenLDAP as it’s directory server (auth and groups). With the end user being able to use finer grain controls. i.e. ldap search filters.

UI Changes

New drop down to denote the LDAP Directory implementation, choices: Active Directory, OpenLDAP and Custom

Update description for field ldap_group_field to state ‘enter the ldap field for user groups Note: this field only works for custom directory type’

Add field LDAP Group OU for base ldap search for ldap groups

Proposed UI

Additional changes noticed not mentioned above are cosmetic with the layout updated to make sense

Logic changes

be able to distinguish LDAP Directory: This is required as some implementations of the LDAP protocol are not standards compliant. i.e. Microsoft Active Directory

new function to fetch user groups: A Standards complaint LDAP implementation does not have a user attribute to denote group membership. Therefore an additional LDAP search is required to fetch the user groups.
Even though Microsoft Active Directory is a non-standards compliant LDAP implementation and this proposal is for standards compliancy. It’s proposed that this be the only non-standard implementation be included as part of this proposal. This is due to it’s widespread usage and removing it would possibly remove users from the enterprise space.

After a quick review of the code ldap_settings.py#L165-#L167 I see this as the place to add the group functionality.

(please correct me if I’m wrong.)

Psuedo code

    groups = None

    groups = self.fetch_user_groups(user)

# Insert functions remaining logic here


def fetch_ldap_groups(self, user):
		import ldap3

		# NOTE: Don't use the user groups attribute from 'ldap.user'. this restricts the end user from creating a user with limited access to LDAP that can view only certain OU.

		# Unit test / validation ??: 
			# UI field is only dropdown with values of "AD", 'openldap' and 'custom'
			# test user is of type string


		conn = self.connect_to_ldap(self.base_dn, self.get_password(raise_exception=False))


		ldap_attributes = ['objectClass']

		ldap_object_class = None
		ldap_group_members_attribute = None
		ldap_group_search_filter = None
		if ldap_directory_server.lower() == 'active directory':
						

			# FIELD: 'member' Group oid: 1.2.840.113556.1.5.8
			ldap_object_class = 'Group'
			ldap_group_members_attribute = 'member'
			

    		else if ldap_directory_server.lower() == 'openldap':

        		# NOTE: OID of group will have to be checked so that the correct field that contains the user is selected.

			# FIELD: 'member' (GroupOfNames oid: 2.5.6.9)
			# FIELD: 'uniqueMember' (GroupOfUniqueNames oid: 2.5.6.17)
			# FIELD: 'MemberUid' (PosixGroup oid: 1.3.6.1.1.1.2.2)

			# NOTE: for openldap use the V3 standard field from the list above or the most common group type, the rest can be set in custom. To Do: find answer

			ldap_attributes.append('field containing usernames')
			ldap_group_members_attribute = ''

    		else if ldap_directory_server.lower() == 'custom':

			ldap_object_class = self.ldap_group_objectclass
			ldap_group_members_attribute = self.ldap_group_member_attribute

			
    		else
			# this path a possible candidate for unit test.
                        # this path will be hit for everyone with preconfigured ldap settings. this must be taken into account so as not to break ldap for those users.
       			frappe.throw('This is a catch all exception that will only fire if the "field Directory" server has been changed')

		ldap_attributes.append(ldap_group_members_attribute)

		conn.search(
			search_base=self.organizational_unit_for_groups,
			search_filter="(&(objectClass={0})({1}={2}))".format(ldap_object_class,ldap_group_members_attribute, user),
			attributes=ldap_attributes) # Build search query

		usergroups = None
		if len(conn.entries) >= 1:

			#LOGIC: iterate through groups and add to usergroups array.

		return usergroups

Adjust methods that use the ldap_group_field to either ignore/use depending on the LDAP Server

Adjust validation of the LDAP search string so that the user can use custom filters. This may lead to python exceptions. These exceptions will need to be caught and presented to the user.

This Proposal closes/fixes/related issues

Frappe:

• Fix [Feature] OpenLDAP group membership frappe/frappe/#10794

• Already fixed LDAP Implementation is broken (in delevop) frappe/frappe#6037

ERPNext:

• fix LDAP Search String needs to end with a placeholder frappe/frappe#18530

• fix OpenLDAP Integration: Upgrade to Python LDAP3 Makes Complex Filtering Impossible frappe/erpnext#18053

Relevant Links

• LDAP Groups

• Active Directory

proposal Checklist / Tasks

User interface

• Add field Directory type Mandatory

• Add fields custom_group_objectclass and group_member_attribute

• ldap_group_field field help updated depreciation message

• clean up interface layout

Logic

• AD Groups Functioning

• OpenLDAP Groups functioning

• ldap search adjusted so user can customize more than *={0}

• exceptions caught on all paths

• depreciation changes still work for users when they upgrade and haven’t updated settings

unit test checklist

Mock LDAP

• OpenLDAP
• Active Directory

LDAP Data for testing (ldif)

• Domain
• bind_user
• ou users
• ou groups
• group users
• group admin
• group three
• user1 (group1, group2)
• user2 (group1, group3)

Test setup validated

• works
• no invalid data
• users in correct groups

Functions

• all work
• exceptions thrown
  • User Filter - ldap3.core.exceptions.LDAPInvalidFilterError
  • for custom filter ldap3.core.exceptions.LDAPInvalidFilterError if user has entered incorrect values
• invalid data fails
• valid data works
• function return type confirmed

User roles

• non configured cause no change
• user1 only assigned to group1 and group2 roles
• user2 only assigned to group1 and group3 roles

User

• login success
• login failure (1x invalid user and 1x invalid password) must output same exception

Check

• simple ldap search string
• complex ldap search string
• additional custom fields validated when custom ldap directory selected

Documentation

• Create ldap integration frapp_docs

• edit erpnext ldap integration docs to point to frappe ldap integration docs

Questions

• I have noticed that there are test files, but there are no actual unit tests. Is this intentional? Should I be writing unit tests? If no it seems pointless to have unit testing with no actual testing.

• will my PR be git-squashed? (I ask as this will enable me to not have to adjust how I compose git comments. Yes I have read the contribution guide, yes I’m aware this makes me sound lazy. Yes I’m lazy, aren’t we all)

• which branch should I be developing on? The contribution docs link to erpnext and aren’t that clear

edit 1: update pseudo code edit 2: added UI gif edit 3: update tasks and questions edit 4: update tasks(docs) edit 5: remove answered questions edit 6: update tasks(unit test)

[jon-nfc]

🔗 Related Items
Proposal: frappe/frappe#13738 PR: frappe/frappe#13777
📖 Documentation
ERPNext: PR frappe/erpnext_documentation#376 Frappe: PR frappe/frappe_docs#168

[revant]

Should I be writing unit tests?

Whenever test that you’ve written fail while merging new PRs you’ll be notified by many people as that creates a blocker during merge. If there are no tests you can’t prevent new code that breaks your feature. You won’t even know which merge request broke your feature. Even when someone upgrades the libs you import in your code, tests will ensure feature to work.

which branch should I be developing on

develop

There are no LDAP docs that I have noticed within frappe, should I create or move the docs from erpnext? given that LDAP is part of frappe not erpnext

I feel LDAP docs need to be part of framework docs. ERPNext docs should have a link to LDAP section of Frappe docs. Wait for others to comment on this.

I’m currently running a complete stack with less than 1.5gb ram.

Comment the watch line in Procfile, it may reduce some RAM usage.