EventHubs.Define does not create all rules

Describe the bug I’m using EventHub.Define to create a new EventHub with both SendRule and ListenRule. After creating the hub I’m using eventHub.ListAuthorizationRulesAsync to get all rules. Since Monday eventHub.ListAuthorizationRulesAsync does not return the correct count of Rules. Even a retry strategy using Polly does not fix that problem.

Exception or Stack Trace There is no exception. The API returns not all rules.

To Reproduce Steps to reproduce the behavior:

Code Snippet

var eventHubs = await azure.EventHubNamespaces.ListByResourceGroupAsync(resourceGroupName);
var eventHubNamespace = eventHubs.FirstOrDefault(x => x.Name == eventHubNamespaceName);

if (eventHubNamespace == null)
{
	eventHubNamespace = await azure.EventHubNamespaces.Define(eventHubNamespaceName)
		.WithRegion(Region.EuropeWest)
		.WithExistingResourceGroup(resourceGroupName)
		.CreateAsync();
}

var eventHub = await azure.EventHubs.Define(eventHubName)
	.WithExistingNamespace(eventHubNamespace)
	.WithPartitionCount(1) // Use only 1 partition, to avoid that event order is destroyed. 
	.WithNewSendRule(Names.EventHubNamespace.SendRule)
	.WithNewListenRule(Names.EventHubNamespace.ListenRule)
	.CreateAsync();

// Retry 8 times because sometimes the keys are not there  
var rules = await Policy
	.Handle<InvalidOperationException>()
	.WaitAndRetryAsync(9, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)))
	.ExecuteAsync(
		async () =>
		{
			await eventHub.RefreshAsync();
			var result = await eventHub.ListAuthorizationRulesAsync();
			
			if (result.Count() < 2) // Count should be 2, because WithNewSendRule & WithNewListenRule is set
			{
				throw new InvalidOperationException("Not all AuthorizationRules where created!");
			}

			return result;
		});

Expected behavior SendRule and ListenRule should be created and can be Listed correctly.

Screenshots

Setup (please complete the following information):

• OS: Windows 10

• IDE : Rider, VS, Azure DevOps Hosted Build Agent, .NET Core 2.2

• Version of the Library used:

  • Microsoft.Azure.Management.Fluent Version="1.27.2
  • Microsoft.Azure.Management.ResourceManager.Fluent Version=“1.27.2”

Additional context The problem starts on Monday (2019.11.14) and was detected by our system tests, which runs on an Azure DevOps HostedAgent (Image: windows-2019). Before Monday everything was fine.

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[nickzhums]

@MariuszKogut Closing this issue as it is resolved

[weidongxu-microsoft]

@MariuszKogut It appears to be intermittent. A possible cause under investigation is concurrent call to create SendRule and ListenRule in parellel. Service might not like this.

Before this issue got triaged and fixed, separate SendRule and ListenRule should help:

var eventHub = await azure.EventHubs.Define(eventHubName)
	.WithExistingNamespace(eventHubNamespace)
	.WithPartitionCount(1) // Use only 1 partition, to avoid that event order is destroyed. 
	.WithNewSendRule(Names.EventHubNamespace.SendRule)
	.CreateAsync();

await eventHub.Update()
	.WithNewListenRule(Names.EventHubNamespace.ListenRule)
	.ApplyAsync();