"az storage message get" crashes (Ubuntu 20.04 LTS, az-cli 2.36.0-1~focal)

Describe the bug

az storage queue get fails

To Reproduce

1. start azurite (or connect to a azure hosted storage account)
2. set AZURE_STORAGE_CONNECTION_STRING
3. az storage queue create --name egress-test
4. az storage message get -q “egress-test”

Expected behavior

This should not crash.

❯   az storage message get -q "egress-test"   
Command group 'storage message' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
The command failed with an unexpected error. Here is the traceback:
End of paging
Traceback (most recent call last):                                                                                                                            
  File "/opt/az/lib/python3.8/site-packages/knack/cli.py", line 231, in invoke                                                                                
    cmd_result = self.invocation.execute(args)                                                                                                                
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute                                                        
    raise ex                                                                                                                                                  
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially                                             
    results.append(self._run_job(expanded_arg, cmd_copy))                                                                                                     
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 713, in _run_job                                                       
    return cmd_copy.exception_handler(ex)                                                                                                                     
  File "/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/storage/__init__.py", line 409, in new_handler                                          
    first(ex)                                                                                                                                                 
  File "/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/storage/__init__.py", line 408, in new_handler                                          
    raise ex                                                                                                                                                  
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job                                                       
    result = cmd_copy(params)                                                                                                                                 
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__                                                       
    return self.handler(*args, **kwargs)                                                                                                                      
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler                                               
    return op(**command_args)                                                                                                                                 
  File "/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/storage/operations/queue.py", line 86, in receive_messages                              
    return list(next(page_iter))                                                                                                                              
  File "/opt/az/lib/python3.8/site-packages/azure/core/paging.py", line 84, in __next__                                                                       
    self.continuation_token, self._current_page = self._extract_data(self._response)                                                                          
  File "/opt/az/lib/python3.8/site-packages/azure/multiapi/storagev2/queue/v2018_03_28/_models.py", line 288, in _extract_data_cb                             
    raise StopIteration("End of paging")                                                                                                                      
StopIteration: End of paging                                                                                                                                  
To open an issue, please run: 'az feedback'

Environment summary

ubuntu 20.04 container with azure-cli installed from ms repo.

ii  azure-cli                  2.36.0-1~focal                    all          Azure CLI

[jalauzon-msft]

Hi @evelyn-ys Yishi / @navba-MSFT, I don’t believe this to be an SDK issue but rather an issue with how the CLI is calling the Track2 receive_messages API. Here is the code snippet I think CLI is using:

https://github.com/Azure/azure-cli/blob/cc0d149c1d3bea7ffbac12f8e97e0286cddfaa23/src/azure-cli/azure/cli/command_modules/storage/operations/queue.py#L84-L86

When using by_page() and then calling next on the resulting Iterator, it is expected that the Iterator will throw a StopIteration exception if there is not a next item, like is the case when the Queue is empty. There are a couple of immediate ways around this that I can think of: Catching the StopIteration exception or using the default value param of next().

# Option 1: Default value for next() to empty list
page_iter = client.receive_messages(**kwargs).by_page()
return list(next(page_iter, []))  # Maybe split lines out to avoid extra list construction

# Option 2: Catch and handle StopIteration exception
page_iter = client.receive_messages(**kwargs).by_page()
try:
    page = next(page_iter)
except StopIteration:
    return []
return list(page)

Either of those should solve the Issue at hand but I’m also curious what the CLI actually is trying to do? Is this intention to receive a certain number of messages? Is that user settable?

• If you just need a single message, I would use receive_message singular from the SDK
• If you need a user-specified amount, there are two approaches:
  • The approach that is done today should work. I assume the user can set messages_per_page or something alike? There is some odd behavior with receive_messages in this case where it will continually get messages unless you ask for each page via by_page() so the approach is correct here.
  • We just introduced a max_messages parameter to this function (in our next release) where you can specify a maximum number of messages so it does not continually grab messages from the Queue. In that case you would not need to use by_page() at all and could just do something like below, which would also eliminate the StopIteration exception on an empty Queue. (Also see the SDK sample)

pages = client.receive_messages(max_messages=5, **kwargs)
return list(pages)

Note: An optimization to the above would be to also set messages_per_page since that controls how many messages the server returns in a single request. If not set, the default is 1.

Sorry for the long message but hopefully that clears up the Issue and provides some potential fixes. Please ping me offline if you need more help. Thanks!

[navba-MSFT]

Thanks @jalauzon-msft . As per your above suggestion, I have done the necessary changes and filed the below PR. Currently it is pending on review.

PR: https://github.com/Azure/azure-cli/pull/22335