skip some tests when run with (ultest-run-file)

Describe the bug language/runner: ruby/rspec There are two problem that I found:

• (ultest-run-file) skips some tests.
• UltestSummary ignores rspec context.

log

15:39:03 | INFO | MainThread | logging.py:create_logger:101 | Logger created
15:39:04 | DEBUG | MainThread | __init__.py:__init__:44 | Handler created
15:39:04 | INFO | MainThread | tracker.py:_init_test_file:97 | Initialising test file /Users/marocchino/Documents/internal_system/internal_system/spec/test_spec.rb
15:39:04 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:62 | Starting job with group update_positions
15:39:04 | INFO | Thread-1 | tracker.py:_async_update:57 | Updating positions in /Users/marocchino/Documents/internal_system/internal_system/spec/test_spec.rb
15:39:04 | DEBUG | Thread-1 | file.py:parse_file_structure:25 | Converted pattern {'test': ['\\v^\\s*def (test_\\w+)', '\\v^\\s*test%(\\(| )%("|\')(.*)%("|\')', '\\v^\\s*it%(\\(| )%("|\')(.*)%("|\')'], 'namespace': ['\\v^\\s*%(class|module) (\\S+)', '\\v^\\s*describe%(\\(| )%("|\')(.*)%("|\')', '\\v^\\s*describe%(\\(| )(\\S+)']} to {'test': [re.compile('^\\s*def (test_\\w+)'), re.compile('^\\s*test(?:\\(| )(?:"|\')(.*)(?:"|\')'), re.compile('^\\s*it(?:\\(| )(?:"|\')(.*)(?:"|\')')], 'namespace': [re.compile('^\\s*(?:class|module) (\\S+)'), re.compile('^\\s*describe(?:\\(| )(?:"|\')(.*)(?:"|\')'), re.compile('^\\s*describe(?:\\(| )(\\S+)')]}
15:39:04 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test /Users/marocchino/Documents/internal_system/internal_system/spec/test_spec.rb found in /Users/marocchino/Documents/internal_system/internal_system/spec/test_spec.rb
15:39:04 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test 16319928273942723129 found in /Users/marocchino/Documents/internal_system/internal_system/spec/test_spec.rb
15:39:04 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test will_be_16824078959065154280 found in /Users/marocchino/Documents/internal_system/internal_system/spec/test_spec.rb
15:39:04 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test 26824078959065154280 found in /Users/marocchino/Documents/internal_system/internal_system/spec/test_spec.rb
15:39:04 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test 36319928273942723129 found in /Users/marocchino/Documents/internal_system/internal_system/spec/test_spec.rb
15:39:04 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test will_be_4-2092094269204336961 found in /Users/marocchino/Documents/internal_system/internal_system/spec/test_spec.rb
15:39:04 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test will_be_5-2092094269204336961 found in /Users/marocchino/Documents/internal_system/internal_system/spec/test_spec.rb
15:39:04 | DEBUG | Thread-1 | tracker.py:_remove_old_positions:129 | No tests removed
15:39:04 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:82 | Finished job with group update_positions
15:39:14 | INFO | MainThread | __init__.py:run_nearest:126 | Running nearest test in spec/test_spec.rb at line 0
15:39:14 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering will_be_16824078959065154280 as started
15:39:14 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering 26824078959065154280 as started
15:39:14 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:62 | Starting job with group will_be_16824078959065154280
15:39:14 | DEBUG | Thread-1 | processes.py:run:50 | Starting test process will_be_16824078959065154280 with command ['bundle', 'exec', 'rspec', 'spec/test_spec.rb:9'], cwd = /Users/marocchino/Documents/internal_system/internal_system, env = None
15:39:14 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering will_be_4-2092094269204336961 as started
15:39:14 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:62 | Starting job with group 26824078959065154280
15:39:14 | DEBUG | Thread-1 | processes.py:run:50 | Starting test process 26824078959065154280 with command ['bundle', 'exec', 'rspec', 'spec/test_spec.rb:17'], cwd = /Users/marocchino/Documents/internal_system/internal_system, env = None
15:39:14 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering will_be_5-2092094269204336961 as started
15:39:14 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:62 | Starting job with group will_be_4-2092094269204336961
15:39:14 | DEBUG | Thread-1 | processes.py:run:50 | Starting test process will_be_4-2092094269204336961 with command ['bundle', 'exec', 'rspec', 'spec/test_spec.rb:26'], cwd = /Users/marocchino/Documents/internal_system/internal_system, env = None
15:39:14 | WARNING | Thread-1 | __init__.py:_handle_coroutine:76 | Exception throw in job: Task <Task pending name='Task-15' coro=<PositionRunner._run_separately.<locals>.run() running at /Users/marocchino/.local/share/nvim/site/pack/packer/start/vim-ultest/rplugin/python3/ultest/handler/runner/__init__.py:153> cb=[_wait.<locals>._on_completion() at /usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py:513]> got Future <Future pending> attached to a different loop
15:39:14 | WARNING | Thread-1 | __init__.py:_handle_coroutine:77 | Traceback (most recent call last):

  File "/Users/marocchino/.local/share/nvim/site/pack/packer/start/vim-ultest/rplugin/python3/ultest/handler/runner/__init__.py", line 153, in run
    (code, output_path) = await self._processes.run(

  File "/Users/marocchino/.local/share/nvim/site/pack/packer/start/vim-ultest/rplugin/python3/ultest/handler/runner/processes.py", line 54, in run
    async with self._vim.semaphore:

  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/locks.py", line 14, in __aenter__
    await self.acquire()

  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/locks.py", line 413, in acquire
    await fut

RuntimeError: Task <Task pending name='Task-15' coro=<PositionRunner._run_separately.<locals>.run() running at /Users/marocchino/.local/share/nvim/site/pack/packer/start/vim-ultest/rplugin/python3/ultest/handler/runner/__init__.py:153> cb=[_wait.<locals>._on_completion() at /usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py:513]> got Future <Future pending> attached to a different loop

15:39:14 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:82 | Finished job with group will_be_4-2092094269204336961
15:39:14 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:62 | Starting job with group will_be_5-2092094269204336961
15:39:14 | DEBUG | Thread-1 | processes.py:run:50 | Starting test process will_be_5-2092094269204336961 with command ['bundle', 'exec', 'rspec', 'spec/test_spec.rb:32'], cwd = /Users/marocchino/Documents/internal_system/internal_system, env = None
15:39:14 | WARNING | Thread-1 | __init__.py:_handle_coroutine:76 | Exception throw in job: Task <Task pending name='Task-18' coro=<PositionRunner._run_separately.<locals>.run() running at /Users/marocchino/.local/share/nvim/site/pack/packer/start/vim-ultest/rplugin/python3/ultest/handler/runner/__init__.py:153> cb=[_wait.<locals>._on_completion() at /usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py:513]> got Future <Future pending> attached to a different loop
15:39:14 | WARNING | Thread-1 | __init__.py:_handle_coroutine:77 | Traceback (most recent call last):

  File "/Users/marocchino/.local/share/nvim/site/pack/packer/start/vim-ultest/rplugin/python3/ultest/handler/runner/__init__.py", line 153, in run
    (code, output_path) = await self._processes.run(

  File "/Users/marocchino/.local/share/nvim/site/pack/packer/start/vim-ultest/rplugin/python3/ultest/handler/runner/processes.py", line 54, in run
    async with self._vim.semaphore:

  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/locks.py", line 14, in __aenter__
    await self.acquire()

  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/locks.py", line 413, in acquire
    await fut

RuntimeError: Task <Task pending name='Task-18' coro=<PositionRunner._run_separately.<locals>.run() running at /Users/marocchino/.local/share/nvim/site/pack/packer/start/vim-ultest/rplugin/python3/ultest/handler/runner/__init__.py:153> cb=[_wait.<locals>._on_completion() at /usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py:513]> got Future <Future pending> attached to a different loop

15:39:14 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:82 | Finished job with group will_be_5-2092094269204336961
15:39:16 | DEBUG | Thread-1 | processes.py:run:76 | Process will_be_16824078959065154280 complete with exit code: 0
15:39:16 | DEBUG | Thread-1 | __init__.py:_register_result:299 | Registering will_be_16824078959065154280 as exited with result {"id": "will_be_16824078959065154280", "file": "/Users/marocchino/Documents/internal_system/internal_system/spec/test_spec.rb", "code": 0, "output": "/var/folders/2w/hk_dhmjx0jv2p00mzx9y59vr0000gp/T/ultestem5g_l06/__Users__marocchino__Documents__internal_system__internal_system__spec__test_spec_rb/will_be_16824078959065154280_out"}
15:39:16 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:82 | Finished job with group will_be_16824078959065154280
15:39:16 | DEBUG | Thread-1 | processes.py:run:76 | Process 26824078959065154280 complete with exit code: 0
15:39:16 | DEBUG | Thread-1 | __init__.py:_register_result:299 | Registering 26824078959065154280 as exited with result {"id": "26824078959065154280", "file": "/Users/marocchino/Documents/internal_system/internal_system/spec/test_spec.rb", "code": 0, "output": "/var/folders/2w/hk_dhmjx0jv2p00mzx9y59vr0000gp/T/ultestem5g_l06/__Users__marocchino__Documents__internal_system__internal_system__spec__test_spec_rb/26824078959065154280_out"}
15:39:16 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:82 | Finished job with group 26824078959065154280

To Reproduce Steps to reproduce the behavior:

# frozen_string_literal: true

require 'rails_helper'

RSpec.describe 'Test' do
  describe '1' do
    let(:number) { 1 }

    it 'will be 1' do
      expect(number).to eq(1)
    end
  end

  context 'when 2' do
    let(:number) { 2 }

    it '2' do
      expect(number).to eq(2)
    end
  end

  describe '3' do
    let(:number) { 3 }

    context 'when 4' do
      it 'will be 4' do
        expect(number).to eq(4)
      end
    end

    context 'when 5' do
      it 'will be 5' do
        expect(number).to eq(5)
      end
    end
  end
end

and run with <Plug>(ultest-run-file)

Expected behavior run against entire suite in this file.

Screenshots

but, skip for nested context. and rspec `context` are ignored in summary.

Additional context But I can run will be 4 and will be 5 tests with (ultest-run-nearest).

[marocchino]

@rcarriga Great! The problem seems to be fixed for ruby. Thank you! 👍🏽 @Rofer11607 it’s name is ‘monofur’.

[rcarriga]

So the ignored context is actually a vim-test problem. It defines this to search for tests/namespaces:

let test#ruby#patterns = {
  \ 'test': [
    \ '\v^\s*def (test_\w+)',
    \ '\v^\s*test%(\(| )%("|'')(.*)%("|'')',
    \ '\v^\s*it%(\(| )%("|'')(.*)%("|'')',
  \],
  \ 'namespace': [
    \ '\v^\s*%(class|module) (\S+)',
    \ '\v^\s*describe%(\(| )%("|'')(.*)%("|'')',
    \ '\v^\s*describe%(\(| )(\S+)',
  \],
\}

You can fix this by setting it to this

let test#ruby#patterns = {
  \ 'test': [
    \ '\v^\s*def (test_\w+)',
    \ '\v^\s*test%(\(| )%("|'')(.*)%("|'')',
    \ '\v^\s*it%(\(| )%("|'')(.*)%("|'')',
  \],
  \ 'namespace': [
    \ '\v^\s*%(class|module) (\S+)',
    \ '\v^\s*describe%(\(| )%("|'')(.*)%("|'')',
    \ '\v^\s*describe%(\(| )(\S+)',
    \ '\v^\s*context%(\(| )%("|'')(.*)%("|'')',
    \ '\v^\s*context%(\(| )(\S+)',
  \],
\}

though I’m not familiar with Ruby so there may be further issues with those patterns.

If that does work I’d suggest creating a PR for vim-test so others can benefit 😁

For the other problem, it doesn’t ignore tests but looks like there is an issue with asyncio which is odd…

RuntimeError: Task <Task pending name='Task-18' coro=<PositionRunner._run_separately.<locals>.run() running at /Users/marocchino/.local/share/nvim/site/pack/packer/start/vim-ultest/rplugin/python3/ultest/handler/runner/__init__.py:153> cb=[_wait.<locals>._on_completion() at /usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py:513]> got Future <Future pending> attached to a different loop

Do you have any other Python remote plugins running?