"validate" cannot detect error in R code

I’m trying to use nbgrader for R language, but the “Validate” button and autograder doesn’t detect errors in answer cells. I tried the following commands to throw error, but “validate” button always returns the message “Success! Your notebook passes all the tests” and I always got full score.

raise NotImplementedError()

fail()

e <- simpleError("test error")
stop(e)

throw('NotImplementedError') 

When I run the cells above, they return Error, so I think nbgrader seems not to detect the thrown error. I’m currently using nbgrader for python on the same system without problems. Is there some solution?

Operating system

Ubuntu 16.04.6 LTS

nbgrader --version

nbgrader version 0.6.1

jupyterhub --version (if used with JupyterHub)

0.9.6

jupyter notebook --version

6.0.2

Expected behavior

Detect errors in notebook

Actual behavior

“Validate” button doesn’t detect erros.

[jhamrick]

Yeah this latter point does indeed seem like a bug, I would also expect the validator to say the test has failed even if it is worth zero points.

[jnishii]

Thank you for your comment.

I found three reasons why “Validate button” doesn’t work as my expectations.

1. “Validate button” doesn’t detect error by

   stop(“NotImplementedError”) or raise(…)

in answer cells, because answer cells are not targets of failure check in _get_failed_cells() in validator.py.

Now, I understand it’s a specification of nbgrader.

1. “Validate button” doesn’t detect an error given by test_that() function, because the error gives "output_type": “stream”, so this is a similar case in #1381. I was pointed out about this problem by Ian before, but I checked the results of the above case (1), and didn’t notice this problem. Sorry Ian.

A different point from #1381 is that test_that() function doen’t produce output.name=stderr but output.name=stdout, so there no clue to find the error in the autograde cell.

To resolve this problem, I redefined the test_that() as below.

test_that <- function(...){
    stopifnot(testthat::test_that(...))
}

It seems working now!

1. “Validate button” doesn’t detect the error in autograde cell when points of the cell is set to 0. I myself think that it would be better to detect errors even for this case. Is this an expected behavior of nbgrader?