The breakpoint in python functions between PyGILState_Ensure and PyGILState_Release is never hit.

I’ve discovered that breakpoints in python functions between PyGILState_Ensure and PyGILState_Release if located in a separated thread is never hit. The following test environment can be used to investigate the problem.

First you have to initialize and build the Python Interpreter and then call the python function “task” from a thread. Here is my code for this purpose:

// test.cpp : Defines the entry point for the console application.
//
#include "stdafx.h"
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#ifdef _DEBUG
#undef _DEBUG
#include <python.h>
#define _DEBUG
#else
#include <python.h>
#endif
#include <numpy/ndarrayobject.h>
#include <string>
#include <thread>

static const int     ERROR = 0;
static const int     OK = 1;

PyThreadState  *g_MainThreadState;
PyObject       *m_pModule    = 0x0;
std::string     m_ModulePath = "mymodule";


void CallPyFunction(const char *function);

int main()
{
	std::string m_SysCommand;

	// 1) Initialise python interpretator
	if (!Py_IsInitialized()) {
		Py_Initialize();
		Py_AtExit(Py_Finalize);
	}

	// 2) Initialise python thread mechanism 
	if (!PyEval_ThreadsInitialized()) {
		PyEval_InitThreads();
	}
	PyEval_SaveThread();


	PyGILState_STATE gstate = PyGILState_Ensure();

	// 3) Save the main thread state
	g_MainThreadState = PyThreadState_Get();

	// 4) create a fake command line directly after Py_Initialize() (Tensorflow can't be imported without this)
	wchar_t const *dummy_args[] = { L"Python", NULL };  // const is needed because literals must not be modified
	wchar_t const **argv = dummy_args;
	int             argc = sizeof(dummy_args) / sizeof(dummy_args[0]) - 1;
	PySys_SetArgv(argc, const_cast<wchar_t **>(argv)); // const_cast allowed, because PySys_SetArgv doesn't change argv
	if (PyErr_Occurred()) {
		return false;
	}

	// 5) Load main module (without reload)
	PyObject  *pName = PyUnicode_FromString(m_ModulePath.c_str());
	m_pModule = PyImport_Import(pName);
	Py_DECREF(pName);

	PyGILState_Release(gstate);




	///////////////// Python Function Call from a Thread ////////////////////
	std::thread worker([]
	{
		PyGILState_STATE gstate = PyGILState_Ensure();

		CallPyFunction("task");

		PyGILState_Release(gstate);
	});
	worker.join();

	/////////////////////////////////////////////////////////////////////////

    return 0;
}


void CallPyFunction(const char *function)
{
	PyObject *pFunc, *pArgs, *pValue;
	pFunc = PyObject_GetAttrString(m_pModule, function);
	if (pFunc && PyCallable_Check(pFunc)) {
		pArgs = PyTuple_New(0);
		pValue = PyObject_CallObject(pFunc, pArgs);
		if (pArgs) {
			Py_XDECREF(pArgs);
		}
		if (pFunc) {
			Py_XDECREF(pFunc);
		}
		if (pValue) {
			Py_XDECREF(pValue);
		}
	}
}

In addition, a test python file “mymodule.py” is required, which looks like this:

# mymodule.py : Defines python test script file
import ptvsd   

ptvsd.enable_attach(secret = 'mysecret', address = ('127.0.0.1', 8080))
ptvsd.wait_for_attach()

txt = 2.3

def task( ):
    global val
    val = 4.9       // <---------- Breakpoint must be set here
   return

The breakpoint, if set in the function “task”, is never reached.

If the full visual studio project is needed to this test environment, I can provide it.

[int19h]

I don’t see why not - it would be a trivial wrapper around existing functions.

BTW, I took a look at the implementation of set_trace in PDB, and it actually walks the frame stack backwards setting f_trace, not just calling sys.settrace - so we can actually do this in pure Python as well, and then debugging would work in the same frame that called the function.

[pauldotknopf]

I am using visual studio code, which as far as I can tell, requires exactly version 3.0.0 of ptvsd.

In 3.0.0, I don’t see the new new_external_thread method.

What would you recommend I do? I am having similar problems with PyOtherSide and it’s background worker thread.