After idling some hours the scheduler returns "running", accepts jobs, but refuses to execute them when "run_date" arrives

I find that if I keep Apscheduler busy, i.e., keep adding jobs to it, it will stay happy and active. But, if I leave it without job for a time, say, 12 hours or more, and add job again, even it is running (scheduler.state returns 1, which is RUNNING, when I consult with a web service), it accepts job, but does not execute it, when the run_date arrives.

I am using APScheduler along with Apache and Django.

Apache/2.4.6 (CentOS) PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5
APScheduler (3.4.0)
django-apscheduler (0.2.3)

APScheduler is running in the web app in Apache, not in another process. I tried using django-apscheduler (0.2.3) and without it, and the result is the same.

Some code:

'''
Scheduler and methods to handle jobs.

'''
class TaskScheduler(object):
    
    def __init__(self):
        '''
        Constructor. 
        Only can call it once because it create the sc.
        '''
        self.started = False
        self.sc = None
        self.sc = BackgroundScheduler()
#         self.sc.add_jobstore(DjangoJobStore(), 'default')
        
    
    def startScheduler(self):
        '''
        Start the scheduler. 
        @return if 'scheduler.start()' is called.
        @rtype: bool
        '''
        if (self.sc != None and self.sc.state == STATE_RUNNING):
            logger.debug("Scheduler already started.")
            return False
        else:
            try:
                self.sc = BackgroundScheduler()
#                 self.sc.add_jobstore(DjangoJobStore(), 'default')
                self.sc.start()
                self.started = True
                logger.info("Scheduler started STATUS: "+str(self.sc.state))
                return True
            except Exception as e:
                tb = traceback.format_exc(1)
                logger.error("EXCEPTION Error starting the Task Scheduler " + repr(e) + " - args - "+repr(tb))
                return False

    def stopScheduler(self):
        '''
        Shutdown the scheduler.
        @return if 'scheduler.shutdown()' is called. 
        @rtype: bool
        '''
        if (self.sc != None and self.sc.state != STATE_RUNNING):
            logger.debug("Scheduler already stopped.")
            return False
        else:
            self.sc.shutdown()
            self.started = False
            logger.info("Scheduler shut down.")
            return True
    
    def getStatusScheduler(self):
        '''
        Shutdown the scheduler.
        @return if 'scheduler.shutdown()' is called. 
        @rtype: bool
        '''
        if self.sc == None:
            logger.info("Scheduler Is None.")
            return -1
        else:
            return self.sc.state
                
    
    def planifyJob(self, name, subdirectory, extension, interface, interface_name, filesize, filerotate, timelimit, dateStart, continuousLoop=0, snaplength=0, filterstr="", autosync=0,username=""):
        try:
            from backEnd import tcpdmanage
            from backEnd.models import TPlanifiedJobs
            import jobscheduler
            
            returned,msg=tcpdmanage.verifyDumpcap(name, subdirectory, extension, interface, interface_name, filesize, filerotate, timelimit, continuousLoop, snaplength, filterstr, autosync, username)
            if not returned:
                return False, "Error Verifying Dumpcap process"
            processPlan=TPlanifiedJobs(name=name,subdirectory=subdirectory,extension=extension,interface=interface,interface_name=interface_name,filesize=filesize,filerotate=filerotate,timelimit=timelimit,autosync=autosync,continuousLoop=continuousLoop,snaplength=snaplength,filterDump=filterstr,state=properties.STATE_PLAN_PREPARING,startDate=dateStart)
            processPlan.save()
            idProcessPlan=processPlan.id
            logger.info("Planify Job "+str(dateStart))
            self.startScheduler()
            self.sc.add_job(jobscheduler.executionJob, 'date', run_date=dateStart, id=str(idProcessPlan), replace_existing=True, kwargs={'idProcessPlan': idProcessPlan, 'username': str(username)})
            return True,";"+str(idProcessPlan);
        except Exception as e:
            tb = traceback.format_exc(1)
            logger.error("ERROR planifyJob "+ repr(e) + " - args " + repr(tb))
            return False, "planifyJob Critical Error: "+repr(e);

[agronholm]

No, simply raising the logger level should be fine. You should be seeing plenty of additional info then.

[WesleyBlancoYuan]

Confirmed: The DB connection is the problem. I have reported the issue to django-apscheduler and is solved in 0.2.5. https://github.com/jarekwg/django-apscheduler/issues/13

With or without any persistent jobstore, to avoid this issue, in the callable of the job, you have to import Django models and do django.setup() to assure that Django comes in and manage the connection again to make sure the connection is alive. Now I do this:

import os
from backEnd import tcpdmanage
import traceback

def executionJob(idProcessPlan, username):
    logger.debug("Starting Execution Job...")
    try:
        os.environ.setdefault("DJANGO_SETTINGS_MODULE", "ManagerCaptureMain.settings")
        import django
        django.setup()
        from backEnd import models
        from backEnd import serializers
        planifyJobs=models.TPlanifiedJobs.objects.get(id=idProcessPlan)
        planifyJobsData=serializers.TPlanifiedJobsSerializer(planifyJobs)

So that even the DB connection dies, I make Django to reconnect first, because Django manages the connection automatically.

Now I think you can close the issue. Thanks.