TravisCI Time Limit

Hi everyone.

Description

TravisCI has a time limit of 50 minutes for each job as per their documentation. And as you can see on my last push https://github.com/thebigG/JobFunnel/commit/2417f6e858875539fe6fd8aeee42d5001e01eb82 the build on TravisCI fails because one of the scrape runs hits that 50 minutes cap.

One possible solution I have thought about is adding a number_of_pages/number of jobs limit to each scraper, but this might add complexity to our settings file and Command Line Interface so I’m just posting this issue to start the discussion on possible solutions to tackle this issue.

Another thought that comes to mind is limiting the scraping we do when testing, but I don’t think that’s optimal because we want our CI pipeline to be reliable and trustworthy, so at the very least we should be doing doing some scraping for every provider, even if it is just one page per provider.

Like I said, just wanted to start the discussion of potential solutions.

Steps to Reproduce

1. Run TravisCI script

Expected behavior

The tests should run without TravisCI intervention.

Actual behavior

The JobFunnel tests stop and fail because of the time limit.

Environment

• Build: JobFunnel 3.0.1 on commit cf80740
• Operating system and version: Pop!_OS 20.04 LTS
• [Linux] Desktop Environment and/or Window Manager: Cinnamon

[PaulMcInnis]

Thanks a bunch @thebigG

Lets open that PR and cut an issue for the scraping, since it sounds like this is easily replicated, will make it easier to fix.

[thebigG]

I’ve been investigating Github Actions and it is completely free for open source projects and it is very similar to TravisCI. There is a yaml file that describes the CI process, but the syntax is just slightly different of course. I’ll keep on investigating further and give you guys feedback on what I can find.