Article `download()` failed with 404 Client Error

Hi,

I keep getting this error message - Article download() failed with 404 Client Error: Not Found for url: http://www.foxnews.com/2017/09/22/sheriff-clarke-trump-wins-either-way-luther-strange-roy-moore-alabama-senate-race on URL http://www.foxnews.com/2017/09/22/sheriff-clarke-trump-wins-either-way-luther-strange-roy-moore-alabama-senate-race

It happens for various article url links.

Here is the code i am using, `news_content = newspaper.build(url) for eachArticle in news_content.articles: i = i +1 article = news_content.articles[i]

    article.download()#now download and parse each articles
    article.parse()

    article.nlp()


    backupfile.write("\n"+ "--------------------------------------------------------------" + "\n")
    backupfile.write(str(article.keywords))


    datasetfile.write("\n" + "----SUMMARY ARTICLE-> No. " + str(i) + "\n")
    datasetfile.write(article.summary) #only summary of the article is written in the dataset directory


    backupfile.write("\n"+"----SUMMARY ARTICLE---" + "\n")
    backupfile.write(article.summary)
    backupfile.write("\n"+"----TEXT INSIDE ARTICLE---" + "\n")
    backupfile.write(article.text)
    time.sleep(2)`

Attached below is the screenshot of the error,

[monajalal]

I posted the solution here:

from newspaper import Article
from newspaper import Config

user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0'
config = Config()

config.browser_user_agent = user_agent
url = "https://www.newsweek.com/new-mexico-compound-charges-dropped-children-1096830".strip()



page = Article(url, config=config)


page.download()
page.parse()
print(page.text)

Here is the link: https://stackoverflow.com/a/63060794/2414957

[moosemachineDK]

I just used a simple try except structure. Seems to works just fine (at least for the 404 error I was seeing)(code below - don’t mind the splitting and stuff’ 😃)

    try:
        article.download()
        article.parse()
        article2 = article.text.split()
    except:
        print('***FAILED TO DOWNLOAD***', article.url)
        continue