question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Issue with running scrapy spider from script.

See original GitHub issue

Hi ! I am trying to run this code

import csv
import scrapy
class SouqSpider(scrapy.Spider):
    name = "Souq"  # Name of the Spider, required value
    start_urls = ["http://deals.souq.com/ae-en/"]  # The starting url, Scrapy will request this URL in parse

    # Entry point for the spider
    def parse(self, response):
        for href in response.css('.sp-cms_unit--ttlattr(href)'):
            url = href.extract()
            yield scrapy.Request(url, callback=self.parse_item)

    # Method for parsing a product page
    def parse_item(self, response):
        original_price = -1
        savings=0
        discounted = False
        seller_rating = response.css('.vip-product-infoats .inline-block small::text'). extract()[0]
        seller_rating = int(filter(unicode.isdigit,seller_rating))

        # Not all deals are discounted
        if response.css('.vip-product-infobhead::text').extract():
            original_price = response.css('.vip-product-infobhead::text').extract()[0].replace("AED", "")
            discounted = True
            savings = response.css('.vip-product-infossage .noWrap::text').extract()[0].replace("AED", "")
        yield {
            'Title': response.css('.product-title:text').extract()[0],
            'Category': response.css('.product-title span a+ a::text').extract()[0],
            'OriginalPrice': original_price,
            'CurrentPrice': response.css('.vip-product-infoice::text').extract()[0].replace(u"\xa0", ""),
            'Discounted': discounted,
            'Savings': savings,
            'SoldBy': response.css('.vip-product-infoats a::text').extract()[0],
            'SellerRating': seller_rating,
            'Url': response.url
        }

However , i run this script i get this following error

class SouqSpider(scrapy.Spider): AttributeError: 'module' object has no attribute 'Spider'

anyone who can fix this error

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:7 (1 by maintainers)

github_iconTop GitHub Comments

2reactions
SiddharthaAnandcommented, Oct 25, 2017

Have you checked the version of scrapy that you are using ? It is possible that you are using an older version of scrapy and referring the new documentation. If it is an older version of scrapy, then run the following command and check again. pip install scrapy --upgrade

1reaction
mahecodecommented, Oct 27, 2017

still facing same error @ppm-khusbu

Read more comments on GitHub >

github_iconTop Results From Across the Web

Common Practices — Scrapy 2.7.1 documentation
By default, Scrapy runs a single spider per process when you run scrapy crawl . However, Scrapy supports running multiple spiders per process...
Read more >
Issue with running scrapy spider from script. #2473 - GitHub
Hi, I'm trying to run scrapy from a script like this: import scrapy from scrapy.crawler import CrawlerProcess class MySpider(scrapy.Spider): ...
Read more >
scrapy run spider from script - python - Stack Overflow
You can just create a normal Python script, and then use Scrapy's command line option runspider , that allows you to run a...
Read more >
How to Run a Scrapy Spider from a Python Script - Finxter
In the normal scrapy workflow, we begin by starting a project with scrapy's startproject command.
Read more >
How to Run Scrapy as a Stand-Alone Script | Teracrawler
While Scrapy is super useful, sometimes it could be a little stifling to create a project and then a spider and all the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found