博客
关于我
【重磅推荐】在Pycharm中调试scrapy爬虫的两种方法(有坑,务必注意)
阅读量:669 次
发布时间:2019-03-15

本文共 673 字,大约阅读时间需要 2 分钟。

PyCharm调试Scrapy常用的命令配置方法

Scrapy是一个强大的网页抓取框架,在PyCharm中调试Scrapy项目可以帮助开发者快速定位问题。以下是两种常用的调试方法:

第一种模式:使用scrapy.cmdline的execute方法

这种方法在项目根目录下创建main.py文件,配置相关调试参数。具体步骤如下:

  • 在项目根目录下找到scrapy.cfg文件,创建同级目录下的main.py文件
  • 在main.py文件中输入以下代码:
  • from scrapy.cmdline import execute
    import sys
    import os
    sys.path.append(os.path.dirname(os.path.abspath(__file__)))
    1. 在需要调试的爬虫文件中设置断点,运行main.py即可在PyCharm中调试
    2. 第二种模式:直接使用Scrapy命令

    3. 执行单文件调试模式1:
    4. scrapy runspider scrapy_first/spiders/book.py
      1. 执行单文件调试模式2:
      2. scrapy crawl spider_name

        注意:这里的spider_name是Spider子类的名称属性,而不是文件名

        示例:

        class Test1Spider(Spider):
        name = 'test'
        def parse(self, response):
        pass

        图1:PyCharm调试界面示例

        (Demo图展示了如何在PyCharm中设置断点和使用调试工具)

    转载地址:http://pimmz.baihongyu.com/

    你可能感兴趣的文章
    NO 157 去掉禅道访问地址中的zentao
    查看>>
    no available service ‘default‘ found, please make sure registry config corre seata
    查看>>
    No compiler is provided in this environment. Perhaps you are running on a JRE rather than a JDK?
    查看>>
    no connection could be made because the target machine actively refused it.问题解决
    查看>>
    No Datastore Session bound to thread, and configuration does not allow creation of non-transactional
    查看>>
    No fallbackFactory instance of type class com.ruoyi---SpringCloud Alibaba_若依微服务框架改造---工作笔记005
    查看>>
    No Feign Client for loadBalancing defined. Did you forget to include spring-cloud-starter-loadbalanc
    查看>>
    No mapping found for HTTP request with URI [/...] in DispatcherServlet with name ...的解决方法
    查看>>
    No mapping found for HTTP request with URI [/logout.do] in DispatcherServlet with name 'springmvc'
    查看>>
    No module named 'crispy_forms'等使用pycharm开发
    查看>>
    No module named cv2
    查看>>
    No module named tensorboard.main在安装tensorboardX的时候遇到的问题
    查看>>
    No module named ‘MySQLdb‘错误解决No module named ‘MySQLdb‘错误解决
    查看>>
    No new migrations found. Your system is up-to-date.
    查看>>
    No qualifying bean of type XXX found for dependency XXX.
    查看>>
    No qualifying bean of type ‘com.netflix.discovery.AbstractDiscoveryClientOptionalArgs<?>‘ available
    查看>>
    No resource identifier found for attribute 'srcCompat' in package的解决办法
    查看>>
    no session found for current thread
    查看>>
    No toolchains found in the NDK toolchains folder for ABI with prefix: mips64el-linux-android
    查看>>
    NO.23 ZenTaoPHP目录结构
    查看>>