Welcome toVigges Developer Community-Open, Learning,Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
469 views
in Technique[技术] by (71.8m points)

scrapy启动后导入自建模块失败,找不到,哪里出了问题?

已经标记源码根了(解除又标记了几次),scrapy项目crawl启动后提示找不到自建的py文件,导入失败,sys.path.append也不行,运行单个py文件却能正常使用导入的自建模块
QQ截图20200730232527.png
导入的模块点进去可以跳转进那个文件,现在该怎么办?
安装了一个第三方useragent库fake-useragent,使用的时候也报错,说找不到这个模块

2020-07-31 01:07:17 [scrapy.utils.log] INFO: Scrapy 2.2.1 started (bot: bttiantang)
2020-07-31 01:07:17 [scrapy.utils.log] INFO: Versions: lxml 4.5.1.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)],
pyOpenSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020), cryptography 3.0, Platform Windows-10-10.0.18362-SP0
2020-07-31 01:07:17 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-07-31 01:07:17 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'bttiantang',
 'DOWNLOAD_DELAY': 3,
 'NEWSPIDER_MODULE': 'bttiantang.spiders',
 'SPIDER_MODULES': ['bttiantang.spiders'],
 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
               '(KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'}
2020-07-31 01:07:17 [scrapy.extensions.telnet] INFO: Telnet Password: xxxxxxxxxxx
2020-07-31 01:07:17 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
Unhandled error in Deferred:
2020-07-31 01:07:17 [twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last):
  File "d:softwarespython3.8libsite-packagesscrapycrawler.py", line 192, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "d:softwarespython3.8libsite-packagesscrapycrawler.py", line 196, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "d:softwarespython3.8libsite-packageswistedinternetdefer.py", line 1613, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "d:softwarespython3.8libsite-packageswistedinternetdefer.py", line 1529, in _cancellableInlineCallbacks
    _inlineCallbacks(None, g, status)
--- <exception caught here> ---
  File "d:softwarespython3.8libsite-packageswistedinternetdefer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "d:softwarespython3.8libsite-packagesscrapycrawler.py", line 87, in crawl
    self.engine = self._create_engine()
  File "d:softwarespython3.8libsite-packagesscrapycrawler.py", line 101, in _create_engine
    return ExecutionEngine(self, lambda _: self.stop())
  File "d:softwarespython3.8libsite-packagesscrapycoreengine.py", line 69, in __init__
    self.downloader = downloader_cls(crawler)
  File "d:softwarespython3.8libsite-packagesscrapycoredownloader\__init__.py", line 83, in __init__
    self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
  File "d:softwarespython3.8libsite-packagesscrapymiddleware.py", line 53, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "d:softwarespython3.8libsite-packagesscrapymiddleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "d:softwarespython3.8libsite-packagesscrapyutilsmisc.py", line 50, in load_object
    mod = import_module(module)
  File "d:softwarespython3.8libimportlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

  File "<frozen importlib._bootstrap>", line 991, in _find_and_load

  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 783, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "F:PythonTest_1ttiantangttiantangmiddlewares.py", line 7, in <module>
    from fake_useragent import UserAgent
builtins.ModuleNotFoundError: No module named 'fake_useragent'

2020-07-31 01:07:17 [twisted] CRITICAL:
Traceback (most recent call last):
  File "d:softwarespython3.8libsite-packageswistedinternetdefer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "d:softwarespython3.8libsite-packagesscrapycrawler.py", line 87, in crawl
    self.engine = self._create_engine()
  File "d:softwarespython3.8libsite-packagesscrapycrawler.py", line 101, in _create_engine
    return ExecutionEngine(self, lambda _: self.stop())
  File "d:softwarespython3.8libsite-packagesscrapycoreengine.py", line 69, in __init__
    self.downloader = downloader_cls(crawler)
  File "d:softwarespython3.8libsite-packagesscrapycoredownloader\__init__.py", line 83, in __init__
    self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
  File "d:softwarespython3.8libsite-packagesscrapymiddleware.py", line 53, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "d:softwarespython3.8libsite-packagesscrapymiddleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "d:softwarespython3.8libsite-packagesscrapyutilsmisc.py", line 50, in load_object
    mod = import_module(module)
  File "d:softwarespython3.8libimportlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "F:PythonTest_1ttiantangttiantangmiddlewares.py", line 7, in <module>
    from fake_useragent import UserAgent
ModuleNotFoundError: No module named 'fake_useragent'

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)
等待大神解答

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to Vigges Developer Community for programmer and developer-Open, Learning and Share
...