scrapy/scrapy
186 findings in 207 Python files · 39,929 lines · scanned in 24.6 s on 2026-11-12
Every number here comes from the same analyzer that runs on your upload, on the tree as it is in the repository. A finding is listed with the function or module it sits in; nothing is matched from text. Tests, examples and benchmarks are excluded from the scan, so "never referenced inside the project" means exactly that.
import_cycle 3
modules that import each other
scrapy.utils.asyncioCircular import between 2 modules: scrapy.utils.asyncio -> scrapy.utils.reactorscrapy.linkextractorsCircular import between 2 modules: scrapy.linkextractors -> scrapy.linkextractors.lxmlhtmlscrapyCircular import between 22 modules: scrapy -> scrapy.http -> scrapy.http.request -> scrapy.http.request.form -> scrapy.http.request.json_request -> scrapy.http.request.rpc -> scrapy.http.response -> scrapy.http.response.body_file -> scrapy.http.response.html -> scrapy.http.response.json -> scrapy.http.response.text -> scrapy.http.response.xml -> scrapy.responsetypes -> scrapy.selector -> scrapy.selector.unified -> scrapy.spiders -> scrapy.spiders.crawl -> scrapy.spiders.feed -> scrapy.spiders.sitemap -> scrapy.utils.defer -> scrapy.utils.iterators -> scrapy.utils.spider
god_class 4
a class that owns too much
scrapy.core.downloader.Downloader34 methods — likely does too muchscrapy.core.engine.ExecutionEngine40 methods — likely does too muchscrapy.http.headers.Headers26 methods — likely does too muchscrapy.settings.BaseSettings38 methods — likely does too much
wide_signature 24
functions that take so many arguments the call sites are unreadable
scrapy.core._http2.stream.Stream.__init__6 parameters — a call site says nothing about them; group them into one object or split the function - reachable from 73 call sites (extras/qps-bench-server.py:12, scrapy/_logging/__init__.py:70, scrapy/commands/__init__.py:330)scrapy.core.downloader.handlers.http11._TunnelProtocol.__init__8 parameters — a call site says nothing about them; group them into one object or split the function - reachable from 73 call sites (extras/qps-bench-server.py:12, scrapy/_logging/__init__.py:70, scrapy/commands/__init__.py:330)scrapy.core.downloader.handlers.http11._TunnelFactory.__init__8 parameters — a call site says nothing about them; group them into one object or split the function - reachable from 73 call sites (extras/qps-bench-server.py:12, scrapy/_logging/__init__.py:70, scrapy/commands/__init__.py:330)scrapy.core.downloader.handlers.http11._TunnelEndpoint.__init__7 parameters — a call site says nothing about them; group them into one object or split the function - reachable from 73 call sites (extras/qps-bench-server.py:12, scrapy/_logging/__init__.py:70, scrapy/commands/__init__.py:330)scrapy.core.downloader.handlers.http11._BindAddressAgent._requestWithEndpoint7 parameters — a call site says nothing about them; group them into one object or split the function - reachable from 3 call sites (scrapy/core/downloader/handlers/http11.py:750, scrapy/core/downloader/handlers/http11.py:842, scrapy/core/downloader/handlers/http11.py:884)scrapy.core.downloader.handlers.http11._TunnelingAgent._requestWithEndpoint7 parameters — a call site says nothing about them; group them into one object or split the function - reachable from 3 call sites (scrapy/core/downloader/handlers/http11.py:750, scrapy/core/downloader/handlers/http11.py:842, scrapy/core/downloader/handlers/http11.py:884)
and 18 more of the same check.
complex 6
branchy functions, counted from the AST
scrapy.commands.complete._iter_candidatescyclomatic complexity 17scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler._make_requestcyclomatic complexity 17scrapy.core.downloader.handlers._base_streaming.BaseStreamingDownloadHandler._read_responsecyclomatic complexity 18scrapy.core.downloader.handlers.ftp.FTPDownloadHandler.download_requestcyclomatic complexity 16scrapy.core.engine.ExecutionEngine.close_spider_asynccyclomatic complexity 25scrapy.selector.unified.Selector.__init__cyclomatic complexity 17
hub 13
a module everything depends on
scrapy.settings.BaseSettings.getboolcalled from 63 places — change here ripples widelyscrapy.settings.BaseSettings.getintcalled from 34 places — change here ripples widelyscrapy.settings.BaseSettings.getfloatcalled from 25 places — change here ripples widelyscrapy.settings.BaseSettings.getprioritycalled from 16 places — change here ripples widelyscrapy.signalmanager.SignalManager.send_catch_logcalled from 15 places — change here ripples widelyscrapy.utils.asyncio.is_asyncio_availablecalled from 21 places — change here ripples widely
and 7 more of the same check.
coupling 4
modules that know too much about each other
scrapy.core.enginedepends on 20 project modulesscrapy.core.scraperdepends on 18 project modulesscrapy.crawlerdepends on 21 project modulesscrapy.pipelines.filesdepends on 19 project modules
suspicious_host 1
a hard-coded host or address
extras.github-release-notessends data to docs.scrapy.org - not a host this project's manifest would explain; review what it uploads
unused 88
never called, never imported, never read
conftest.pytest_addoptionnever referenced inside the project (dead code or public API)conftest.pytest_configurenever referenced inside the project (dead code or public API)conftest.pytest_collection_modifyitemsnever referenced inside the project (dead code or public API)conftest.pytest_runtest_setupnever referenced inside the project (dead code or public API)extras.qps-bench-server.Root.getChildnever referenced inside the project (dead code or public API)extras.qps-bench-server.Root.rendernever referenced inside the project (dead code or public API)
and 82 more of the same check.
bad_name 36
a name that does not say what the thing holds
scrapy.commands.parse.Command.prepare_request.<locals>.callbacknames say nothing (cb, d) — rename to what they holdscrapy.core._http2.agent.H2Agent.requestnames say nothing (d, d2) — rename to what they holdscrapy.core.downloader.handlers.http11._ScrapyAgent.download_requestnames say nothing (d, d2, d3) — rename to what they holdscrapy.core.engine.ExecutionEngine._start_scheduled_requestnames say nothing (d, d2) — rename to what they holdscrapy.core.engine.ExecutionEngine._spider_idlenames say nothing (ex, res) — rename to what they holdscrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware.robot_parser.<locals>.cbnames say nothing (cb) — rename to what they hold
and 30 more of the same check.
no_contract 7
a function whose docstring promises nothing
conftest.pytest_addoptionno docstring and no type hints — what it takes and returns is only visible by reading the bodyconftest.pytest_configureno docstring and no type hints — what it takes and returns is only visible by reading the bodyconftest.pytest_collection_modifyitemsno docstring and no type hints — what it takes and returns is only visible by reading the bodyconftest.pytest_runtest_setupno docstring and no type hints — what it takes and returns is only visible by reading the bodyextras.qps-bench-server.Root.renderno docstring and no type hints — what it takes and returns is only visible by reading the bodyextras.qpsclient.QPSSpider.from_crawlerno docstring and no type hints — what it takes and returns is only visible by reading the body
and 1 more of the same check.
Five projects free. A .zip or a public GitHub URL.
Other repositories, same treatment
- Textualize/rich 97 findings
- celery/celery 1052 findings
- encode/django-rest-framework 337 findings
- encode/httpx 26 findings
- fastapi/fastapi 163 findings
- pallets/click 85 findings
- pallets/flask 45 findings
- psf/requests 47 findings
- pydantic/pydantic 164 findings