The Dysdera Web Crawler is a asynchronous web crawler written in Python. Designed for extensibility and adaptability, it provides fine-grained control over crawling policies.
- Python version >= 3.9
- MongoDB for saving data (Community edition is free) or in alternative a Json file
- some python packages: to be installed with pip with the comand: 'pip install -r requirements.txt'
- motor to interact with MongoDB,
- json and aiofiles for saving in Json file
- asyncio for the asynchronous logic,
- aiohttp for http managing,
- lxml for the html parsing,
- brotli for http response compression,
- pytz and python-dateutil for more precise datetime management,
- chardet for encoding detection
You will find the documentation here
-
dysderacrawler.py contains the logic of the crawler,
-
extractors.py contains the logic of the extractors,
-
policy.py contains the structure of the crawler policy,
-
selectionpolicy.py some selection policy
-
web.py the logic for manage webpages and more,
-
parser.py the necessary parser
-
logger.py the logic for the logs