~/0xWizard
← All projects

$ cat /projects/robots_txt_scrapper

Robots.txt Scrapper Tool

A simple python script that fetches the robots.txt of a site to a file

#Automation#Scripting

Description

A tool written in python that fetches and analyzes the robots.txt file of a website and save the the results to a file for future use.

Usage

python3 /path/to/file

robots.txt Link: https://github.com/kwesilarry1/Robot.txt-Scrapper