Description

A tool written in python that fetches and analyzes the robots.txt file of a website and save the the results to a file for future use.

Usage

python3 /path/to/file

robots.txt Link: https://github.com/kwesilarry1/Robot.txt-Scrapper