Robots.txt Scrapper Tool
Description A tool written in python that fetches and analyzes the robots.txt file of a website and save the the results to a file for future use. Usage python3 /path/to/file Link: https://github.com/kwesilarry1/Robot.txt-Scrapper