计算机网络实用工具之parsero

简介

Parsero是一个用Python编写的免费脚本，它读取web服务器的robots.txt文件，探测“Disallow”的条目并返回响应状态码。

例：

200 OK          The request has succeeded.
403 Forbidden   The server understood the request, but is refusing to fulfill it.
404 Not Found   The server hasn't found anything matching the Request-URI.
302 Found       The requested resource resides temporarily under a different URI.
...

官网

GitHub - behindthefirewalls/Parsero: Parsero | Robots.txt audit tool

安装

# ubuntu 20.04
sudo apt install parsero

使用帮助

root@ubuntu:~# parsero -h
usage: parsero [-h] [-u URL] [-o] [-sb] [-f FILE]optional arguments:-h, --help  show this help message and exit-u URL      Type the URL which will be analyzed-o          Show only the "HTTP 200" status code-sb         Search in Bing indexed Disallows-f FILE     Scan a list of domains from a list

使用示例

# 查看百度声明禁止访问哪些链接
root@ubuntu:~# curl www.baidu.com/robots.txt
User-agent: Baiduspider
Disallow: /baidu
Disallow: /s?
Disallow: /ulink?
Disallow: /link?
Disallow: /home/news/data/
Disallow: /bhUser-agent: Googlebot
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?# parsero针对禁止访问的链接进行探测
root@ubuntu:~# parsero -u www.baidu.com____                               |  _ \ __ _ _ __ ___  ___ _ __ ___  | |_) / _` | '__/ __|/ _ \ '__/ _ \ |  __/ (_| | |  \__ \  __/ | | (_) ||_|   \__,_|_|  |___/\___|_|  \___/ Starting Parsero v0.81 (https://github.com/behindthefirewalls/Parsero) at 12/19/23 13:31:55
Parsero scan report for www.baidu.com
http://www.baidu.com/ 200 OK
http://www.baidu.com/homepage/ 302 Found
http://www.baidu.com/baidu 302 Found
http://www.baidu.com/s? 302 Found
http://www.baidu.com/ulink? 404 Not Found
http://www.baidu.com/cpro 404 Not Found
http://www.baidu.com/bh 302 Found
http://www.baidu.com/link? 404 Not Found
http://www.baidu.com/shifen/ 200 OK
http://www.baidu.com/home/news/data/ 302 Found[+] 10 links have been analyzed and 2 of them are available!!!Finished in 0.45 seconds.