We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
There was an error while loading. Please reload this page.
1 parent 59f96f0 commit ced1e8cCopy full SHA for ced1e8c
README.md
@@ -2,3 +2,11 @@
2
[](https://travis-ci.com/rapidclock/web-octopus)
3
<br>
4
A concurent web crawler to crawl the web.
5
+
6
+### Current Features:
7
+- Depth Limited Crawling
8
+- User specified valid protocols
9
+- User buildable adapters that the crawler feeds output to.
10
+- Filter Duplicates.
11
+- Filter URLs that fail a HEAD request.
12
+- User specifiable max timeout between two successive url requests.
0 commit comments