Skip to content

Commit ced1e8c

Browse files
authored
updated current feratures
1 parent 59f96f0 commit ced1e8c

File tree

1 file changed

+8
-0
lines changed

1 file changed

+8
-0
lines changed

README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,11 @@
22
[![Build Status](https://travis-ci.com/rapidclock/web-octopus.svg?token=hJhLfHtyz41UyuLTTdFx&branch=master)](https://travis-ci.com/rapidclock/web-octopus)
33
<br>
44
A concurent web crawler to crawl the web.
5+
6+
### Current Features:
7+
- Depth Limited Crawling
8+
- User specified valid protocols
9+
- User buildable adapters that the crawler feeds output to.
10+
- Filter Duplicates.
11+
- Filter URLs that fail a HEAD request.
12+
- User specifiable max timeout between two successive url requests.

0 commit comments

Comments
 (0)