Search results

Can this be used as a validator?

Status: Open.

#19 In chrisakroyd/robots-txt-parser;

· stevenvachon opened on Sep 23, 2025

sitemaps links are not returned for some websites (ex: https://www.sainsburys.co.uk)

Status: Open.

#17 In chrisakroyd/robots-txt-parser;

· kvamsij opened on Aug 15, 2024

canCrawl stalls in nodejs but not browser

Status: Open.

#16 In chrisakroyd/robots-txt-parser;

· salivity opened on Jun 16, 2024

canCrawl() returns incorrect value for middle paths

Status: Open.

#15 In chrisakroyd/robots-txt-parser;

· ssaumya1711 opened on Jan 14, 2024

Valid URL results in error

Status: Open.

#12 In chrisakroyd/robots-txt-parser;

· dskvr opened on Feb 19, 2023

canCrawl() returns incorrect result when matching middle of path

Status: Open.

#8 In chrisakroyd/robots-txt-parser;

· Trott opened on Feb 1, 2023

Bug (in <code>useRobotsFor()</code>?) causes <code>canCrawl()</code> to sometimes return incorrect result

Status: Open.

#5 In chrisakroyd/robots-txt-parser;

· Trott opened on Nov 5, 2021