Skip to content

canCrawl stalls in nodejs but not browser #16

@salivity

Description

@salivity

I created a project at https://davidclews.com/article/158.html and the robots.txt for https://www.homebase.co.uk/robots.txt appears a parse correctly but in node 20 when running canCrawl on a url the function stalls with no exception. What could this be?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions