Add support for --robots crawler flag to Browsertrix#3029

tw4l wants to merge 1 commit intomainwebrecorder/browsertrix:mainfrom

issue-2935-robotswebrecorder/browsertrix:issue-2935-robotsCopy head branch name to clipboard