alt Hacker NewsIt's a fairly trivial inconvenience. You can just add something to the effect of the below code, and you'll not get stuck and realistically not skip over crawling anything of value.
if (response_time > 8 seconds && response_payload < 2048 bytes) {
extract_links = false;
}