Following, you need to make a decision how you need the crawler to take care of a variety of World-wide-web systems. There's an ongoing discussion about the intelligence of internet search engine crawlers. It's not completely clear If they're full-blown headless browsers or just glorified curl scripts (or a https://www.youtube.com/watch?v=EVeTP0aV8gY