How to overcome distil_r_captcha when webscraping?? | Sololearn: Learn to code for FREE!


How to overcome distil_r_captcha when webscraping??

Hello, I am trying to scrape a website with scrapy (code below) but I keep getting redirected to a captcha page. I get a [200] response, but when I run the scraping I get stopped. Is my IP blacklisted? This is the output I get: DEBUG: Redirecting (meta refresh) to <GET> from <GET> 2019-11-03 02:28:55 [scrapy.core.engine] DEBUG: Crawled (405) <GET> (referer: None) 2019-11-03 02:28:55 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <405>: HTTP status code is not handled or not allowed # -*- coding: utf-8 -*- import scrapy from scrapy.crawler import CrawlerProcess import requests class HousesearchspiderSpider(scrapy.Spider): name = "housesearchspider" download_delay = 10.0 start_urls = [ '', ] def parse(self, response): for detail in response.css(''): yield {'price':detail.css(' search-result-info-price ::text').get(), 'size': detail.css(' ::text').get(), 'postcode': detail.css(' ::text').get(), 'street': detail.css(' ::text').get(), } next_page = response.css(' a::attr(href)').get() if next_page is not None: next_page = response.urljoin(next_page) sleep(5) yiel

11/3/2019 1:29:39 AM

Brunna Villar

2 Answers

New Answer


Well you use a bot to scrape a website and their bot detection triggered so everything is working as intended I'd say. Your IP is probably not blacklisted. But nobody other than google knows for sure how recaptcha works. They track everything you do on the website and then some algorithm figures out if a human would use a browser like that. I've gotten around this before by scraping with headless chrome and filling out the captchas by hand when they come up, before the program takes over. But that's obviously not ideal. There's no point in fighting recaptcha I think. If the website doesn't have an API where you can get the data in a nice way then there's not much you can do. I hear some bots can solve recaptchas but I don't know too much about that. You can also hire indian people to solve captchas for you (no joke) but I'm not sure I'd want to support a company that makes people solve captchas all day long.


I came across this thread from a Google search, and I am seeing the same 'distil_r_captcha.html' issue. As luck would have it, I am also trying to crawl (things we do to beat the AMS housing market, am I right?), but I haven't found a work-around either, and wanted to see if you made any progress since you posted your question. Funda probably blocks bots for their /koop/ URLs, as you can see from their robots.txt that they disallow bots in several of their URLs, with /koop/ being one of them, so I'm starting to think that it may be impossible to use scraping bots for Funda.