selenium - Update response.body in scrapy(without reload) -
i use scrapy , selenium crawl! site use ajax pagination! actully , url no change , response.body no change! want click selenium (for pagination) , self.driver.page_source , use instead response.body! writed code :
res = scrapy.http.textresponse(url=self.driver.current_url, body=self.driver.page_source, encoding='utf-8') print(str(res)) //nothing print! quote in res.css("#ctl00_contentplaceholder1_grd_dr_dxmaintable > tr.dxgvdatarow_office2003blue"): = i+1 item = dict() item['id'] = int(quote.css("td.dxgv:nth-child(1)::text").extract_first())
and no error !
you can replace body of original response in scrapy using response.replace()
method:
def parse(self, response): response = response.replace(body=driver.page_source)
Comments
Post a Comment