Welcome, Guest: Register On Nairaland / LOGIN! / Trending / Recent / NewStats: 3,158,581 members, 7,837,192 topics. Date: Wednesday, 22 May 2024 at 06:37 PM |
Nairaland Forum / Science/Technology / Programming / What Can Cause A 403 Error While Scraping (368 Views)
I'm Getting An Error While I Run This Python Codes / Website Showing Error 403 After Unzipping File To File Manager / What Is The Meaning Of This:403 Forbidden And How Do I Resolve It (2) (3) (4)
What Can Cause A 403 Error While Scraping by Kaka5(m): 11:40am On Jan 26, 2023 |
Good day everyone please what can cause a 403 error while scraping even after changing the user agent? How does a website detect bots that are scraping their site? |
Re: What Can Cause A 403 Error While Scraping by DataMiner(f): 12:06pm On Jan 26, 2023 |
Why not use proxy rotation |
Re: What Can Cause A 403 Error While Scraping by Kvngfrosh(m): 12:52pm On Jan 26, 2023 |
Kaka5:HttpStatus: 403 {Not Authorized.} |
Re: What Can Cause A 403 Error While Scraping by Kaka5(m): 1:36pm On Jan 26, 2023 |
DataMiner: I'm just practicing I don't really know much about proxy rotation but I guess it changes your ip but it gave me 403 on first request to the site. Thanks for your input. |
Re: What Can Cause A 403 Error While Scraping by spartan117(m): 8:53am On Jan 28, 2023 |
Kaka5:403 error means you are not authorized to view the site. Since you said it's the first request it may be that the site automatically bans IP addresses from your country. Tips on debugging: - Try to scrap another site maybe google.com to confirm your code actually works. - Change your IP using a VPN and see if it works on the site you want to scrap. - If it does consider implementing proxy rotation using proxy addresses for countries that are allowed on that site. |
Re: What Can Cause A 403 Error While Scraping by Kaka5(m): 4:29pm On Jan 28, 2023 |
spartan117: Thanks for your input. I've tried most of the things you said above. The site I'm trying to scrape is https://indeed.com. Can you make a request on your end and tell me what you got?? |
(1) (Reply)
Artificial Super Intelligence ASI Is Coming Soon / I Need Someone To Develop A Game App / Android Emulator Hypervisor Driver Keeps Failing To Install On My Andriod Studio
(Go Up)
Sections: politics (1) business autos (1) jobs (1) career education (1) romance computers phones travel sports fashion health religion celebs tv-movies music-radio literature webmasters programming techmarket Links: (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Nairaland - Copyright © 2005 - 2024 Oluwaseun Osewa. All rights reserved. See How To Advertise. 10 |