Welcome, Guest: Register On Nairaland / LOGIN! / Trending / Recent / New
Stats: 3,158,581 members, 7,837,192 topics. Date: Wednesday, 22 May 2024 at 06:37 PM

What Can Cause A 403 Error While Scraping - Programming - Nairaland

Nairaland Forum / Science/Technology / Programming / What Can Cause A 403 Error While Scraping (368 Views)

I'm Getting An Error While I Run This Python Codes / Website Showing Error 403 After Unzipping File To File Manager / What Is The Meaning Of This:403 Forbidden And How Do I Resolve It (2) (3) (4)

(1) (Reply) (Go Down)

What Can Cause A 403 Error While Scraping by Kaka5(m): 11:40am On Jan 26, 2023
Good day everyone please what can cause a 403 error while scraping even after changing the user agent?

How does a website detect bots that are scraping their site?
Re: What Can Cause A 403 Error While Scraping by DataMiner(f): 12:06pm On Jan 26, 2023
Why not use proxy rotation
Re: What Can Cause A 403 Error While Scraping by Kvngfrosh(m): 12:52pm On Jan 26, 2023
Kaka5:
Good day everyone please what can cause a 403 error while scraping even after changing the user agent?

How does a website detect bots that are scraping their site?
HttpStatus: 403 {Not Authorized.}
Re: What Can Cause A 403 Error While Scraping by Kaka5(m): 1:36pm On Jan 26, 2023
DataMiner:
Why not use proxy rotation

I'm just practicing I don't really know much about proxy rotation but I guess it changes your ip but it gave me 403 on first request to the site. Thanks for your input.
Re: What Can Cause A 403 Error While Scraping by spartan117(m): 8:53am On Jan 28, 2023
Kaka5:


I'm just practicing I don't really know much about proxy rotation but I guess it changes your ip but it gave me 403 on first request to the site. Thanks for your input.
403 error means you are not authorized to view the site. Since you said it's the first request it may be that the site automatically bans IP addresses from your country.

Tips on debugging:
- Try to scrap another site maybe google.com to
confirm your code actually works.
- Change your IP using a VPN and see if it works on
the site you want to scrap.
- If it does consider implementing proxy rotation
using proxy addresses for countries that are
allowed on that site.
Re: What Can Cause A 403 Error While Scraping by Kaka5(m): 4:29pm On Jan 28, 2023
spartan117:

403 error means you are not authorized to view the site. Since you said it's the first request it may be that the site automatically bans IP addresses from your country.

Tips on debugging:
- Try to scrap another site maybe google.com to
confirm your code actually works.
- Change your IP using a VPN and see if it works on
the site you want to scrap.
- If it does consider implementing proxy rotation
using proxy addresses for countries that are
allowed on that site.

Thanks for your input. I've tried most of the things you said above. The site I'm trying to scrape is https://indeed.com. Can you make a request on your end and tell me what you got??

(1) (Reply)

Artificial Super Intelligence ASI Is Coming Soon / I Need Someone To Develop A Game App / Android Emulator Hypervisor Driver Keeps Failing To Install On My Andriod Studio

(Go Up)

Sections: politics (1) business autos (1) jobs (1) career education (1) romance computers phones travel sports fashion health
religion celebs tv-movies music-radio literature webmasters programming techmarket

Links: (1) (2) (3) (4) (5) (6) (7) (8) (9) (10)

Nairaland - Copyright © 2005 - 2024 Oluwaseun Osewa. All rights reserved. See How To Advertise. 10
Disclaimer: Every Nairaland member is solely responsible for anything that he/she posts or uploads on Nairaland.