How to Rotate Proxies with Python – adsmember

Great video.
I think 3 things are REALLY worthy of note because the answer to the failures isn't necessarily that the proxies were blocked:
– Sometimes the proxies weren't able to be connected to in the first place
– Sometimes the Server can't handle so many back to back requests

AND
Some proxies are set up by malicious actors to gain unauthorized access to connecting hosts.

What if i want to send a POST instead of get, how can i use proxies with post ?

thank you

Is secure to use a free proxy page? In terms of cybersecurity I mean. Thank you

Hi John, great video and thank you for your time and effort for creating these videos for us. I was wondering if you added the updated version of this video as you mentioned because I could not find any other tutorial on proxy on your site.

How to rotate proxy and rotate payload login use selenium or use faster than selenium with cookies enable, sir give me sample pls 😥

Hey bro, how are u?
I'm looking to buy a rotate proxy ip, would you know where you have it?

Congrats John, it amazing video 🙂
Regards, Nelson

Every time I have a python scraping problem I search your name
Great channel mate 👍

how to use proxy and open 5 chrome browser at same time with different proxy and give individual task to them?

It's really helping material. Thanks dear professor

thanks for your video. so is it possible to rotate headers?

Thanks John for yet another useful video – I'm new to web scraping & have been blocked from a site I want to scrape, I was wondering & Im sure there are packages out there to save the full content of a website locally so we can scrape with no issues & Im not talking about big sites such as Amazon – do you think this is possible if so why no one else is talking about it? how would you go about it please?

Can you do a video on this but with scrapy?

Mine keeps failing… any ideas why? Good video!

Hey, add this if you want to scrape data without getting caught in google

.

sleep = random.randint(3,9)

time.sleep(sleep)

.

a little slow but when you send random requests, google thinks you are human 🙂

thank you very much

I couldn't find any working proxies on that list. So, I created my own proxy pool, hahahah

Great Video. I tried to use a proxy available online and returned back with a 200 status code. But if then try to print the text (page.text), I get a nonetype object. Can you help me why this would be a case

i try your code, the problem is 2:59, the response is 200, when i use print(r.json) there is error so go to except, but without json, proxylist show working, please tell me why print(r) and print(r.json) are different result

you said that the proxies will not work for Google but do you think they will work for other GCP products?

That was what I exactly looking for

Thanks; very helpful and unique tutorial; more like this, please!👍

Hi John ,why result of crapeProxy is no port ?

Tks John

Sir, I saw a video identical at yours from Indian phythonista. Pls ck it out!!!! Thx

great explanation. How can I used this code to rotate proxies in my existing file of scraped data?

Aahh man, coding is not much of a big deal to test proxies for me. I was looking for a solution if anyone can help me with something please reply. I built an api that return basic insta info like hd_dp, follower_count, caption, bio etc but the problem now is it is working from my ip but when I hosted it on a vps the robot page popped. I guess insta has a list of vps companies ip or previously someone used this ip too much.

Just a question. What keyboard are you using. The clickity mechanical sounds are soothing

Hey John! new subscriber here..! Im enjoying your channel very much, I have one suggestion though, in most of your videos you refer to previous ones and say that you're going to post the links somewhere but you don't. As a newcomer it is a bit difficult to find the video you're referring to since well your thumbnails and titles are in general, similar. Links will help new subscribers drive through your content smoothly. Cheers!

This video was really helpful. Thank you for your videos!

really informative video…. but is it possible to use proxy for python program or module?….
i mean, can i use proxy for smtplib python module etc?….. sir, if you have any solution or reference please tell me…..

Hey John,

I am a Uni student studying Data Analytics. Currently doing a unit on "Data Acquisition" and your videos are far better walking through the complexities of web scraping than this current course!

I'm doing enough web scraping now where I think it is beneficial for me to start looking at paid for rotating residential proxies.

Do you have a service that you recommend? Even if you have affiliate links.

If you don't have any links, I think it would be beneficial to seek out such sponsorship possibilities soon.

thanks for your sharing ,how can i come out the proxy:port????thank you again

Hello John, I am trying to implement this code with a list of proxies that already work in a csv file and the code runs without any errors but does not give me an outcome whatsoever. I believe my issue is originating from the extract function and I was hoping you could lend me a hand if that is possible. I am looking forward to the sequel of this video you said you would make so I can further understand. Thank you

fully watch this video twice, but honestly in the video no coding is about "how to rotate proxies', can you make video truly about 'rotate'?

Well, unfortunately I see first your video of web scraping and I just get blocked big time, that's why I am here now lol

Send to me file proxy csv bro

Thanks to the man who has all the answers to my questions.
Man, you have the well of wisdom on behalve of scraping/python, JSON and what matters in that field.
Although I am a pro IT guy for many many years (PM, Consultant, Architect and Advisor) this field of expertise is rather unexplored for me,
but following your video's made it crystal clear to me.
Thanks again for sharing

I'm currently scraping Facebook with Selenium for my final project. (I can't use the API for many reasons and I can't change the source as my project depends solely on facebook: if you're going to say it's illegal)
I switch user-agents but should I use proxy too? I get blocked quite often and I'm fairly new to this.

Genius

Thanks for this video. I'm using requests_html for my scrapper, do you know what is the equivalent of (print(r.json)) ? I'd like to be sure that the scrapper is using the right proxy. Thank you!

What is the process for socks5 proxy

Hello even request.get response value 200 for a url and it looks like proxy is working
but when we load a this website etc then it's always show can't access, load timeout,rendering timeout..etc. So do we have any way to check those proxy could work as normal? Thank you so much

Great video! How would I go about getting the equivalent of a r.json response (What IP used is what I want to know) when targeting a URL like Google for example, where the .json will not work?

For some reason this script just returns my ip address not the proxy address?

I had the below error, and solved it by going into the documentation and used the example under proxies to setup the proxies. Maybe the requests library changed a bit since.

"requests.exceptions.InvalidURL: Proxy URL had no scheme"

Thank you

Can i use this method with residential proxies or datacenter?