Sounds like an advert for an expensive product (proxycrawl)

Proxycrawl seemed interesting, so I just tried it out. It appears to have problems with redirects, which is something I expect they would have figured out.

To me, it seems proxycrawl is very expensive! If I may ask, can you talk little about your crawl volume and cost?

I’m crawling around 80-120M per month and the price for me fits my needs. But I suggest that you contact them if you have special needs or requirements.

Also you have to consider the amount of work, time and money that you will save by not maintaining your own system to avoid blocks and bans from the websites you are trying to crawl. With them you just call an API endpoint and you don't have to care about all that

Thanks. One follow up questions, in JS heavy pages, how operations like infinite scrolling etc are exposed/executed?

I can't tell, I haven't scraped any page with infinite scrolling yet

ProxyCrawl looks very interesting, I have already made 1000 free requests to Instagram. I will investigate more on it

thanks, I just talked to their support and got onboard very quickly, seems to work good for LinkedIn but only if support activates the token for you.

yes I also had to contact them in the past to activate linkedin scraping. Now it's perfect without blocks :)

how long have you been using ProxyCrawl? I have rate limit to linkedIn and I want to buy a bigger package, do you recommend?

I've been using it for around 3-4 months with different sites. For linkedin it's been a bit more than 2 months. They are a good startup and they've been improving a lot their services. They only count successful requests so you don't have to worry about fails. If you get a bigger package they will raise your limits I guess. But I suggest that you contact them directly

check what I've just answered to @altareq :)