Managed data workflows
Proxy infrastructure for repeatable scraping
Keep crawlers, schedulers, and pipelines in your environment. Use Premip as the access layer across sites, markets, and collection windows.
Who this is for
Scraping and data-platform teams who already own extraction code and need reliable proxy controls rather than an outsourced scraper.
Operate jobs with explicit access settings
Choose product, location, and session behavior per target while extraction, parsing, and storage stay in your stack.
Why teams use this setup
Less operational guesswork
Document proxy type and location next to each spider.
Per-target routing
Mobile, residential, or datacenter depending on the site.
Scale on your schedule
Add locations and concurrency after a spider is stable.
Infrastructure controls
Workload-specific routing
Match product and location to each spider’s target.
Rotation control
Dashboard rotation changes the exit IP; credentials stay put.
Bring your crawler
Scrapy, custom workers, and commercial crawlers that support HTTP/SOCKS5.
Scraping programs
Scheduled catalogs
Recurring listing and detail crawls.
Regional content
The same spider, different dashboard locations.
Monitoring programs
Long-running jobs with clear session rules.
Recommended setup
Match the target
Default to residential or mobile when blocks are common. Use datacenter when the target is stable and cost-efficient access is enough.
Sessions
Sticky for detail-page walks on one site. Rotate between jobs or batches when you need a new IP.
Location
Set location in the dashboard before launch. Pass the market name into job metadata.
Before you start
- Target sites, volume, and required countries.
- A crawler that can set an HTTP or SOCKS5 proxy.
- An active Premip plan.
- Metrics for success vs block pages.
How to put a spider on Premip
Ship a single spider to production on one location, then templatize.
Step 1
Assess the workload
Record allowed paths, expected volume, session needs, and whether pages are geo-specific.
Step 2
Select infrastructure
Create the proxy product that fits the target. Complete signup so credentials exist.
Step 3
Set location
Choose the market in the dashboard. For multi-country programs, one job config per location is easier to debug.
Step 4
Inject credentials
Configure the crawler’s proxy middleware with USERNAME, PASSWORD, HOST, and PORT from environment variables.
Step 5
Run a smoke crawl
Fetch a handful of URLs. Confirm status codes, encoding, and that bodies are not block pages.
Step 6
Decide sticky vs rotate
Enable stickiness for pagination and carts. After a batch, rotate from the dashboard if the next batch should use a new IP.
Step 7
Add backoff and alerts
Pause on rising error rates. Do not rotate mid-request; finish or fail the request, then rotate.
Step 8
Measure and expand
Promote the spider, then add locations and rate. Keep parsers versioned with sample HTML per market.
curl -x http://USERNAME:PASSWORD@HOST:PORT https://example.comScraping infrastructure questions
- Does Premip provide the scraping code?
- No. Premip provides proxy infrastructure. You connect your scraper or collection platform through standard proxy access.
- Can I discuss a larger deployment?
- Yes. Enterprise teams can contact sales about scale, targeting, and support.
- HTTP or SOCKS5?
- Use HTTP unless the crawler requires SOCKS5. Credentials are shared across both.
- Will rotation change my username?
- No. Host, port, username, and password stay the same. Only the exit IP changes.
Message us
Reach support on Telegram, email, phone, or the contact form. Include your account email and, if you have one, your order ID.