Creating Rotating Mobile Proxies from VPS and 4G SIM: Unlimited Data Scraping Solution
A detailed guide on building rotating mobile proxies using VPS and 4G SIM cards to enable unlimited data scraping, protect privacy, and avoid detection. Includes hardware setup,...
Introduction to Rotating Mobile Proxies
Rotating mobile proxies offer a powerful solution for unlimited data scraping from websites without being detected or blocked. Unlike static IP addresses or standard proxies, rotating mobile proxies automatically change the IP address with each request, mimicking real user behavior on 4G mobile networks.
This approach is especially useful when collecting data from platforms such as e-commerce sites, news portals, or public datasets that employ anti-scraping protections. By using a VPS (Virtual Private Server) and 4G SIM cards, you can build a custom mobile proxy system that gives you full control and cost savings compared to commercial proxy services.
Hardware Requirements and Setup
Necessary Devices
- High-RAM VPS with at least 8GB RAM (recommended 16GB) to handle multiple concurrent connections.
- 4G SIM card from a reliable carrier (e.g., Viettel, Vinaphone, Mobifone) with unlimited or high-capacity data plans.
- USB 4G LTE modem compatible with Linux (e.g., Huawei E3372, ZTE MF190).
- Router or small server to connect the modem and VPS (optional, if direct connection isn't possible).
Software Setup on VPS
On the VPS, install the following tools:
- Ubuntu 22.04 LTS or equivalent.
- To manage SIM connectivity, install
usb_modeswitch,ppp, andNetworkManager. - To create the proxy, use
PrivoxyorSquid.
Setting Up 4G SIM Connection on VPS
First, connect the USB 4G modem to the VPS. Then, verify the device is recognized:
lsusb
If the device appears (e.g., Huawei E3372), proceed with software installation:
sudo apt update
sudo apt install usb_modeswitch ppp network-manager
Check the connection status:
sudo nmcli device
Ensure the modem appears in the list and is recognized as a mobile connection. If not, verify drivers or try a different device.
Creating Rotating Proxy with Privoxy
Once the SIM connection is stable, configure Privoxy to create a proxy and automate IP rotation.
Edit the configuration file:
sudo nano /etc/privoxy/config
Add the following lines:
forward-socks5t / 127.0.0.1:9050 .
listen-address 127.0.0.1:8118
Restart Privoxy:
sudo systemctl restart privoxy
To automate IP changes, write a simple script that runs ppp restart or nmcli connection down/up every 5–10 minutes.
Verification and Optimization
Verify Connection
Run the following command to check the current IP:
curl ifconfig.me
The IP should change after each connection restart. If it doesn’t, recheck the SIM, modem, or NetworkManager configuration.
Test Scraping
Run a simple Python script to test the proxy:
import requests
proxies = {
'http': 'http://127.0.0.1:8118',
'https': 'http://127.0.0.1:8118'
}
response = requests.get('https://httpbin.org/ip', proxies=proxies)
print(response.json())
Alternative Solutions and Notes
To avoid legal risks, ensure you comply with the terms of service of the websites you are scraping. You may also consider these alternatives:
- High-RAM Cloud VPS to self-host scraping tools.
- Dedicated Server for higher performance needs.
- Configuring Unbound to enhance security and privacy.
This solution is ideal for enterprises, developers, or individuals who want full control over their scraping infrastructure and avoid dependency on third-party services.
High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache
Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.
VPS-4E
10% VAT included
- CPU: 4 vCPU
- RAM: 8 GB
- SSD: 50 GB NVMe
