Usage¶
proxy-db uses free proxy pages to feed a local proxy database. By default the database is created in
~/.local/var/lib/proxy-db/db.sqlite3
, although you can change the location and use other db engines.
The proxies score is stored in this database. To increase the score of a proxy (the proxy works) use the
proxy.positive()
method, and use proxy.negative()
if it fails.
>>> from proxy_db.proxies import ProxiesList
>>> p = next(ProxiesList())
>>> p
<Proxy http://5.0.0.0:8080>
>>> p.positive() # Increase score
>>> p.votes # current score
1
>>> str(p) # return proxy string
'http://5.0.0.0:8080'
>>> p.country # proxy country
'ES'
By default ProxiesList will return proxies ordered by number of votes.
Use with requests¶
proxy-db is very easy to use with requests. In this example it vote positive if it works and negative if the proxy fails.
import requests
from requests.exceptions import Timeout, ConnectionError, ProxyError
from proxy_db.proxies import ProxiesList
proxy = next(ProxiesList())
try:
requests.get('http://site.com/', proxies=proxy)
except (Timeout, ConnectionError, ProxyError):
proxy.negative()
else:
proxy.positive()
In this example test proxies to find a proxy that works:
import requests
from requests.exceptions import Timeout, ConnectionError, ProxyError
from proxy_db.proxies import ProxiesList
resp = None
for proxy in ProxiesList():
try:
resp = requests.get('http://site.com/', proxies=proxy)
except (Timeout, ConnectionError, ProxyError):
proxy.negative()
else:
proxy.positive()
break
if resp is not None:
print(f'Response: {resp.text}')
else:
print('Could not get response')
Countries¶
To force the country of the proxies use the country code in ProxiesList
:
>>> from proxy_db.proxies import ProxiesList
>>> p = next(ProxiesList('es'))
>>> p.country
'ES'
Countries use the 2-character iso code.
Change list strategy¶
By default proxy-db sorts proxies by votes and only returns those with more than 1 vote (ignore with negative votes). There are other strategies available like returning random proxies:
>>> from proxy_db.proxies import ProxiesList, RandomListingStrategy
>>> p = next(ProxiesList(strategy=RandomListingStrategy))
It is also possible to change the default strategy options:
>>> from proxy_db.proxies import ProxiesList, VotesListingStrategy
>>> p = next(ProxiesList(strategy=VotesListingStrategy(min_votes=-5)))
Change database¶
To change the path to the sqlite database define the environment variable PROXY_DB_FILE
, by default
~/.local/var/lib/proxy-db/db.sqlite3
. The environment variable PROXY_DB_DB_URL
changes the
database configuration, by default sqlite:///{PROXY_DB_FILE}
.
proxy-db uses sqlalchemy. For more information about how to configure the url to the database, check its documentation.
Add proxies manually¶
You can add one or more proxies per command line to insert them into the database. To add proxies:
$ proxy-db add[ <proxy_1>][ <proxy_2>][ <proxy_n>]
For example:
$ proxy-db add http://5.0.0.0:8080 http://6.0.0.0:8080 http://7.0.0.0:8080
You can also import proxies from a file, with one proxy per line:
$ proxy-db add --file <filename>
For example:
$ proxy-db add --file proxies.txt
You can also send the proxies stdin, one proxy per line:
$ proxy-db add < proxies.txt
By default the proxies are created with the “manual” provider but this can be changed using the
--provider <provider_anem>
parameter. For example:
$ proxy-db add --provider "my_provider" http://5.0.0.0:8080
Added proxies have 10 positive votes by default. These votes will increase with successful requests and
will decrease if the proxy fails. To change the default votes use the --votes <votes>
parameter. For example:
$ proxy-db add --votes 50 http://5.0.0.0:8080
List proxies¶
To list all the proxies already available you can use the following command:
$ proxy-db list
http://185.146.167.226:1080
http://185.89.182.32:86
http://186.28.217.19:80
http://185.116.137.248:1080
http://185.176.129.14:1080
...
By default the proxies will be listed line by line as in the previous example. You can change the format
using --format <format>
. Available options: line, json. More options are available by installing
the tabulate
package using pip install tabulate
. To see all the options after installing
tabulate use proxy-db list --help
.
$ proxy-db list --format json
It is also possible to choose the columns to display. To see the available columns use proxy-db list --help
:
$ proxy-db list --columns <column1[,<column2>]>
For example:
$ proxy-db list --columns id,votes,country,protocol,providers
Proxies can be filtered using various options:
$ proxy list[ --min-votes <votes>][ --country <country code>]
[ --protocol <protocol>][ --provider <provider>]
For example:
$ proxy list --min-votes 10 --country ES \
--protocol https --provider "Nord VPN"
Payment providers¶
Some providers require a payment and additional steps to use.
Nord VPN¶
Login in Nord VPN.
Go to Nord VPN service details.
In advanced configuration copy/create your username and password for Service credentials (manual setup). These credentials are different from the username and password to log into the Nord VPN website.
Set environment variables
PROXYDB_NORDVPN_USERNAME
andPROXYDB_NORDVPN_PASSWORD
in your program.
To set environment variables in Python:
import os
os.environ['PROXYDB_NORDVPN_USERNAME'] = '2dybg3pvxN4XYpLpF2iBE3wz'
os.environ['PROXYDB_NORDVPN_PASSWORD'] = 'hjFq8QkKsnKM42o4Yzta8y2K'
To set environment variables in Bash (before run your program):
$ export PROXYDB_NORDVPN_USERNAME=2dybg3pvxN4XYpLpF2iBE3wz
$ export PROXYDB_NORDVPN_PASSWORD=hjFq8QkKsnKM42o4Yzta8y2K
$ ./your-program.py