dirhunt.sources package

Submodules

dirhunt.sources.base module

class dirhunt.sources.base.Source(result_callback, error_callback, max_workers=None)[source]

Bases: dirhunt.pool.Pool

add_domain(domain)[source]
add_error(message)[source]
add_result(url)[source]
callback(domain)[source]

dirhunt.sources.robots module

class dirhunt.sources.robots.DirhuntRobotFileParser(url='')[source]

Bases: urllib.robotparser.RobotFileParser

read()[source]

Reads the robots.txt URL and feeds it to the parser.

class dirhunt.sources.robots.Robots(result_callback, error_callback, max_workers=None)[source]

Bases: dirhunt.sources.base.Source

callback(domain, protocol='http')[source]
dirhunt.sources.robots.get_url(protocol, domain, path)[source]

Module contents

class dirhunt.sources.Sources(callback, error_callback, excluded_sources=())[source]

Bases: object

add_domain(domain)[source]
finished()[source]
dirhunt.sources.get_source_name(cls)[source]