Index

A

AbstractPolicyModule (built-in class) active (Task attribute) add_content() (BerkeleyContentDB method) add_link() (BaseLinkDB method) (GraphAndBTreeDB method) (GraphDB method) add_links() (TaskServer method) add_task() (built-in function), [1] added_records_num() (BerkeleyContentDB method) address (Command attribute) (Crawler attribute) (TaskServer attribute) alive (built-in class), [1] api_docs_declaration() (built-in function)

api_docs_resources() (built-in function) api_keys() (built-in function) app (Server attribute) (WebServer attribute) assign_crawlers() (TaskServer method) assign_crawlers_to_servers() (Command method) assign_speed() (TaskServer method) AutoLogout (built-in class) autoscale() (Command method) autoscale_change (Task attribute) AUTOSCALING_PERIOD (built-in variable)

B

BaseCrawlingDepthPolicy (built-in class) BaseLinkDB (built-in class) BerkeleyContentDB (built-in class)

blacklist (CreateTaskForm attribute) (Task attribute) (TaskServer attribute) browser (Crawler attribute)

C

calculate_depth() (BaseCrawlingDepthPolicy static method) (IgnoreDepthPolicy static method) (RealDepthCrawlingDepthPolicy static method) (SimpleCrawlingDepthPolicy static method) calculate_priority() (AbstractPolicyModule static method) (SimplePolicyModule static method) change_link_priority() (BaseLinkDB method) (GraphAndBTreeDB method) (GraphDB method) change_priority() (Task method) changed (Command attribute) ChangePasswordForm (built-in class) CHECK_EFFICIENCY_PERIOD (built-in variable) check_server_assignment() (Command method) check_tasks_state() (Command method) clean() (Task method) clear() (BaseLinkDB method) (BerkeleyContentDB method) (GraphAndBTreeDB method) (GraphDB method) close() (BaseLinkDB method) (GraphAndBTreeDB method) CLOSING (CrawlerState attribute) Command (built-in class)

contains() (MimeContentType method) content_db (BerkeleyContentDB attribute) (TaskServer attribute) crawled_links (Crawler attribute) (TaskServer attribute) Crawler (built-in class), [1] crawler_port (Command attribute) crawlers (built-in class) (TaskServer attribute) CrawlerState (built-in class) create_api_keys() (built-in function) create_superuser() (UserManager method) create_task() (TaskManager static method) create_user() (UserManager method) created (Task attribute) CreateTaskForm (built-in class) CURRENT_PATH (built-in variable)

D

date (MailSent attribute) DATE_FORMAT (built-in variable) DEFAULT_LINK_QUEUE_SIZE (built-in variable)

DEFAULT_PRIORITY (SimplePolicyModule attribute) delete() (TaskServer method) delete_task() (built-in function)

E

EditTaskForm (built-in class) EFFICIENCY_THRESHOLD (built-in variable)

expire (CreateTaskForm attribute) expire_date (Task attribute) (TaskServer attribute)

F

feedback (built-in class) feedback() (GraphAndBTreeDB method) (Task method) (TaskServer method) FEEDBACK_PRIORITY_MAPPING (SimplePolicyModule attribute)

finished (Task attribute) found_links (GraphAndBTreeDB attribute) FOUND_LINKS_DB (GraphAndBTreeDB attribute)

G

generate_key() (AbstractPolicyModule static method) (SimplePolicyModule static method) generate_url_hierarchy() (URLProcessor static method) get_address() (Crawler method) get_connected() (GraphDB method) get_data (built-in class) get_data() (built-in function) (TaskServer method) get_data_from_crawler() (built-in function) get_data_iter (BerkeleyContentDB attribute) get_details() (BaseLinkDB method) (GraphAndBTreeDB method) (GraphDB method) get_feedback_propagation_depth() (AbstractPolicyModule static method) (SimplePolicyModule static method) get_file_with_data_package() (BerkeleyContentDB method)

get_host() (WebServer method) get_idle_crawlers() (TaskServer method) get_link() (BaseLinkDB method) (GraphAndBTreeDB method) get_parsed_blacklist() (Task method) get_parsed_whitelist() (Task method) get_parser() (ParserProvider static method) get_state() (Crawler method) get_stats() (Crawler method) graph (GraphDB attribute) GraphAndBTreeDB (built-in class) GraphDB (built-in class)

H

handle() (Command method)

handle_priority_changes() (Command method)

I

id_iter (BerkeleyContentDB attribute) identical_hosts() (URLProcessor static method) IgnoreDepthPolicy (built-in class) index (built-in class), [1] index() (built-in function) INIT (Status attribute)

INIT_CRAWLER_PORT (built-in variable) INIT_SERVER_PORT (built-in variable) is_alive() (Crawler method) (TaskServer method) is_in_base() (BaseLinkDB method) (GraphAndBTreeDB method) (GraphDB method) is_waiting_for_server() (Task method)

K

KEEP_STATS_SECONDS (built-in variable), [1] kill (built-in class), [1]

kill() (Crawler method), [1] (Server method) (TaskServer method), [1] KILLED (Status attribute)

L

last_data_download (Task attribute) last_scaling (Command attribute) last_server_spawn (Task attribute) link_db (TaskServer attribute) link_package_queue (Crawler attribute)

link_pool (Quota attribute) list_tasks() (built-in function) LOOP_PERIOD (built-in variable) LOWER_LOAD_THRESHOLD (built-in variable)

M

MailSent (built-in class) MAX_CRAWLERS_NUM (built-in variable) max_links (CreateTaskForm attribute) (Quota attribute) (Task attribute) (TaskServer attribute) max_priority (Quota attribute) MAX_PRIORITY (SimplePolicyModule attribute) max_tasks (Quota attribute)

mime_type (CreateTaskForm attribute) (Task attribute) (TaskServer attribute) MimeContentType (built-in class) MIN_CRAWLER_STATS_PERIOD (built-in variable) MIN_LINK_PACKAGE_SIZE (built-in variable) MIN_PRIORITY (SimplePolicyModule attribute) MIN_SERVER_STATS_PERIOD (built-in variable)

N

name (CreateTaskForm attribute) (Task attribute)

notify_about_crawler_data() (built-in function)

O

old_crawlers (Command attribute)

one_of() (MimeContentType method)

P

package_cache (TaskServer attribute) package_id (TaskServer attribute) parse() (Parser method) (TextHtmlParser method) Parser (built-in class) ParserProvider (built-in class) parsers (ParserProvider attribute) parts_iter (BerkeleyContentDB attribute) PATH_TO_CRAWLER (built-in variable) PATH_TO_SERVER (built-in variable) pause() (Task method) (TaskServer method) pause_task() (built-in function), [1] PAUSED (Status attribute)

points() (GraphAndBTreeDB method) (GraphDB method) print_tasks() (Command method) priority (CreateTaskForm attribute) (Task attribute) priority_queue (GraphAndBTreeDB attribute) PRIORITY_QUEUE_DB (GraphAndBTreeDB attribute) priority_queue_db_name (GraphAndBTreeDB attribute) process_request() (AutoLogout method) processing_crawlers (TaskServer attribute) put_data (built-in class) put_data() (TaskServer method) put_into_link_queue() (Crawler method) put_links (built-in class)

Q

Quota (built-in class)

QuotaException (built-in class)

R

raise_exc() (ThreadWithExc method) RealDepthCrawlingDepthPolicy (built-in class) register_crawler() (built-in function) register_task_server() (built-in function) remove_old_mail_data() (built-in function)

resume() (Task method) (TaskServer method) resume_task() (built-in function), [1] run() (Crawler method) (Server method) (TaskServer method) (WebServer method) RUNNING (Status attribute)

S

save() (Task method) send() (Crawler method) (TaskServer method) send_feedback() (built-in function) send_update_to_task_server() (Task method) SendFeedbackForm (built-in class) Server (built-in class) server (Task attribute) server_port (Command attribute) SERVER_SPAWN_TIMEOUT (built-in variable) set_as_fetched() (BaseLinkDB method) (GraphAndBTreeDB method) (GraphDB method) show() (BerkeleyContentDB method) show_quota() (built-in function) show_task() (built-in function) sigint_signal_handler() (built-in function) SimpleCrawlingDepthPolicy (built-in class) SimplePolicyModule (built-in class)

size() (BerkeleyContentDB method) (GraphAndBTreeDB method) spawn_crawler() (Command method) spawn_task_server() (Command method) speed (built-in class) start_links (CreateTaskForm attribute) (Task attribute) STARTING (Status attribute) stats (built-in class), [1] STATS_PERIOD (built-in variable) stats_reset_time (Crawler attribute) (TaskServer attribute) Status (built-in class) status (TaskServer attribute) stop (built-in class), [1] stop() (Crawler method), [1] (Task method) (TaskServer method) (WebServer method) stop_task() (built-in function), [1] STOPPING (Status attribute) subtype (MimeContentType attribute)

T

Task (built-in class) TaskFilterForm (built-in class) TaskManager (built-in class) tasks (MailSent attribute)

TaskServer (built-in class), [1] TextHtmlParser (built-in class) ThreadWithExc (built-in class) type (MimeContentType attribute)

U

UNDEFINED (CrawlerState attribute) unregister_crawler() (built-in function) unregister_task_server() (built-in function) update (built-in class) update() (TaskServer method) UPPER_LOAD_THRESHOLD (built-in variable) URL_PACKAGE_TIMEOUT (built-in variable)

URLProcessor (built-in class) urls (Server attribute) (WebServer attribute) urls_per_min (Quota attribute) (TaskServer attribute), [1] User (built-in class) user (Quota attribute) (Task attribute) UserManager (built-in class) uuid (Crawler attribute), [1] (TaskServer attribute), [1]

V

validate() (URLProcessor static method)

W

WAIT_FOR_DOWNLOAD_TIME (built-in variable) WAITING (CrawlerState attribute) WebServer (built-in class)

whitelist (CreateTaskForm attribute) (Task attribute) (TaskServer attribute) WORKING (CrawlerState attribute)