aiopipe¶
Utilities for Asynchronous Processing
Classes
|
Function object type called by Scanner |
|
Processes items in an asyncio pipeline |
|
Provides helpers for async access to URLs |
Exceptions
Raised by |
|
|
Raised to indicate that an item should not be processed further |
Documentation
- exception bioconda_utils.aiopipe.EndProcessing[source]¶
Raised by
AsyncFilter
to tellAsyncPipeline
to stop processing
- exception bioconda_utils.aiopipe.EndProcessingItem(item, *args)[source]¶
Raised to indicate that an item should not be processed further
- log(uselogger=<Logger bioconda_utils.aiopipe (INFO)>, level=None)[source]¶
Print message using provided logging func
- property name¶
Name of class
- class bioconda_utils.aiopipe.AsyncFilter(pipeline, *_args, **_kwargs)[source]¶
Function object type called by Scanner
- class bioconda_utils.aiopipe.AsyncPipeline(threads=None)[source]¶
Processes items in an asyncio pipeline
- loop¶
our asyncio loop
- threads¶
number of threads to use
- io_sem: Semaphore¶
semaphore to limit io parallelism
- conda_sem: Semaphore¶
must never run more than one conda at the same time (used by PyPi when running skeleton)
- filters: List[AsyncFilter]¶
the filters successively applied to each item
- proc_pool_executor¶
executor running things in separate python processes
- class bioconda_utils.aiopipe.AsyncRequests(cache_fn=None)[source]¶
Provides helpers for async access to URLs
- USER_AGENT = 'bioconda/bioconda-utils'¶
Used as user agent in http requests and as requester in github API requests
- session: ClientSession¶
aiohttp session (only exists while running)
- get_text_from_url(url)[source]¶
Fetch content at url and return as text
On non-permanent errors (429, 502, 503, 504), the GET is retried 10 times with increasing wait times according to fibonacci series.
Permanent errors raise a ClientResponseError
- Return type:
- async get_checksum_from_url(url, desc)[source]¶
Compute sha256 checksum of content at url
Shows TQDM progress monitor with label desc.
Caches result
- Return type:
- get_checksum_from_http(url, desc)[source]¶
Compute sha256 checksum of content at http url
Shows TQDM progress monitor with label desc.
- Return type: