hosters¶

Check package URLs for updates

Subclasses of Hoster define how to handle each hoster. Hosters are selected by regex matching each source URL in a recipe. The HTMLHoster provides parsing for hosting sites listing new releases in HTML format (probably covers most). Adding a hoster is as simple as defining a regex to match the existing source URL, a formatting string creating the URL of the relases page and a regex to match links and extract their version.

We need to use regex rather than re to allow recursive matching to manipulate capture groups in URL patterns as needed. (Technically, we could avoid this using a Snakemake wildcard type syntax to define the patterns - implementers welcome).

Functions

`dedup_named_capture_group`(pattern)	Replaces repetitions of capture groups with matches to first instance
`replace_named_capture_group`(pattern, vals)	Replaces capture groups with values from vals

Classes

`Bioarchive`(url, match)	Scans for updates to packages hosted on bioarchive.galaxyproject.org
`Bioconductor`(url, match)	Matches R packages hosted at Bioconductor
`BitBucketBase`(url, match)	Base class for hosting at bitbucket.org
`BitBucketDownload`(url, match)	Uploaded releases hosted at bitbucket.org
`BitBucketTag`(url, match)	Tag based releases hosted at bitbucket.org
`CPAN`(url, match)	Scans for updates to Perl packages hosted on CPAN
`CRAN`(url, match)	R packages hosted on r-project.org (CRAN)
`CargoPort`(url, match)	Matches source backup urls created by cargo-port
`FTPHoster`(url, match)	Scans for updates on FTP servers
`GithubBase`(url, match)	Base class for software hosted on github.com
`GithubRelease`(url, match)	Matches release artifacts uploaded to Github
`GithubReleaseAttachment`(url, match)	Matches release artifacts uploaded as attachment to release notes
`GithubRepoStore`(url, match)	Matches release artifacts stored in a github repo
`GithubTag`(url, match)	Matches GitHub repository archives created automatically from tags
`GitlabTag`(url, match)	Tag based releases hosted at gitlab.com
`HTMLHoster`(url, match)	Base for Hosters handling release listings in HTML format
`Hoster`(url, match)	Hoster Baseclass
`HosterMeta`	Meta-Class for Hosters
`HrefParser`(link_re)	Extract link targets from HTML
`JSONHoster`(url, match)	Base for Hosters handling release listings in JSON format
`OrderedHTMLHoster`(url, match)	HTMLHoster for which we can expected newest releases at top
`PyPi`(url, match)	Scans PyPi for updates
`SourceForge`(url, match)	Matches packages hosted at SourceForge

Documentation

class bioconda_utils.hosters.Bioarchive(url, match)[source]¶

Bases: bioconda_utils.hosters.JSONHoster

Scans for updates to packages hosted on bioarchive.galaxyproject.org

async get_versions_from_json(data, req, orig_version)[source]¶: Extract matches from json data in data

class bioconda_utils.hosters.Bioconductor(url, match)[source]¶

Bases: bioconda_utils.hosters.HTMLHoster

Matches R packages hosted at Bioconductor

class bioconda_utils.hosters.BitBucketBase(url, match)[source]¶

Bases: bioconda_utils.hosters.OrderedHTMLHoster

Base class for hosting at bitbucket.org

class bioconda_utils.hosters.BitBucketDownload(url, match)[source]¶

Bases: bioconda_utils.hosters.BitBucketBase

Uploaded releases hosted at bitbucket.org

class bioconda_utils.hosters.BitBucketTag(url, match)[source]¶

Bases: bioconda_utils.hosters.BitBucketBase

Tag based releases hosted at bitbucket.org

class bioconda_utils.hosters.CPAN(url, match)[source]¶

Bases: bioconda_utils.hosters.JSONHoster

Scans for updates to Perl packages hosted on CPAN

async get_versions_from_json(data, req, orig_version)[source]¶: Extract matches from json data in data

static parse_deps(data)[source]¶: Parse CPAN format dependencies

class bioconda_utils.hosters.CRAN(url, match)[source]¶

Bases: bioconda_utils.hosters.JSONHoster

R packages hosted on r-project.org (CRAN)

async get_versions_from_json(data, _, orig_version)[source]¶: Extract matches from json data in data

class bioconda_utils.hosters.CargoPort(url, match)[source]¶

Bases: bioconda_utils.hosters.HTMLHoster

Matches source backup urls created by cargo-port

class bioconda_utils.hosters.FTPHoster(url, match)[source]¶

Bases: bioconda_utils.hosters.Hoster

Scans for updates on FTP servers

async get_versions(req, orig_version)[source]¶: Gets list of versions from upstream hosting site

class bioconda_utils.hosters.GithubBase(url, match)[source]¶

Bases: bioconda_utils.hosters.OrderedHTMLHoster

Base class for software hosted on github.com

class bioconda_utils.hosters.GithubRelease(url, match)[source]¶

Bases: bioconda_utils.hosters.GithubBase

Matches release artifacts uploaded to Github

class bioconda_utils.hosters.GithubReleaseAttachment(url, match)[source]¶

Bases: bioconda_utils.hosters.GithubBase

Matches release artifacts uploaded as attachment to release notes

class bioconda_utils.hosters.GithubRepoStore(url, match)[source]¶

Bases: bioconda_utils.hosters.GithubBase

Matches release artifacts stored in a github repo

class bioconda_utils.hosters.GithubTag(url, match)[source]¶

Bases: bioconda_utils.hosters.GithubBase

Matches GitHub repository archives created automatically from tags

class bioconda_utils.hosters.GitlabTag(url, match)[source]¶

Bases: bioconda_utils.hosters.OrderedHTMLHoster

Tag based releases hosted at gitlab.com

class bioconda_utils.hosters.HTMLHoster(url, match)[source]¶

Bases: bioconda_utils.hosters.Hoster

Base for Hosters handling release listings in HTML format

async get_versions(req, orig_version)[source]¶: Gets list of versions from upstream hosting site

class bioconda_utils.hosters.Hoster(url, match)[source]¶

Bases: object

Hoster Baseclass

exclude = ['version']¶: named patterns that will change with a version upgrade

ext_pattern = '(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))'¶: matches archive file extensions

abstract classmethod get_versions(req, orig_version)[source]¶

Gets list of versions from upstream hosting site

Return type: List[Mapping[str, Any]]

abstract property link_pattern¶

matches links on relase page

Return type: str

abstract property releases_formats¶

format template for release page URL

Return type: List[str]

classmethod try_make_hoster(url, config)[source]¶

Creates hoster if url is matched by its url_pattern

Return type: Optional[Hoster]

abstract property url_pattern¶

matches upstream package url

Return type: str

url_re = None¶: will be generated as each class is created

version_pattern = '(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})'¶: matches upstream version - begins with a number - then only numbers, characters or one of -, +, ., :, ~ - at most 31 characters length (to avoid matching checksums) - accept v or r as prefix if after slash, dot, underscore or dash

class bioconda_utils.hosters.HosterMeta[source]¶

Bases: abc.ABCMeta

Meta-Class for Hosters

By making Hosters classes of a metaclass, rather than instances of a class, we leave the option to add functions to a Hoster.

Creates Hoster classes

expands references among {var}_pattern attributes
compiles {var}_pattern attributes to {var}_re
registers complete classes

classmethod select_hoster(url, config)[source]¶

Select Hoster able to handle url

Returns: Hoster or None

Return type: Optional[Hoster]

class bioconda_utils.hosters.HrefParser(link_re)[source]¶

Bases: html.parser.HTMLParser

Extract link targets from HTML

get_matches()[source]¶

Return matches found for link_re in href links

Return type: List[Mapping[str, Any]]

handle_a_href(href)[source]¶

Process href attributes of anchor tags

Return type: None

class bioconda_utils.hosters.JSONHoster(url, match)[source]¶

Bases: bioconda_utils.hosters.Hoster

Base for Hosters handling release listings in JSON format

async get_versions(req, orig_version)[source]¶: Gets list of versions from upstream hosting site

abstract async get_versions_from_json(data, req, orig_version)[source]¶

Extract matches from json data in data

Return type: List[Dict[str, Any]]

class bioconda_utils.hosters.OrderedHTMLHoster(url, match)[source]¶

Bases: bioconda_utils.hosters.HTMLHoster

HTMLHoster for which we can expected newest releases at top

The point isn’t performance, but avoiding hassle with old versions which may follow different versioning schemes. E.g. 0.09 -> 0.10 -> 0.2 -> 0.2.1

FIXME: If the current version is not in the list, that’s likely: a pathologic case. Should be handled somewhere.

async get_versions(req, orig_version)[source]¶: Gets list of versions from upstream hosting site

class bioconda_utils.hosters.PyPi(url, match)[source]¶

Bases: bioconda_utils.hosters.JSONHoster

Scans PyPi for updates

async get_deps(pipeline, build_config, package, rel)[source]¶

Get dependencies for package using version data rel

This is messy even though we use conda_build.skeleton.pypi to extract the requirements from a setup.py. Since the setup.py actually gets executed, all manner of things can happen (e.g. for one Bioconda package, this triggers compilation of a binary module).

async get_versions_from_json(data, req, orig_version)[source]¶: Extract matches from json data in data

bioconda_utils.hosters.RE_CAPGROUP = regex.Regex('\\(\\?P<(\\w+)>(?>[^()]+|\\\\\\(|\\\\\\)|(\\((?>[^()]+|\\\\\\(|\\\\\\)|(?2))*\\)))*\\)', flags=regex.V0)¶: Matches named capture groups This is so complicated because we need to parse matched, not-escaped parentheses to determine where the clause ends. Requires regex package for recursion.

class bioconda_utils.hosters.SourceForge(url, match)[source]¶

Bases: bioconda_utils.hosters.HTMLHoster

Matches packages hosted at SourceForge

bioconda_utils.hosters.dedup_named_capture_group(pattern)[source]¶: Replaces repetitions of capture groups with matches to first instance

bioconda_utils.hosters.replace_named_capture_group(pattern, vals)[source]¶: Replaces capture groups with values from vals