hosters

Check package URLs for updates

Subclasses of Hoster define how to handle each hoster. Hosters are selected by regex matching each source URL in a recipe. The HTMLHoster provides parsing for hosting sites listing new releases in HTML format (probably covers most). Adding a hoster is as simple as defining a regex to match the existing source URL, a formatting string creating the URL of the relases page and a regex to match links and extract their version.

• We need to use regex rather than re to allow recursive matching to manipulate capture groups in URL patterns as needed. (Technically, we could avoid this using a Snakemake wildcard type syntax to define the patterns - implementers welcome).

Functions

dedup_named_capture_group(pattern)	Replaces repetitions of capture groups with matches to first instance
replace_named_capture_group(pattern, vals)	Replaces capture groups with values from vals

Classes

Bioarchive(url, match)	Scans for updates to packages hosted on bioarchive.galaxyproject.org
Bioconductor(url, match)	Matches R packages hosted at Bioconductor
BitBucketBase(url, match)	Base class for hosting at bitbucket.org
BitBucketDownload(url, match)	Uploaded releases hosted at bitbucket.org
BitBucketTag(url, match)	Tag based releases hosted at bitbucket.org
CPAN(url, match)	Scans for updates to Perl packages hosted on CPAN
CRAN(url, match)	R packages hosted on r-project.org (CRAN)
CargoPort(url, match)	Matches source backup urls created by cargo-port
FTPHoster(url, match)	Scans for updates on FTP servers
GithubBase(url, match)	Base class for software hosted on github.com
GithubRelease(url, match)	Matches release artifacts uploaded to Github
GithubReleaseAttachment(url, match)	Matches release artifacts uploaded as attachment to release notes
GithubRepoStore(url, match)	Matches release artifacts stored in a github repo
GithubTag(url, match)	Matches GitHub repository archives created automatically from tags
GitlabTag(url, match)	Tag based releases hosted at gitlab.com
HTMLHoster(url, match)	Base for Hosters handling release listings in HTML format
Hoster(url, match)	Hoster Baseclass
HosterMeta(name, bases, …], namespace, …)	Meta-Class for Hosters
HrefParser(link_re)	Extract link targets from HTML
JSONHoster(url, match)	Base for Hosters handling release listings in JSON format
OrderedHTMLHoster(url, match)	HTMLHoster for which we can expected newest releases at top
PyPi(url, match)	Scans PyPi for updates
SourceForge(url, match)	Matches packages hosted at SourceForge

Documentation

class bioconda_utils.hosters.Bioarchive(url, match)

Scans for updates to packages hosted on bioarchive.galaxyproject.org

async get_versions_from_json(data, req, orig_version)

Extract matches from json data in data

class bioconda_utils.hosters.Bioconductor(url, match)

Matches R packages hosted at Bioconductor

class bioconda_utils.hosters.BitBucketBase(url, match)

Base class for hosting at bitbucket.org

class bioconda_utils.hosters.BitBucketDownload(url, match)

Uploaded releases hosted at bitbucket.org

class bioconda_utils.hosters.BitBucketTag(url, match)

Tag based releases hosted at bitbucket.org

class bioconda_utils.hosters.CPAN(url, match)

Scans for updates to Perl packages hosted on CPAN

async get_versions_from_json(data, req, orig_version)

Extract matches from json data in data

static parse_deps(data)

Parse CPAN format dependencies

class bioconda_utils.hosters.CRAN(url, match)

R packages hosted on r-project.org (CRAN)

async get_versions_from_json(data, _, orig_version)

Extract matches from json data in data

class bioconda_utils.hosters.CargoPort(url, match)

Matches source backup urls created by cargo-port

class bioconda_utils.hosters.FTPHoster(url, match)

Scans for updates on FTP servers

async get_versions(req, orig_version)

Gets list of versions from upstream hosting site

class bioconda_utils.hosters.GithubBase(url, match)

Base class for software hosted on github.com

class bioconda_utils.hosters.GithubRelease(url, match)

Matches release artifacts uploaded to Github

class bioconda_utils.hosters.GithubReleaseAttachment(url, match)

Matches release artifacts uploaded as attachment to release notes

class bioconda_utils.hosters.GithubRepoStore(url, match)

Matches release artifacts stored in a github repo

class bioconda_utils.hosters.GithubTag(url, match)

Matches GitHub repository archives created automatically from tags

class bioconda_utils.hosters.GitlabTag(url, match)

Tag based releases hosted at gitlab.com

class bioconda_utils.hosters.HTMLHoster(url, match)

Base for Hosters handling release listings in HTML format

async get_versions(req, orig_version)

Gets list of versions from upstream hosting site

class bioconda_utils.hosters.Hoster(url, match)

Hoster Baseclass

exclude = ['version']

named patterns that will change with a version upgrade

ext_pattern: str = '(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))'

matches archive file extensions

abstract classmethod get_versions(req, orig_version)

Gets list of versions from upstream hosting site

Return type

List[Mapping[str, Any]]

abstract property link_pattern

matches links on relase page

Return type

str

abstract property releases_formats

format template for release page URL

Return type

List[str]

classmethod try_make_hoster(url, config)

Creates hoster if url is matched by its url_pattern

Return type

Optional[Hoster]

abstract property url_pattern

matches upstream package url

Return type

str

url_re: Pattern[str] = None

will be generated as each class is created

version_pattern: str = '(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})'

matches upstream version - begins with a number - then only numbers, characters or one of -, +, ., :, ~ - at most 31 characters length (to avoid matching checksums) - accept v or r as prefix if after slash, dot, underscore or dash

class bioconda_utils.hosters.HosterMeta(name: str, bases: Tuple[type, …], namespace: Dict[str, Any], **kwargs)

Meta-Class for Hosters

By making Hosters classes of a metaclass, rather than instances of a class, we leave the option to add functions to a Hoster.

Creates Hoster classes

• expands references among {var}_pattern attributes

• compiles {var}_pattern attributes to {var}_re

• registers complete classes

classmethod select_hoster(url, config)

Select Hoster able to handle url

Returns: Hoster or None

Return type

Optional[Hoster]

class bioconda_utils.hosters.HrefParser(link_re)

Extract link targets from HTML

Initialize and reset this instance.

If convert_charrefs is True (the default), all character references are automatically converted to the corresponding Unicode characters.

get_matches()

Return matches found for link_re in href links

Return type

List[Mapping[str, Any]]

handle_a_href(href)

Process href attributes of anchor tags

Return type

None

class bioconda_utils.hosters.JSONHoster(url, match)

Base for Hosters handling release listings in JSON format

async get_versions(req, orig_version)

Gets list of versions from upstream hosting site

abstract async get_versions_from_json(data, req, orig_version)

Extract matches from json data in data

Return type

List[Dict[str, Any]]

class bioconda_utils.hosters.OrderedHTMLHoster(url, match)

HTMLHoster for which we can expected newest releases at top

The point isn’t performance, but avoiding hassle with old versions which may follow different versioning schemes. E.g. 0.09 -> 0.10 -> 0.2 -> 0.2.1

FIXME: If the current version is not in the list, that’s likely

a pathologic case. Should be handled somewhere.

async get_versions(req, orig_version)

Gets list of versions from upstream hosting site

class bioconda_utils.hosters.PyPi(url, match)

Scans PyPi for updates

async get_deps(pipeline, build_config, package, rel)

Get dependencies for package using version data rel

This is messy even though we use conda_build.skeleton.pypi to extract the requirements from a setup.py. Since the setup.py actually gets executed, all manner of things can happen (e.g. for one Bioconda package, this triggers compilation of a binary module).

async get_versions_from_json(data, req, orig_version)

Extract matches from json data in data

bioconda_utils.hosters.RE_CAPGROUP = regex.Regex('\\(\\?P<(\\w+)>(?>[^()]+|\\\\\\(|\\\\\\)|(\\((?>[^()]+|\\\\\\(|\\\\\\)|(?2))*\\)))*\\)', flags=regex.V0)

Matches named capture groups This is so complicated because we need to parse matched, not-escaped parentheses to determine where the clause ends. Requires regex package for recursion.

class bioconda_utils.hosters.SourceForge(url, match)

Matches packages hosted at SourceForge

bioconda_utils.hosters.dedup_named_capture_group(pattern)

Replaces repetitions of capture groups with matches to first instance

bioconda_utils.hosters.replace_named_capture_group(pattern, vals)

Replaces capture groups with values from vals