hosters

Check package URLs for updates

Subclasses of Hoster define how to handle each hoster. Hosters are selected by regex matching each source URL in a recipe. The HTMLHoster provides parsing for hosting sites listing new releases in HTML format (probably covers most). Adding a hoster is as simple as defining a regex to match the existing source URL, a formatting string creating the URL of the relases page and a regex to match links and extract their version.

  • We need to use regex rather than re to allow recursive matching to manipulate capture groups in URL patterns as needed. (Technically, we could avoid this using a Snakemake wildcard type syntax to define the patterns - implementers welcome).

Functions

dedup_named_capture_group(pattern)

Replaces repetitions of capture groups with matches to first instance

replace_named_capture_group(pattern, vals)

Replaces capture groups with values from vals

Classes

Bioarchive(url, match)

Scans for updates to packages hosted on bioarchive.galaxyproject.org

Bioconductor(url, match)

Matches R packages hosted at Bioconductor

BitBucketBase(url, match)

Base class for hosting at bitbucket.org

BitBucketDownload(url, match)

Uploaded releases hosted at bitbucket.org

BitBucketTag(url, match)

Tag based releases hosted at bitbucket.org

CPAN(url, match)

Scans for updates to Perl packages hosted on CPAN

CRAN(url, match)

R packages hosted on r-project.org (CRAN)

CargoPort(url, match)

Matches source backup urls created by cargo-port

FTPHoster(url, match)

Scans for updates on FTP servers

GithubBase(url, match)

Base class for software hosted on github.com

GithubRelease(url, match)

Matches release artifacts uploaded to Github

GithubReleaseAttachment(url, match)

Matches release artifacts uploaded as attachment to release notes

GithubRepoStore(url, match)

Matches release artifacts stored in a github repo

GithubTag(url, match)

Matches GitHub repository archives created automatically from tags

GitlabTag(url, match)

Tag based releases hosted at gitlab.com

HTMLHoster(url, match)

Base for Hosters handling release listings in HTML format

Hoster(url, match)

Hoster Baseclass

HosterMeta

Meta-Class for Hosters

HrefParser(link_re)

Extract link targets from HTML

JSONHoster(url, match)

Base for Hosters handling release listings in JSON format

OrderedHTMLHoster(url, match)

HTMLHoster for which we can expected newest releases at top

PyPi(url, match)

Scans PyPi for updates

SourceForge(url, match)

Matches packages hosted at SourceForge

Documentation

class bioconda_utils.hosters.Bioarchive(url, match)[source]

Bases: bioconda_utils.hosters.JSONHoster

Scans for updates to packages hosted on bioarchive.galaxyproject.org

async get_versions_from_json(data, req, orig_version)[source]

Extract matches from json data in data

class bioconda_utils.hosters.Bioconductor(url, match)[source]

Bases: bioconda_utils.hosters.HTMLHoster

Matches R packages hosted at Bioconductor

class bioconda_utils.hosters.BitBucketBase(url, match)[source]

Bases: bioconda_utils.hosters.OrderedHTMLHoster

Base class for hosting at bitbucket.org

class bioconda_utils.hosters.BitBucketDownload(url, match)[source]

Bases: bioconda_utils.hosters.BitBucketBase

Uploaded releases hosted at bitbucket.org

class bioconda_utils.hosters.BitBucketTag(url, match)[source]

Bases: bioconda_utils.hosters.BitBucketBase

Tag based releases hosted at bitbucket.org

class bioconda_utils.hosters.CPAN(url, match)[source]

Bases: bioconda_utils.hosters.JSONHoster

Scans for updates to Perl packages hosted on CPAN

async get_versions_from_json(data, req, orig_version)[source]

Extract matches from json data in data

static parse_deps(data)[source]

Parse CPAN format dependencies

class bioconda_utils.hosters.CRAN(url, match)[source]

Bases: bioconda_utils.hosters.JSONHoster

R packages hosted on r-project.org (CRAN)

async get_versions_from_json(data, _, orig_version)[source]

Extract matches from json data in data

class bioconda_utils.hosters.CargoPort(url, match)[source]

Bases: bioconda_utils.hosters.HTMLHoster

Matches source backup urls created by cargo-port

class bioconda_utils.hosters.FTPHoster(url, match)[source]

Bases: bioconda_utils.hosters.Hoster

Scans for updates on FTP servers

async get_versions(req, orig_version)[source]

Gets list of versions from upstream hosting site

class bioconda_utils.hosters.GithubBase(url, match)[source]

Bases: bioconda_utils.hosters.OrderedHTMLHoster

Base class for software hosted on github.com

class bioconda_utils.hosters.GithubRelease(url, match)[source]

Bases: bioconda_utils.hosters.GithubBase

Matches release artifacts uploaded to Github

class bioconda_utils.hosters.GithubReleaseAttachment(url, match)[source]

Bases: bioconda_utils.hosters.GithubBase

Matches release artifacts uploaded as attachment to release notes

class bioconda_utils.hosters.GithubRepoStore(url, match)[source]

Bases: bioconda_utils.hosters.GithubBase

Matches release artifacts stored in a github repo

class bioconda_utils.hosters.GithubTag(url, match)[source]

Bases: bioconda_utils.hosters.GithubBase

Matches GitHub repository archives created automatically from tags

class bioconda_utils.hosters.GitlabTag(url, match)[source]

Bases: bioconda_utils.hosters.OrderedHTMLHoster

Tag based releases hosted at gitlab.com

class bioconda_utils.hosters.HTMLHoster(url, match)[source]

Bases: bioconda_utils.hosters.Hoster

Base for Hosters handling release listings in HTML format

async get_versions(req, orig_version)[source]

Gets list of versions from upstream hosting site

class bioconda_utils.hosters.Hoster(url, match)[source]

Bases: object

Hoster Baseclass

exclude = ['version']

named patterns that will change with a version upgrade

ext_pattern = '(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))'

matches archive file extensions

abstract classmethod get_versions(req, orig_version)[source]

Gets list of versions from upstream hosting site

Return type

List[Mapping[str, Any]]

matches links on relase page

Return type

str

abstract property releases_formats

format template for release page URL

Return type

List[str]

classmethod try_make_hoster(url, config)[source]

Creates hoster if url is matched by its url_pattern

Return type

Optional[Hoster]

abstract property url_pattern

matches upstream package url

Return type

str

url_re = None

will be generated as each class is created

version_pattern = '(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})'

matches upstream version - begins with a number - then only numbers, characters or one of -, +, ., :, ~ - at most 31 characters length (to avoid matching checksums) - accept v or r as prefix if after slash, dot, underscore or dash

class bioconda_utils.hosters.HosterMeta[source]

Bases: abc.ABCMeta

Meta-Class for Hosters

By making Hosters classes of a metaclass, rather than instances of a class, we leave the option to add functions to a Hoster.

Creates Hoster classes

  • expands references among {var}_pattern attributes

  • compiles {var}_pattern attributes to {var}_re

  • registers complete classes

classmethod select_hoster(url, config)[source]

Select Hoster able to handle url

Returns: Hoster or None

Return type

Optional[Hoster]

class bioconda_utils.hosters.HrefParser(link_re)[source]

Bases: html.parser.HTMLParser

Extract link targets from HTML

Initialize and reset this instance.

If convert_charrefs is True (the default), all character references are automatically converted to the corresponding Unicode characters.

get_matches()[source]

Return matches found for link_re in href links

Return type

List[Mapping[str, Any]]

handle_a_href(href)[source]

Process href attributes of anchor tags

Return type

None

class bioconda_utils.hosters.JSONHoster(url, match)[source]

Bases: bioconda_utils.hosters.Hoster

Base for Hosters handling release listings in JSON format

async get_versions(req, orig_version)[source]

Gets list of versions from upstream hosting site

abstract async get_versions_from_json(data, req, orig_version)[source]

Extract matches from json data in data

Return type

List[Dict[str, Any]]

class bioconda_utils.hosters.OrderedHTMLHoster(url, match)[source]

Bases: bioconda_utils.hosters.HTMLHoster

HTMLHoster for which we can expected newest releases at top

The point isn’t performance, but avoiding hassle with old versions which may follow different versioning schemes. E.g. 0.09 -> 0.10 -> 0.2 -> 0.2.1

FIXME: If the current version is not in the list, that’s likely

a pathologic case. Should be handled somewhere.

async get_versions(req, orig_version)[source]

Gets list of versions from upstream hosting site

class bioconda_utils.hosters.PyPi(url, match)[source]

Bases: bioconda_utils.hosters.JSONHoster

Scans PyPi for updates

async get_deps(pipeline, build_config, package, rel)[source]

Get dependencies for package using version data rel

This is messy even though we use conda_build.skeleton.pypi to extract the requirements from a setup.py. Since the setup.py actually gets executed, all manner of things can happen (e.g. for one Bioconda package, this triggers compilation of a binary module).

async get_versions_from_json(data, req, orig_version)[source]

Extract matches from json data in data

bioconda_utils.hosters.RE_CAPGROUP = regex.Regex('\\(\\?P<(\\w+)>(?>[^()]+|\\\\\\(|\\\\\\)|(\\((?>[^()]+|\\\\\\(|\\\\\\)|(?2))*\\)))*\\)', flags=regex.V0)

Matches named capture groups This is so complicated because we need to parse matched, not-escaped parentheses to determine where the clause ends. Requires regex package for recursion.

class bioconda_utils.hosters.SourceForge(url, match)[source]

Bases: bioconda_utils.hosters.HTMLHoster

Matches packages hosted at SourceForge

bioconda_utils.hosters.dedup_named_capture_group(pattern)[source]

Replaces repetitions of capture groups with matches to first instance

bioconda_utils.hosters.replace_named_capture_group(pattern, vals)[source]

Replaces capture groups with values from vals