hosters

Check package URLs for updates

Subclasses of Hoster define how to handle each hoster. Hosters are selected by regex matching each source URL in a recipe. The HTMLHoster provides parsing for hosting sites listing new releases in HTML format (probably covers most). Adding a hoster is as simple as defining a regex to match the existing source URL, a formatting string creating the URL of the relases page and a regex to match links and extract their version.

• We need to use regex rather than re to allow recursive matching to manipulate capture groups in URL patterns as needed. (Technically, we could avoid this using a Snakemake wildcard type syntax to define the patterns - implementers welcome).

Functions

dedup_named_capture_group(pattern)	Replaces repetitions of capture groups with matches to first instance
replace_named_capture_group(pattern, vals)	Replaces capture groups with values from vals

Classes

Bioarchive(url, match)	Scans for updates to packages hosted on bioarchive.galaxyproject.org
Bioconductor(url, match)	Matches R packages hosted at Bioconductor
BitBucketBase(url, match)	Base class for hosting at bitbucket.org
BitBucketDownload(url, match)	Uploaded releases hosted at bitbucket.org
BitBucketTag(url, match)	Tag based releases hosted at bitbucket.org
CPAN(url, match)	Scans for updates to Perl packages hosted on CPAN
CRAN(url, match)	R packages hosted on r-project.org (CRAN)
CargoPort(url, match)	Matches source backup urls created by cargo-port
FTPHoster(url, match)	Scans for updates on FTP servers
GithubBase(url, match)	Base class for software hosted on github.com
GithubRelease(url, match)	Matches release artifacts uploaded to Github
GithubReleaseAttachment(url, match)	Matches release artifacts uploaded as attachment to release notes
GithubRepoStore(url, match)	Matches release artifacts stored in a github repo
GithubTag(url, match)	Matches GitHub repository archives created automatically from tags
GitlabTag(url, match)	Tag based releases hosted at gitlab.com
HTMLHoster(url, match)	Base for Hosters handling release listings in HTML format
Hoster(url, match)	Hoster Baseclass
HosterMeta(name, bases, namespace, **kwargs)	Meta-Class for Hosters
HrefParser(link_re)	Extract link targets from HTML
JSONHoster(url, match)	Base for Hosters handling release listings in JSON format
OrderedHTMLHoster(url, match)	HTMLHoster for which we can expected newest releases at top
PyPi(url, match)	Scans PyPi for updates
SourceForge(url, match)	Matches packages hosted at SourceForge

Documentation

bioconda_utils.hosters.RE_CAPGROUP = regex.Regex('\\(\\?P<(\\w+)>(?>[^()]+|\\\\\\(|\\\\\\)|(\\((?>[^()]+|\\\\\\(|\\\\\\)|(?2))*\\)))*\\)', flags=regex.V0)

Matches named capture groups This is so complicated because we need to parse matched, not-escaped parentheses to determine where the clause ends. Requires regex package for recursion.

bioconda_utils.hosters.dedup_named_capture_group(pattern)

Replaces repetitions of capture groups with matches to first instance

bioconda_utils.hosters.replace_named_capture_group(pattern, vals)

Replaces capture groups with values from vals

class bioconda_utils.hosters.HosterMeta(name: str, bases: Tuple[type, ...], namespace: Dict[str, Any], **kwargs)

Meta-Class for Hosters

By making Hosters classes of a metaclass, rather than instances of a class, we leave the option to add functions to a Hoster.

Creates Hoster classes

• expands references among {var}_pattern attributes

• compiles {var}_pattern attributes to {var}_re

• registers complete classes

classmethod select_hoster(url, config)

Select Hoster able to handle url

Returns: Hoster or None

Return type:

Optional[Hoster]

class bioconda_utils.hosters.Hoster(url, match)

Hoster Baseclass

version_pattern: str = '(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})'

matches upstream version - begins with a number - then only numbers, characters or one of -, +, ., :, ~ - at most 31 characters length (to avoid matching checksums) - accept v or r as prefix if after slash, dot, underscore or dash

ext_pattern: str = '(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))'

matches archive file extensions

exclude = ['version']

named patterns that will change with a version upgrade

abstract property url_pattern: str

matches upstream package url

Return type:

str

url_re: Pattern[str] = None

will be generated as each class is created

abstract property link_pattern: str

matches links on relase page

Return type:

str

abstract property releases_formats: List[str]

format template for release page URL

Return type:

List[str]

classmethod try_make_hoster(url, config)

Creates hoster if url is matched by its url_pattern

Return type:

Optional[Hoster]

abstract classmethod get_versions(req, orig_version)

Gets list of versions from upstream hosting site

Return type:

List[Mapping[str, Any]]

class bioconda_utils.hosters.HrefParser(link_re)

Extract link targets from HTML

Initialize and reset this instance.

If convert_charrefs is True (the default), all character references are automatically converted to the corresponding Unicode characters.

get_matches()

Return matches found for link_re in href links

Return type:

List[Mapping[str, Any]]

handle_a_href(href)

Process href attributes of anchor tags

Return type:

None

class bioconda_utils.hosters.HTMLHoster(url, match)

Base for Hosters handling release listings in HTML format

async get_versions(req, orig_version)

Gets list of versions from upstream hosting site

class bioconda_utils.hosters.FTPHoster(url, match)

Scans for updates on FTP servers

async get_versions(req, orig_version)

Gets list of versions from upstream hosting site

version_pattern: str = '(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30}?)'

matches upstream version - begins with a number - then only numbers, characters or one of -, +, ., :, ~ - at most 31 characters length (to avoid matching checksums) - accept v or r as prefix if after slash, dot, underscore or dash

url_re: Pattern[str] = regex.Regex('ftp://(?P<host>[-_.\\w]+)/(?P<path>[-_/.\\w]+/)(?P<package>[-_\\w]+)(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30}?)(?P<suffix>([-_](lin|linux|Linux|x64|x86|src|64|OSX))*)(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.OrderedHTMLHoster(url, match)

HTMLHoster for which we can expected newest releases at top

The point isn’t performance, but avoiding hassle with old versions which may follow different versioning schemes. E.g. 0.09 -> 0.10 -> 0.2 -> 0.2.1

FIXME: If the current version is not in the list, that’s likely

a pathologic case. Should be handled somewhere.

async get_versions(req, orig_version)

Gets list of versions from upstream hosting site

class bioconda_utils.hosters.GithubBase(url, match)

Base class for software hosted on github.com

exclude = ['version', 'fname']

named patterns that will change with a version upgrade

class bioconda_utils.hosters.GithubRelease(url, match)

Matches release artifacts uploaded to Github

url_re: Pattern[str] = regex.Regex('github\\.com/(?P<account>[-\\w]+)/(?P<project>[-.\\w]+)/releases/download/(?P<prefix>[-_./\\w]+?)??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})/(?P<fname>[^/]+)(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))?', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.GithubTag(url, match)

Matches GitHub repository archives created automatically from tags

url_re: Pattern[str] = regex.Regex('github\\.com/(?P<account>[-\\w]+)/(?P<project>[-.\\w]+)/archive(/refs/tags)?/(?P<prefix>[-_./\\w]+?)??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.GithubReleaseAttachment(url, match)

Matches release artifacts uploaded as attachment to release notes

url_re: Pattern[str] = regex.Regex('github\\.com/(?P<account>[-\\w]+)/(?P<project>[-.\\w]+)/files/\\d+/(?P<prefix>[-_./\\w]+?)??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.GithubRepoStore(url, match)

Matches release artifacts stored in a github repo

url_re: Pattern[str] = regex.Regex('(?:(?P<raw>raw\\.githubusercontent)|github)\\.com/(?P<account>[-\\w]+)/(?P<project>[-.\\w]+)/(?(raw)|(?:(?P<blob>blob/)|raw/))(master|[\\da-f]{40})/(?P<subdir>([-._\\w]+/)+)?(?P<prefix>[-_./\\w]+?)??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))(?(blob)\\?raw|)', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.Bioconductor(url, match)

Matches R packages hosted at Bioconductor

url_re: Pattern[str] = regex.Regex('bioconductor.org/packages/(?P<bioc>[\\d\\.]+)/(bioc|data/annotation|data/experiment)/src/contrib/(?P<package>[^/]+)_(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.CargoPort(url, match)

Matches source backup urls created by cargo-port

url_re: Pattern[str] = regex.Regex('depot.galaxyproject.org/software/(?P<package>[^/]+)/(?P=package)_(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})_(?P<os>src_all|linux_x86|darwin_x86)(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.SourceForge(url, match)

Matches packages hosted at SourceForge

url_re: Pattern[str] = regex.Regex('sourceforge\\.net/project(s)?/(?P<project>[-\\w]+)/(?(1)files/|)((?P<subproject>[-\\w%]+)/)?(?P<package>[-\\w_\\.+]*?[a-zA-Z+])(?P<sep>[-_.]?)((?P<type2>((linux|x?(64|86)|src|source|all|core|java\\d?)[-_.])*)(?P=sep))?(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})((?P=sep)(?P<type>((linux|x?(64|86)|src|source|all|core|java\\d?)[-_.])*))?(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.JSONHoster(url, match)

Base for Hosters handling release listings in JSON format

async get_versions(req, orig_version)

Gets list of versions from upstream hosting site

abstract async get_versions_from_json(data, req, orig_version)

Extract matches from json data in data

Return type:

List[Dict[str, Any]]

class bioconda_utils.hosters.PyPi(url, match)

Scans PyPi for updates

async get_versions_from_json(data, req, orig_version)

Extract matches from json data in data

async get_deps(pipeline, build_config, package, rel)

Get dependencies for package using version data rel

This is messy even though we use conda_build.skeleton.pypi to extract the requirements from a setup.py. Since the setup.py actually gets executed, all manner of things can happen (e.g. for one Bioconda package, this triggers compilation of a binary module).

url_re: Pattern[str] = regex.Regex('(?P<hoster>files.pythonhosted.org/packages|pypi.python.org/packages|pypi.io/packages)/.*/(?P<package>[\\w\\-\\.]+)[-_](?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.Bioarchive(url, match)

Scans for updates to packages hosted on bioarchive.galaxyproject.org

async get_versions_from_json(data, req, orig_version)

Extract matches from json data in data

url_re: Pattern[str] = regex.Regex('bioarchive.galaxyproject.org/(?P<package>[-\\w.]+)_(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.CPAN(url, match)

Scans for updates to Perl packages hosted on CPAN

static parse_deps(data)

Parse CPAN format dependencies

async get_versions_from_json(data, req, orig_version)

Extract matches from json data in data

url_re: Pattern[str] = regex.Regex('(www.cpan.org|cpan.metacpan.org|search.cpan.org/CPAN)/authors/id/./../(?P<author>[A-Z]+)/([^/]+/|)(?P<package>[-\\w.+]+)-v?(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.CRAN(url, match)

R packages hosted on r-project.org (CRAN)

async get_versions_from_json(data, _, orig_version)

Extract matches from json data in data

url_re: Pattern[str] = regex.Regex('r-project\\.org/src/contrib(/Archive)?/(?P<package>[\\w.]+)(?(1)/(?P=package)|)_(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.BitBucketBase(url, match)

Base class for hosting at bitbucket.org

class bioconda_utils.hosters.BitBucketTag(url, match)

Tag based releases hosted at bitbucket.org

url_re: Pattern[str] = regex.Regex('bitbucket\\.org/(?P<account>[-\\w]+)/(?P<project>[-.\\w]+)/get/(?P<prefix>[-_./\\w]+?)??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.BitBucketDownload(url, match)

Uploaded releases hosted at bitbucket.org

url_re: Pattern[str] = regex.Regex('bitbucket\\.org/(?P<account>[-\\w]+)/(?P<project>[-.\\w]+)/downloads/(?P<prefix>[-_./\\w]+?)??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created

class bioconda_utils.hosters.GitlabTag(url, match)

Tag based releases hosted at gitlab.com

url_re: Pattern[str] = regex.Regex('gitlab\\.com/(?P<account>[-\\w]+)(?P<subgroup>(?:/[-\\w]+|))/(?P<project>[-.\\w]+)/(repository|-/archive)/(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})/(archive|(?P=project)-(?:(?<=[/._-])[rv])?(?P=version))(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)

will be generated as each class is created