hosters¶
Check package URLs for updates
Subclasses of Hoster
define how to handle each hoster. Hosters are
selected by regex matching each source URL in a recipe. The
HTMLHoster
provides parsing for hosting sites listing new
releases in HTML format (probably covers most). Adding a hoster is
as simple as defining a regex to match the existing source URL, a
formatting string creating the URL of the relases page and a regex
to match links and extract their version.
We need to use
regex
rather thanre
to allow recursive matching to manipulate capture groups in URL patterns as needed. (Technically, we could avoid this using a Snakemake wildcard type syntax to define the patterns - implementers welcome).
Functions
|
Replaces repetitions of capture groups with matches to first instance |
|
Replaces capture groups with values from vals |
Classes
|
Scans for updates to packages hosted on bioarchive.galaxyproject.org |
|
Matches R packages hosted at Bioconductor |
|
Base class for hosting at bitbucket.org |
|
Uploaded releases hosted at bitbucket.org |
|
Tag based releases hosted at bitbucket.org |
|
Scans for updates to Perl packages hosted on CPAN |
|
R packages hosted on r-project.org (CRAN) |
|
Matches source backup urls created by cargo-port |
|
Scans for updates on FTP servers |
|
Base class for software hosted on github.com |
|
Matches release artifacts uploaded to Github |
|
Matches release artifacts uploaded as attachment to release notes |
|
Matches release artifacts stored in a github repo |
|
Matches GitHub repository archives created automatically from tags |
|
Tag based releases hosted at gitlab.com |
|
Base for Hosters handling release listings in HTML format |
|
Hoster Baseclass |
|
Meta-Class for Hosters |
|
Extract link targets from HTML |
|
Base for Hosters handling release listings in JSON format |
|
HTMLHoster for which we can expected newest releases at top |
|
Scans PyPi for updates |
|
Matches packages hosted at SourceForge |
Documentation
- bioconda_utils.hosters.RE_CAPGROUP = regex.Regex('\\(\\?P<(\\w+)>(?>[^()]+|\\\\\\(|\\\\\\)|(\\((?>[^()]+|\\\\\\(|\\\\\\)|(?2))*\\)))*\\)', flags=regex.V0)¶
Matches named capture groups This is so complicated because we need to parse matched, not-escaped parentheses to determine where the clause ends. Requires regex package for recursion.
- bioconda_utils.hosters.dedup_named_capture_group(pattern)[source]¶
Replaces repetitions of capture groups with matches to first instance
- bioconda_utils.hosters.replace_named_capture_group(pattern, vals)[source]¶
Replaces capture groups with values from vals
- class bioconda_utils.hosters.HosterMeta(name: str, bases: Tuple[type, ...], namespace: Dict[str, Any], **kwargs)[source]¶
Meta-Class for Hosters
By making Hosters classes of a metaclass, rather than instances of a class, we leave the option to add functions to a Hoster.
Creates Hoster classes
expands references among
{var}_pattern
attributescompiles
{var}_pattern
attributes to{var}_re
registers complete classes
- class bioconda_utils.hosters.Hoster(url, match)[source]¶
Hoster Baseclass
- version_pattern: str = '(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})'¶
matches upstream version - begins with a number - then only numbers, characters or one of -, +, ., :, ~ - at most 31 characters length (to avoid matching checksums) - accept v or r as prefix if after slash, dot, underscore or dash
- ext_pattern: str = '(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))'¶
matches archive file extensions
- exclude = ['version']¶
named patterns that will change with a version upgrade
- class bioconda_utils.hosters.HrefParser(link_re)[source]¶
Extract link targets from HTML
Initialize and reset this instance.
If convert_charrefs is True (the default), all character references are automatically converted to the corresponding Unicode characters.
- class bioconda_utils.hosters.HTMLHoster(url, match)[source]¶
Base for Hosters handling release listings in HTML format
- class bioconda_utils.hosters.FTPHoster(url, match)[source]¶
Scans for updates on FTP servers
- version_pattern: str = '(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30}?)'¶
matches upstream version - begins with a number - then only numbers, characters or one of -, +, ., :, ~ - at most 31 characters length (to avoid matching checksums) - accept v or r as prefix if after slash, dot, underscore or dash
- url_re: Pattern[str] = regex.Regex('ftp://(?P<host>[-_.\\w]+)/(?P<path>[-_/.\\w]+/)(?P<package>[-_\\w]+)(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30}?)(?P<suffix>([-_](lin|linux|Linux|x64|x86|src|64|OSX))*)(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.OrderedHTMLHoster(url, match)[source]¶
HTMLHoster for which we can expected newest releases at top
The point isn’t performance, but avoiding hassle with old versions which may follow different versioning schemes. E.g. 0.09 -> 0.10 -> 0.2 -> 0.2.1
- FIXME: If the current version is not in the list, that’s likely
a pathologic case. Should be handled somewhere.
- class bioconda_utils.hosters.GithubBase(url, match)[source]¶
Base class for software hosted on github.com
- exclude = ['version', 'fname']¶
named patterns that will change with a version upgrade
- class bioconda_utils.hosters.GithubRelease(url, match)[source]¶
Matches release artifacts uploaded to Github
- url_re: Pattern[str] = regex.Regex('github\\.com/(?P<account>[-\\w]+)/(?P<project>[-.\\w]+)/releases/download/(?P<prefix>[-_./\\w]+?)??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})/(?P<fname>[^/]+)(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))?', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.GithubTag(url, match)[source]¶
Matches GitHub repository archives created automatically from tags
- url_re: Pattern[str] = regex.Regex('github\\.com/(?P<account>[-\\w]+)/(?P<project>[-.\\w]+)/archive(/refs/tags)?/(?P<prefix>[-_./\\w]+?)??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.GithubReleaseAttachment(url, match)[source]¶
Matches release artifacts uploaded as attachment to release notes
- url_re: Pattern[str] = regex.Regex('github\\.com/(?P<account>[-\\w]+)/(?P<project>[-.\\w]+)/files/\\d+/(?P<prefix>[-_./\\w]+?)??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.GithubRepoStore(url, match)[source]¶
Matches release artifacts stored in a github repo
- url_re: Pattern[str] = regex.Regex('(?:(?P<raw>raw\\.githubusercontent)|github)\\.com/(?P<account>[-\\w]+)/(?P<project>[-.\\w]+)/(?(raw)|(?:(?P<blob>blob/)|raw/))(master|[\\da-f]{40})/(?P<subdir>([-._\\w]+/)+)?(?P<prefix>[-_./\\w]+?)??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))(?(blob)\\?raw|)', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.Bioconductor(url, match)[source]¶
Matches R packages hosted at Bioconductor
- url_re: Pattern[str] = regex.Regex('bioconductor.org/packages/(?P<bioc>[\\d\\.]+)/(bioc|data/annotation|data/experiment)/src/contrib/(?P<package>[^/]+)_(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.CargoPort(url, match)[source]¶
Matches source backup urls created by cargo-port
- url_re: Pattern[str] = regex.Regex('depot.galaxyproject.org/software/(?P<package>[^/]+)/(?P=package)_(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})_(?P<os>src_all|linux_x86|darwin_x86)(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.SourceForge(url, match)[source]¶
Matches packages hosted at SourceForge
- url_re: Pattern[str] = regex.Regex('sourceforge\\.net/project(s)?/(?P<project>[-\\w]+)/(?(1)files/|)((?P<subproject>[-\\w%]+)/)?(?P<package>[-\\w_\\.+]*?[a-zA-Z+])(?P<sep>[-_.]?)((?P<type2>((linux|x?(64|86)|src|source|all|core|java\\d?)[-_.])*)(?P=sep))?(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})((?P=sep)(?P<type>((linux|x?(64|86)|src|source|all|core|java\\d?)[-_.])*))?(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.JSONHoster(url, match)[source]¶
Base for Hosters handling release listings in JSON format
- class bioconda_utils.hosters.PyPi(url, match)[source]¶
Scans PyPi for updates
- async get_versions_from_json(data, req, orig_version)[source]¶
Extract matches from json data in data
- async get_deps(pipeline, build_config, package, rel)[source]¶
Get dependencies for package using version data rel
This is messy even though we use conda_build.skeleton.pypi to extract the requirements from a setup.py. Since the setup.py actually gets executed, all manner of things can happen (e.g. for one Bioconda package, this triggers compilation of a binary module).
- url_re: Pattern[str] = regex.Regex('(?P<hoster>files.pythonhosted.org/packages|pypi.python.org/packages|pypi.io/packages)/.*/(?P<package>[\\w\\-\\.]+)[-_](?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.Bioarchive(url, match)[source]¶
Scans for updates to packages hosted on bioarchive.galaxyproject.org
- class bioconda_utils.hosters.CPAN(url, match)[source]¶
Scans for updates to Perl packages hosted on CPAN
- async get_versions_from_json(data, req, orig_version)[source]¶
Extract matches from json data in data
- url_re: Pattern[str] = regex.Regex('(www.cpan.org|cpan.metacpan.org|search.cpan.org/CPAN)/authors/id/./../(?P<author>[A-Z]+)/([^/]+/|)(?P<package>[-\\w.+]+)-v?(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.CRAN(url, match)[source]¶
R packages hosted on r-project.org (CRAN)
- url_re: Pattern[str] = regex.Regex('r-project\\.org/src/contrib(/Archive)?/(?P<package>[\\w.]+)(?(1)/(?P=package)|)_(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.BitBucketBase(url, match)[source]¶
Base class for hosting at bitbucket.org
- class bioconda_utils.hosters.BitBucketTag(url, match)[source]¶
Tag based releases hosted at bitbucket.org
- url_re: Pattern[str] = regex.Regex('bitbucket\\.org/(?P<account>[-\\w]+)/(?P<project>[-.\\w]+)/get/(?P<prefix>[-_./\\w]+?)??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.BitBucketDownload(url, match)[source]¶
Uploaded releases hosted at bitbucket.org
- url_re: Pattern[str] = regex.Regex('bitbucket\\.org/(?P<account>[-\\w]+)/(?P<project>[-.\\w]+)/downloads/(?P<prefix>[-_./\\w]+?)??(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)¶
will be generated as each class is created
- class bioconda_utils.hosters.GitlabTag(url, match)[source]¶
Tag based releases hosted at gitlab.com
- url_re: Pattern[str] = regex.Regex('gitlab\\.com/(?P<account>[-\\w]+)(?P<subgroup>(?:/[-\\w]+|))/(?P<project>[-.\\w]+)/(repository|-/archive)/(?:(?<=[/._-])[rv])?(?P<version>\\d[\\da-zA-Z\\-+\\.:\\~_]{0,30})/(archive|(?P=project)-(?:(?<=[/._-])[rv])?(?P=version))(?P<ext>(?i)\\.(?:(?:(tar\\.|t)(?:xz|bz2|gz))|zip|jar))', flags=regex.I | regex.V0)¶
will be generated as each class is created