autobump¶
Updates packages when new upstream releases become available
Overview:
The
Scanner
initializes theasyncio
loop and for each recipe, loads themeta.yaml
usingRecipe
and executes itsFilter
objects for each recipe.The
Recipe
handles reading, modification and writing ofmeta.yaml
files.The filters
ExcludeSubrecipe
andExcludeOtherChannel
exclude recipes in sub folders and present in other channels as configured.The filter
ExcludeNoActiveUpdate
excludes packages that don’t already have an ongoing autobump update.The filters
LoadRecipe
andGitLoadRecipe
fill the stub Recipe with content.The filter
UpdateVersion
determines available upstream verions usingHoster
and its subclasses, selects the most recent, acceptable version and usesRecipe
to replaceThe filter
FetchUpstreamDependencies
gathers additional dependencies not found by theUpdateVersion
filter. (Currently, that’s only trying to get Python deps by loading thesetup.py
)The filter
UpdateChecksums
downloads the modified source URLs and updates the checksums for each source of a recipe.The filter
WriteRecipe
or the filterGitWriteRecipe
are used to write the recipe back, either to the current folder structure, or the git repo and creating a branch.The filter
CreatePullRequest
creates pull requests on GitHub for changes made to per-recipe-branches.The filter
MaxUpdates
terminates the loop after a number of recipes have come past it, to avoid too many PRs opened at once.
Classes
|
|
|
Bump recipes in need of rebuild after pinning changes |
|
Create or Update PR on GitHub |
|
Exclude blacklisted recipes |
|
Exclude recipes depending on packages in need of update |
|
Exclude recipes disabled via config |
|
Exclude recipes not currently undergoing an update |
|
Exclude recipes in channels |
|
Exclude sub-recipes |
|
Fetch additional upstream dependencies (PyPi) |
|
Filter for Scanner - class exists primarily to silence mypy |
|
Base class for |
|
Load recipe from git (honoring active branches) |
|
Write recipe to per-recipe git branch |
|
Load the recipe from the filesystem |
|
Terminate pipeline after max_updates recipes have been updated. |
|
|
|
Source for Recipe objects to feed into Scanner |
|
Scans recipes and applies filters in asyncio loop |
|
Update source checksums |
|
Scan upstream for new releases and update recipe |
|
Write recipe to filesystem |
|
Documentation
- class bioconda_utils.autobump.RecipeSource(recipe_base, packages, exclude, shuffle=True)[source]¶
Source for Recipe objects to feed into Scanner
- class bioconda_utils.autobump.RecipeGraphSource(recipe_base, packages, exclude, shuffle, config, cache_fn=None)[source]¶
- class bioconda_utils.autobump.Scanner(recipe_source, cache_fn=None, status_fn=None)[source]¶
Scans recipes and applies filters in asyncio loop
- Parameters:
- recipe_source¶
recipe source
- req¶
async requests helper
- io_sem: Semaphore¶
semaphore to limit io parallelism
- conda_sem: Semaphore¶
must never run more than one conda at the same time (used by PyPi when running skeleton)
- filters: List[AsyncFilter]¶
the filters successively applied to each item
- class bioconda_utils.autobump.Filter(pipeline, *_args, **_kwargs)[source]¶
Filter for Scanner - class exists primarily to silence mypy
- class bioconda_utils.autobump.ExcludeOtherChannel(scanner, channels, cache)[source]¶
Exclude recipes in channels
- class bioconda_utils.autobump.ExcludeSubrecipe(scanner, always=False)[source]¶
Exclude sub-recipes
Unless always is True, subrecipes specifically enabled via
extra: watch: enable: yes
will not be filtered.
- class bioconda_utils.autobump.ExcludeDisabled(pipeline, *_args, **_kwargs)[source]¶
Exclude recipes disabled via config
- class bioconda_utils.autobump.ExcludeBlacklisted(scanner, recipe_base, config)[source]¶
Exclude blacklisted recipes
- class bioconda_utils.autobump.ExcludeDependencyPending(scanner, dag)[source]¶
Exclude recipes depending on packages in need of update
- class bioconda_utils.autobump.CheckPinning(scanner)[source]¶
Bump recipes in need of rebuild after pinning changes
- class bioconda_utils.autobump.UpdateVersion(scanner, hoster_factory, unparsed_file=None)[source]¶
Scan upstream for new releases and update recipe
In the most simple case, a package has a single URL which also indicates the version
Recipes may have alternative download locations => Update to newest available, disable others as comment
Recipes may have multiple sources => Ignore URLs not matching main recipe version
- stages:
select hoster based on source URL
hoster determines releases page url
fetch url
hoster extracts (link,version) pairs
select newest
update sources
- exception Metapackage(item, *args)[source]¶
This recipe only builds a meta package - it has no upstream sources.
- exception UpdateVersionFailure(item, *args)[source]¶
This recipe did not show the expected version number after updating.
Something probably went wrong trying to replace the version number in the meta.yaml.
- exception NoUrlInSource(item, *args)[source]¶
This recipe has a source without an URL.
Most likely, this is due to a git-url.
- exception NoRecognizedSourceUrl(item, *args)[source]¶
None of the source URLs in this recipe were recognized.
- exception UrlNotVersioned(item, *args)[source]¶
This recipe has an URL unaffected by the version change. the recipe
- exception NoReleases(item, *args)[source]¶
No releases at all were found for this recipe.
This is unusual - something probably went wrong finding releases for the source urls in this recipe.
- unparsed_file¶
output file name for failed urls
- hoster_factory¶
function selecting hoster
- build_config: Config¶
conda build config
- async get_versions(recipe, source, source_idx)[source]¶
Select hosters and retrieve versions for this source
- static select_version(current, versions)[source]¶
Chooses the most recent, acceptable version out of versions
must be newer than current (as defined by conda VersionOrder)
may only be a pre-release if current is pre-release (as defined by parse_version)
may only be “Legacy” (=strange) if current is Legacy (as defined by parse_version)
- Return type:
- class bioconda_utils.autobump.FetchUpstreamDependencies(scanner)[source]¶
Fetch additional upstream dependencies (PyPi)
This currently only affects PyPi. Dependencies for Python packages are determined by
setup.py
at installation time and need not be static in many cases (there is work to change this going on).- build_config: Config¶
conda build config
- class bioconda_utils.autobump.UpdateChecksums(scanner, failed_file=None)[source]¶
Update source checksums
If the recipe has had a version change, all sources will be downloaded, the checksums calculated and the recipe updated accordingly.
Automatically moves packages from md5 to sha256.
Aborts processing the recipe - if the checksum did not change - if there were no valid URLs - if checksums do not match among alternate URLs for a source
- Parameters:
failed_file (
Optional
[str
]) – Store the list of URLs for which downloading failed to this file upon pipeline completion.
- FIXME:
Should ignore sources for which no change has been made.
- exception NoValidUrls(item, *args)[source]¶
Failed to download any file for a source to generate new checksum
- exception SourceUrlMismatch(item, *args)[source]¶
The URLs in a source point to different files after update
- exception ChecksumReplaceFailed(item, *args)[source]¶
The checksum could not be updated after version bump
- failed_file¶
unparsed urls - for later inspection
- async update_source(recipe, source, source_idx)[source]¶
Updates one source
Each source has its own checksum, but all urls in one source must have the same checksum (if the link is available).
We don’t fail if the link is not available as cargo-port will only update after the recipe was built and uploaded
- Return type:
- class bioconda_utils.autobump.GitFilter(scanner, git_handler)[source]¶
Base class for
Filter
types needing access to the git repo- Parameters:
scanner (
Scanner
) – Scanner we are called fromgit_handler (
GitHandler
) – Instance of GitHandler for repo access
- git¶
The
GitHandler
class for accessing repo
- classmethod branch_name(recipe)[source]¶
Render branch name from recipe
Replace dashes with underscores (GitPython does not like dash)
Replace slashes with
.d/
slash as Git will use directories when separating parts with slashes, sobump/toolx
cannot be a branch at the same time asbump/toolx/1.2.x
. Note: this will break if we haverecipe/x
andrecipe/x.d
in the repo.
- Return type:
- class bioconda_utils.autobump.ExcludeNoActiveUpdate(scanner, git_handler)[source]¶
Exclude recipes not currently undergoing an update
Requires that the recipes are loaded from git - recipes with active updates are those for which a branch with commits newer than master exists (which is checked by GitLoadRecipe).
- class bioconda_utils.autobump.GitLoadRecipe(scanner, git_handler)[source]¶
Load recipe from git (honoring active branches)
We have three locations for the recipe: 1. master in upstream remote repo 2. auto_update_xxx in local repo 3. auto_update_xxx in origin remote repo
If the remote branch exists and is newer than master, we either have an active update that has not been merged yet, or an update for which the PR has been closed. We work from there so that we don’t recreate the “same” update.
Always remove the local branch - it’s ephemereal.
If there is a remote branch that has not been superceded by master, work from there.
Otherwise load master
- class bioconda_utils.autobump.GitWriteRecipe(scanner, git_handler)[source]¶
Write recipe to per-recipe git branch
- class bioconda_utils.autobump.CreatePullRequest(scanner, git_handler, github_handler)[source]¶
Create or Update PR on GitHub
- exception UpdateRejected(item, *args)[source]¶
This update has been rejected previously on Github
A PR created by Autobump for this recipe and this version already exists on Github and has been closed. We take this as indication that the update should not be repeated
- exception FailedToCreatePR(item, *args)[source]¶
Something went wrong trying to create a new PR on Github.
- static render_deps_diff(recipe)[source]¶
Renders a “diff” of the recipes upstream dependencies.
This relies on the ‘depends’ data structure in the recipe’s
version_data
. This structure is expected to be a dict with two keys ‘host’ and ‘run’, each of which contain dict of package to version dependency mappings. The ‘depends’ data can be created by the hoster (currently done by CPAN and CRAN) or filled in later by theFetchUpstreamDependencies
filter (currently for PyPi).