autobump¶
Updates packages when new upstream releases become available
Overview:
The
Scanner
initializes theasyncio
loop and for each recipe, loads themeta.yaml
usingRecipe
and executes itsFilter
objects for each recipe.The
Recipe
handles reading, modification and writing ofmeta.yaml
files.The filters
ExcludeSubrecipe
andExcludeOtherChannel
exclude recipes in sub folders and present in other channels as configured.The filter
ExcludeNoActiveUpdate
excludes packages that don’t already have an ongoing autobump update.The filters
LoadRecipe
andGitLoadRecipe
fill the stub Recipe with content.The filter
UpdateVersion
determines available upstream verions usingHoster
and its subclasses, selects the most recent, acceptable version and usesRecipe
to replaceThe filter
FetchUpstreamDependencies
gathers additional dependencies not found by theUpdateVersion
filter. (Currently, that’s only trying to get Python deps by loading thesetup.py
)The filter
UpdateChecksums
downloads the modified source URLs and updates the checksums for each source of a recipe.The filter
WriteRecipe
or the filterGitWriteRecipe
are used to write the recipe back, either to the current folder structure, or the git repo and creating a branch.The filter
CreatePullRequest
creates pull requests on GitHub for changes made to per-recipe-branches.The filter
MaxUpdates
terminates the loop after a number of recipes have come past it, to avoid too many PRs opened at once.
Classes
|
|
|
Bump recipes in need of rebuild after pinning changes |
|
Create or Update PR on GitHub |
|
Exclude blacklisted recipes |
|
Exclude recipes depending on packages in need of update |
|
Exclude recipes disabled via config |
|
Exclude recipes not currently undergoing an update |
|
Exclude recipes in channels |
|
Exclude sub-recipes |
|
Fetch additional upstream dependencies (PyPi) |
|
Filter for Scanner - class exists primarily to silence mypy |
|
Base class for |
|
Load recipe from git (honoring active branches) |
|
Write recipe to per-recipe git branch |
|
Load the recipe from the filesystem |
|
Terminate pipeline after max_updates recipes have been updated. |
|
|
|
Source for Recipe objects to feed into Scanner |
|
Scans recipes and applies filters in asyncio loop |
|
Update source checksums |
|
Scan upstream for new releases and update recipe |
|
Write recipe to filesystem |
|
Documentation
-
class
bioconda_utils.autobump.
CheckPinning
(scanner)[source]¶ Bump recipes in need of rebuild after pinning changes
-
class
bioconda_utils.autobump.
CreatePullRequest
(scanner, git_handler, github_handler)[source]¶ Create or Update PR on GitHub
-
exception
FailedToCreatePR
(item, *args)[source]¶ Something went wrong trying to create a new PR on Github.
-
exception
UpdateRejected
(item, *args)[source]¶ This update has been rejected previously on Github
A PR created by Autobump for this recipe and this version already exists on Github and has been closed. We take this as indication that the update should not be repeated
Fetches upstream Github account name if applicable
For recipes with upstream sources hosted on Github, we can extract the account name from the source URL, so that we can add an @mention notifying the upstream author of the update.
-
static
render_deps_diff
(recipe)[source]¶ Renders a “diff” of the recipes upstream dependencies.
This relies on the ‘depends’ data structure in the recipe’s
version_data
. This structure is expected to be a dict with two keys ‘host’ and ‘run’, each of which contain dict of package to version dependency mappings. The ‘depends’ data can be created by the hoster (currently done by CPAN and CRAN) or filled in later by theFetchUpstreamDependencies
filter (currently for PyPi).
-
exception
-
class
bioconda_utils.autobump.
ExcludeBlacklisted
(scanner, recipe_base, config)[source]¶ Exclude blacklisted recipes
-
exception
Blacklisted
(item, *args)[source]¶ This recipe has been blacklisted (fails to build, see blacklist file)
-
exception
-
class
bioconda_utils.autobump.
ExcludeDependencyPending
(scanner, dag)[source]¶ Exclude recipes depending on packages in need of update
-
class
bioconda_utils.autobump.
ExcludeDisabled
(pipeline, *_args, **_kwargs)[source]¶ Exclude recipes disabled via config
-
class
bioconda_utils.autobump.
ExcludeNoActiveUpdate
(scanner, git_handler)[source]¶ Exclude recipes not currently undergoing an update
Requires that the recipes are loaded from git - recipes with active updates are those for which a branch with commits newer than master exists (which is checked by GitLoadRecipe).
-
class
bioconda_utils.autobump.
ExcludeOtherChannel
(scanner, channels, cache)[source]¶ Exclude recipes in channels
-
exception
OtherChannel
(item, *args)[source]¶ This recipe builds one or more packages that are also present in at least one other channel
-
exception
-
class
bioconda_utils.autobump.
ExcludeSubrecipe
(scanner, always=False)[source]¶ Exclude sub-recipes
Unless always is True, subrecipes specifically enabled via
extra: watch: enable: yes
will not be filtered.
-
class
bioconda_utils.autobump.
FetchUpstreamDependencies
(scanner)[source]¶ Fetch additional upstream dependencies (PyPi)
This currently only affects PyPi. Dependencies for Python packages are determined by
setup.py
at installation time and need not be static in many cases (there is work to change this going on).-
build_config
: conda_build.config.Config¶ conda build config
-
-
class
bioconda_utils.autobump.
Filter
(pipeline, *_args, **_kwargs)[source]¶ Filter for Scanner - class exists primarily to silence mypy
-
class
bioconda_utils.autobump.
GitFilter
(scanner, git_handler)[source]¶ Base class for
Filter
types needing access to the git repo- Parameters
scanner (
Scanner
) – Scanner we are called fromgit_handler (
GitHandler
) – Instance of GitHandler for repo access
-
abstract async
apply
(recipe)[source]¶ Process a recipe. Returns False if processing should stop
- Return type
-
classmethod
branch_name
(recipe)[source]¶ Render branch name from recipe
Replace dashes with underscores (GitPython does not like dash)
Replace slashes with
.d/
slash as Git will use directories when separating parts with slashes, sobump/toolx
cannot be a branch at the same time asbump/toolx/1.2.x
. Note: this will break if we haverecipe/x
andrecipe/x.d
in the repo.
- Return type
-
git
¶ The
GitHandler
class for accessing repo
-
class
bioconda_utils.autobump.
GitLoadRecipe
(scanner, git_handler)[source]¶ Load recipe from git (honoring active branches)
We have three locations for the recipe: 1. master in upstream remote repo 2. auto_update_xxx in local repo 3. auto_update_xxx in origin remote repo
If the remote branch exists and is newer than master, we either have an active update that has not been merged yet, or an update for which the PR has been closed. We work from there so that we don’t recreate the “same” update.
Always remove the local branch - it’s ephemereal.
If there is a remote branch that has not been superceded by master, work from there.
Otherwise load master
-
class
bioconda_utils.autobump.
GitWriteRecipe
(scanner, git_handler)[source]¶ Write recipe to per-recipe git branch
-
class
bioconda_utils.autobump.
MaxUpdates
(scanner, max_updates=0)[source]¶ Terminate pipeline after max_updates recipes have been updated.
-
class
bioconda_utils.autobump.
RecipeGraphSource
(recipe_base, packages, exclude, shuffle, config, cache_fn=None)[source]¶
-
class
bioconda_utils.autobump.
RecipeSource
(recipe_base, packages, exclude, shuffle=True)[source]¶ Source for Recipe objects to feed into Scanner
-
class
bioconda_utils.autobump.
Scanner
(recipe_source, cache_fn=None, status_fn=None)[source]¶ Scans recipes and applies filters in asyncio loop
- Parameters
-
recipe_source
¶ recipe source
-
req
¶ async requests helper
-
stats
: collections.Counter¶ counter to gather stats on various states
-
class
bioconda_utils.autobump.
UpdateChecksums
(scanner, failed_file=None)[source]¶ Update source checksums
If the recipe has had a version change, all sources will be downloaded, the checksums calculated and the recipe updated accordingly.
Automatically moves packages from md5 to sha256.
Aborts processing the recipe - if the checksum did not change - if there were no valid URLs - if checksums do not match among alternate URLs for a source
- Parameters
failed_file (
Optional
[str
]) – Store the list of URLs for which downloading failed to this file upon pipeline completion.
- FIXME:
Should ignore sources for which no change has been made.
-
exception
ChecksumReplaceFailed
(item, *args)[source]¶ The checksum could not be updated after version bump
-
exception
NoValidUrls
(item, *args)[source]¶ Failed to download any file for a source to generate new checksum
-
exception
SourceUrlMismatch
(item, *args)[source]¶ The URLs in a source point to different files after update
-
failed_file
¶ unparsed urls - for later inspection
-
async
update_source
(recipe, source, source_idx)[source]¶ Updates one source
Each source has its own checksum, but all urls in one source must have the same checksum (if the link is available).
We don’t fail if the link is not available as cargo-port will only update after the recipe was built and uploaded
- Return type
-
class
bioconda_utils.autobump.
UpdateVersion
(scanner, hoster_factory, unparsed_file=None)[source]¶ Scan upstream for new releases and update recipe
In the most simple case, a package has a single URL which also indicates the version
Recipes may have alternative download locations => Update to newest available, disable others as comment
Recipes may have multiple sources => Ignore URLs not matching main recipe version
- stages:
select hoster based on source URL
hoster determines releases page url
fetch url
hoster extracts (link,version) pairs
select newest
update sources
-
exception
Metapackage
(item, *args)[source]¶ This recipe only builds a meta package - it has no upstream sources.
-
exception
NoRecognizedSourceUrl
(item, *args)[source]¶ None of the source URLs in this recipe were recognized.
-
exception
NoReleases
(item, *args)[source]¶ No releases at all were found for this recipe.
This is unusual - something probably went wrong finding releases for the source urls in this recipe.
-
exception
NoUrlInSource
(item, *args)[source]¶ This recipe has a source without an URL.
Most likely, this is due to a git-url.
-
exception
UpdateVersionFailure
(item, *args)[source]¶ This recipe did not show the expected version number after updating.
Something probably went wrong trying to replace the version number in the meta.yaml.
-
exception
UrlNotVersioned
(item, *args)[source]¶ This recipe has an URL unaffected by the version change. the recipe
-
build_config
: conda_build.config.Config¶ conda build config
-
check_version_pin_conflict
(recipe, versions)[source]¶ Find items in versions conflicting with pins
Example
If ggplot 7.0 needs r-base 4.0, but our pin is 3.5.1, it conflicts
TODO: currently, this only logs an error
- Return type
-
async
get_versions
(recipe, source, source_idx)[source]¶ Select hosters and retrieve versions for this source
-
hoster_factory
¶ function selecting hoster
-
static
select_version
(current, versions)[source]¶ Chooses the most recent, acceptable version out of versions
must be newer than current (as defined by conda VersionOrder)
may only be a pre-release if current is pre-release (as defined by parse_version)
may only be “Legacy” (=strange) if current is Legacy (as defined by parse_version)
- Return type
-
unparsed_file
¶ output file name for failed urls