Updates packages when new upstream releases become available

Overview:

• The Scanner initializes the asyncio loop and for each recipe, loads the meta.yaml using Recipe and executes its Filter objects for each recipe.

• The Recipe handles reading, modification and writing of meta.yaml files.

• The filters ExcludeSubrecipe and ExcludeOtherChannel exclude recipes in sub folders and present in other channels as configured.

• The filter ExcludeNoActiveUpdate excludes packages that don’t already have an ongoing autobump update.

• The filters LoadRecipe and GitLoadRecipe fill the stub Recipe with content.

• The filter UpdateVersion determines available upstream verions using Hoster and its subclasses, selects the most recent, acceptable version and uses Recipe to replace

• The filter FetchUpstreamDependencies gathers additional dependencies not found by the UpdateVersion filter. (Currently, that’s only trying to get Python deps by loading the setup.py)

• The filter UpdateChecksums downloads the modified source URLs and updates the checksums for each source of a recipe.

• The filter WriteRecipe or the filter GitWriteRecipe are used to write the recipe back, either to the current folder structure, or the git repo and creating a branch.

• The filter CreatePullRequest creates pull requests on GitHub for changes made to per-recipe-branches.

• The filter MaxUpdates terminates the loop after a number of recipes have come past it, to avoid too many PRs opened at once.

Classes

AutoBumpConfigMixin()
CheckPinning(scanner, bump_only_python)	Bump recipes in need of rebuild after pinning changes
CreatePullRequest(scanner, git_handler, …)	Create or Update PR on GitHub
ExcludeBlacklisted(scanner, recipe_base, config)	Exclude blacklisted recipes
ExcludeDependencyPending(scanner, dag)	Exclude recipes depending on packages in need of update
ExcludeDisabled(pipeline, *_args, **_kwargs)	Exclude recipes disabled via config
ExcludeNoActiveUpdate(scanner, git_handler)	Exclude recipes not currently undergoing an update
ExcludeOtherChannel(scanner, channels, cache)	Exclude recipes in channels
ExcludeSubrecipe(scanner[, always])	Exclude sub-recipes
FetchUpstreamDependencies(scanner)	Fetch additional upstream dependencies (PyPi)
Filter(pipeline, *_args, **_kwargs)	Filter for Scanner - class exists primarily to silence mypy
GitFilter(scanner, git_handler)	Base class for Filter types needing access to the git repo
GitLoadRecipe(scanner, git_handler)	Load recipe from git (honoring active branches)
GitWriteRecipe(scanner, git_handler)	Write recipe to per-recipe git branch
LoadRecipe(scanner)	Load the recipe from the filesystem
MaxUpdates(scanner[, max_updates])	Terminate pipeline after max_updates recipes have been updated.
RecipeGraphSource(recipe_base, packages, …)
RecipeSource(recipe_base, packages, exclude)	Source for Recipe objects to feed into Scanner
Scanner(recipe_source[, cache_fn, status_fn])	Scans recipes and applies filters in asyncio loop
UpdateChecksums(scanner[, failed_file])	Update source checksums
UpdateVersion(scanner, hoster_factory[, …])	Scan upstream for new releases and update recipe
WriteRecipe(scanner)	Write recipe to filesystem
WritingFilter()

Documentation

class bioconda_utils.autobump.CheckPinning(scanner, bump_only_python)

Bump recipes in need of rebuild after pinning changes

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

static match_version(spec, version)

Check if version satisfies spec

>>> match_version('>1.2,<1.3.0a0', '1.2.1')
True

Args: spec: The version range specification version: The version to test against the spec Returns: True if the spec is satisfied by the version.

class bioconda_utils.autobump.CreatePullRequest(scanner, git_handler, github_handler)

Create or Update PR on GitHub

exception FailedToCreatePR(item, *args)

Something went wrong trying to create a new PR on Github.

exception UpdateInProgress(item, *args)

The recipe has an active update PR

exception UpdateRejected(item, *args)

This update has been rejected previously on Github

A PR created by Autobump for this recipe and this version already exists on Github and has been closed. We take this as indication that the update should not be repeated

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

async async_init()

Create gidget GithubAPI object from session

static get_github_author(recipe)

Fetches upstream Github account name if applicable

For recipes with upstream sources hosted on Github, we can extract the account name from the source URL, so that we can add an @mention notifying the upstream author of the update.

Return type

Optional[str]

static render_deps_diff(recipe)

Renders a “diff” of the recipes upstream dependencies.

This relies on the ‘depends’ data structure in the recipe’s version_data. This structure is expected to be a dict with two keys ‘host’ and ‘run’, each of which contain dict of package to version dependency mappings. The ‘depends’ data can be created by the hoster (currently done by CPAN and CRAN) or filled in later by the FetchUpstreamDependencies filter (currently for PyPi).

class bioconda_utils.autobump.ExcludeBlacklisted(scanner, recipe_base, config)

Exclude blacklisted recipes

exception Blacklisted(item, *args)

This recipe has been blacklisted (fails to build, see blacklist file)

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

get_info()

Return description of filter for logging

Return type

str

class bioconda_utils.autobump.ExcludeDependencyPending(scanner, dag)

Exclude recipes depending on packages in need of update

exception DependencyPending(item, *args)

A dependency of this recipe is pending rebuild

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

class bioconda_utils.autobump.ExcludeDisabled(pipeline, *_args, **_kwargs)

Exclude recipes disabled via config

exception IsDisabled(item, *args)

This recipe was explicitly disabled

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

class bioconda_utils.autobump.ExcludeNoActiveUpdate(scanner, git_handler)

Exclude recipes not currently undergoing an update

Requires that the recipes are loaded from git - recipes with active updates are those for which a branch with commits newer than master exists (which is checked by GitLoadRecipe).

exception NoActiveUpdate(item, *args)

Recipe is not currently being updated

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

class bioconda_utils.autobump.ExcludeOtherChannel(scanner, channels, cache)

Exclude recipes in channels

exception OtherChannel(item, *args)

This recipe builds one or more packages that are also present in at least one other channel

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

get_info()

Return description of filter for logging

Return type

str

class bioconda_utils.autobump.ExcludeSubrecipe(scanner, always=False)

Exclude sub-recipes

Unless always is True, subrecipes specifically enabled via extra: watch: enable: yes will not be filtered.

exception IsSubRecipe(item, *args)

This recipe is a sub-recipe (i.e. blast/2.4.0).

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

class bioconda_utils.autobump.FetchUpstreamDependencies(scanner)

Fetch additional upstream dependencies (PyPi)

This currently only affects PyPi. Dependencies for Python packages are determined by setup.py at installation time and need not be static in many cases (there is work to change this going on).

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

build_config: conda_build.config.Config

conda build config

class bioconda_utils.autobump.Filter(pipeline, *_args, **_kwargs)

Filter for Scanner - class exists primarily to silence mypy

abstract async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

get_info()

Return description of filter for logging

Return type

str

class bioconda_utils.autobump.GitFilter(scanner, git_handler)

Base class for Filter types needing access to the git repo

Parameters

• scanner (Scanner) – Scanner we are called from

• git_handler (GitHandler) – Instance of GitHandler for repo access

abstract async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

classmethod branch_name(recipe)

Render branch name from recipe

• Replace dashes with underscores (GitPython does not like dash)

• Replace slashes with .d/ slash as Git will use directories when separating parts with slashes, so bump/toolx cannot be a branch at the same time as bump/toolx/1.2.x. Note: this will break if we have recipe/x and recipe/x.d in the repo.

Return type

str

git

The GitHandler class for accessing repo

class bioconda_utils.autobump.GitLoadRecipe(scanner, git_handler)

Load recipe from git (honoring active branches)

We have three locations for the recipe: 1. master in upstream remote repo 2. auto_update_xxx in local repo 3. auto_update_xxx in origin remote repo

If the remote branch exists and is newer than master, we either have an active update that has not been merged yet, or an update for which the PR has been closed. We work from there so that we don’t recreate the “same” update.

• Always remove the local branch - it’s ephemereal.

• If there is a remote branch that has not been superceded by master, work from there.

• Otherwise load master

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

class bioconda_utils.autobump.GitWriteRecipe(scanner, git_handler)

Write recipe to per-recipe git branch

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

class bioconda_utils.autobump.LoadRecipe(scanner)

Load the recipe from the filesystem

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

class bioconda_utils.autobump.MaxUpdates(scanner, max_updates=0)

Terminate pipeline after max_updates recipes have been updated.

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

get_info()

Return description of filter for logging

Return type

str

class bioconda_utils.autobump.RecipeGraphSource(recipe_base, packages, exclude, shuffle, config, cache_fn=None)

class bioconda_utils.autobump.RecipeSource(recipe_base, packages, exclude, shuffle=True)

Source for Recipe objects to feed into Scanner

Parameters

• packages (List[str]) – list of packages, may contain globs

• shuffle (bool) – If true, package order will be randomized.

class bioconda_utils.autobump.Scanner(recipe_source, cache_fn=None, status_fn=None)

Scans recipes and applies filters in asyncio loop

Parameters

• recipe_source (Iterable[Recipe]) – Iteratable providing Recipe stubs

• cache_fn (Optional[str]) – Filename prefix for caching

• status_fn (Optional[str]) – Filename for status output

async process(recipe)

Applies the filters to a recipe

Return type

bool

recipe_source

recipe source

req

async requests helper

run()

Runs scanner

Return type

bool

stats: Counter

counter to gather stats on various states

status: List[Tuple[str, str]]

collect end status for each recipe

status_fn: str

filename to write statuses to

class bioconda_utils.autobump.UpdateChecksums(scanner, failed_file=None)

Update source checksums

If the recipe has had a version change, all sources will be downloaded, the checksums calculated and the recipe updated accordingly.

• Automatically moves packages from md5 to sha256.

• Aborts processing the recipe - if the checksum did not change - if there were no valid URLs - if checksums do not match among alternate URLs for a source

Parameters

failed_file (Optional[str]) – Store the list of URLs for which downloading failed to this file upon pipeline completion.

FIXME:

Should ignore sources for which no change has been made.

exception ChecksumReplaceFailed(item, *args)

The checksum could not be updated after version bump

exception ChecksumUnchanged(item, *args)

The checksum did not change after version bump

exception NoValidUrls(item, *args)

Failed to download any file for a source to generate new checksum

exception SourceUrlMismatch(item, *args)

The URLs in a source point to different files after update

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

failed_file

unparsed urls - for later inspection

failed_urls: List[str]

failed urls - for later inspection

finalize()

Store list of URLs that could not be downloaded

async update_source(recipe, source, source_idx)

Updates one source

Each source has its own checksum, but all urls in one source must have the same checksum (if the link is available).

We don’t fail if the link is not available as cargo-port will only update after the recipe was built and uploaded

Return type

None

class bioconda_utils.autobump.UpdateVersion(scanner, hoster_factory, unparsed_file=None)

Scan upstream for new releases and update recipe

• In the most simple case, a package has a single URL which also indicates the version

• Recipes may have alternative download locations => Update to newest available, disable others as comment

• Recipes may have multiple sources => Ignore URLs not matching main recipe version

stages:

1. select hoster based on source URL

2. hoster determines releases page url

3. fetch url

4. hoster extracts (link,version) pairs

5. select newest

6. update sources

exception Metapackage(item, *args)

This recipe only builds a meta package - it has no upstream sources.

exception NoRecognizedSourceUrl(item, *args)

None of the source URLs in this recipe were recognized.

exception NoReleases(item, *args)

No releases at all were found for this recipe.

This is unusual - something probably went wrong finding releases for the source urls in this recipe.

exception NoUrlInSource(item, *args)

This recipe has a source without an URL.

Most likely, this is due to a git-url.

exception UpToDate(item, *args)

This recipe appears to be up to date!

exception UpdateVersionFailure(item, *args)

This recipe did not show the expected version number after updating.

Something probably went wrong trying to replace the version number in the meta.yaml.

exception UrlNotVersioned(item, *args)

This recipe has an URL unaffected by the version change. the recipe

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None

build_config: conda_build.config.Config

conda build config

check_version_pin_conflict(recipe, versions)

Find items in versions conflicting with pins

Example

If ggplot 7.0 needs r-base 4.0, but our pin is 3.5.1, it conflicts

TODO: currently, this only logs an error

Return type

None

finalize()

Save unparsed urls to file and print stats to log

Return type

None

async get_version_map(recipe)

Scan all source urls

async get_versions(recipe, source, source_idx)

Select hosters and retrieve versions for this source

hoster_factory

function selecting hoster

static select_version(current, versions)

Chooses the most recent, acceptable version out of versions

• must be newer than current (as defined by conda VersionOrder)

• may only be a pre-release if current is pre-release (as defined by parse_version)

• may only be “Legacy” (=strange) if current is Legacy (as defined by parse_version)

Return type

str

unparsed_file

output file name for failed urls

unparsed_urls: List[str]

output file name for unparsed urls

class bioconda_utils.autobump.WriteRecipe(scanner)

Write recipe to filesystem

async apply(recipe)

Process a recipe. Returns False if processing should stop

Return type

None