Low-level API

This API is built on top of Git and Git-LFS.

Renku repository management.

class renku.core.management.LocalClient(path=<function default_path>, renku_home='.renku', parent=None, external_storage_requested=True, *, data_dir='data')

A low-level client for communicating with a local Renku repository.

Datasets

Client for handling datasets.

class renku.core.management.datasets.DatasetsApiMixin

Client for handling datasets.

CACHE = 'cache'

Directory to cache transient data.

DATASETS = 'datasets'

Directory for storing dataset metadata in Renku.

POINTERS = 'pointers'

Directory for storing external pointer files.

add_data_to_dataset(dataset, urls, force=False, overwrite=False, sources=(), destination='', ref=None, external=False, extract=False, all_at_once=False, destination_names=None, progress=None)

Import the data into the data directory.

add_dataset_tag(dataset, tag, description='', force=False)

Adds a new tag to a dataset.

Validates if the tag already exists and that the tag follows the same rules as docker tags. See https://docs.docker.com/engine/reference/commandline/tag/ for a documentation of docker tag syntax.

Raises

errors.ParameterError

create_dataset(name=None, title=None, description=None, creators=None, keywords=None)

Create a dataset.

dataset_commits(dataset, max_results=None)

Gets the newest commit for a dataset or its files.

Commits are returned sorted from newest to oldest.

property datasets

Return mapping from path to dataset.

datasets_from_commit(commit=None)

Return datasets defined in a commit.

get_dataset_path(name)

Get dataset path from name.

get_relative_url(url)

Determine if the repo url should be relative.

has_external_files()

Return True if project has external files.

load_dataset(name=None)

Load dataset reference file.

load_dataset_from_path(path, commit=None)

Return a dataset from a given path.

prepare_git_repo(url, ref=None)

Clone and cache a Git repo.

remove_dataset_tags(dataset, tags)

Removes tags from a dataset.

remove_file(filepath)

Remove a file/symlink and its pointer file (for external files).

property renku_datasets_path

Return a Path instance of Renku dataset metadata folder.

property renku_pointers_path

Return a Path instance of Renku pointer files folder.

update_dataset_files(files, ref, delete=False)

Update files and dataset metadata according to their remotes.

Parameters

• files – List of files to be updated

• delete – Indicates whether to delete files or not

Returns

List of files that should be deleted

update_external_files(records)

Update files linked to external storage.

with_dataset(name=None, create=False)

Yield an editable metadata object for a dataset.

class renku.core.management.datasets.DownloadProgressCallback(description, total_size)

Interface to report various stages of a download.

Default initializer.

finalize()

Called once when the download is finished.

update(size)

Update the status.

Repository

Client for handling a local repository.

class renku.core.management.repository.PathMixin(path=<function default_path>)

Define a default path attribute.

class renku.core.management.repository.RepositoryApiMixin(renku_home='.renku', parent=None, *, data_dir='data')

Client for handling a local repository.

ACTIVITY_INDEX = 'activity_index.yaml'

Caches activities that generated a path.

LOCK_SUFFIX = '.lock'

Default suffix for Renku lock file.

METADATA = 'metadata.yml'

Default name of Renku config file.

WORKFLOW = 'workflow'

Directory for storing workflow in Renku.

activities_for_paths(paths, file_commit=None, revision='HEAD')

Get all activities involving a path.

property activity_index_path

Path to the activity filepath cache.

add_to_activity_index(activity)

Add an activity and it’s generations to the cache.

property cwl_prefix

Return a CWL prefix.

data_dir

Define a name of the folder for storing datasets.

find_previous_commit(paths, revision='HEAD', return_first=False, full=False)

Return a previous commit for a given path starting from revision.

Parameters

• revision – revision to start from, defaults to HEAD

• return_first – show the first commit in the history

• full – return full history

Raises

KeyError – if path is not present in the given commit

import_from_template(template_path, metadata, force=False)

Render template files from a template directory.

init_repository(force=False, user=None)

Initialize an empty Renku repository.

is_project_set()

Return if project is set for the client.

is_workflow(path)

Check if the path is a valid CWL file.

property lock

Create a Renku config lock.

parent

Store a pointer to the parent repository.

property path_activity_cache

Cache of all activities and their generated paths.

process_commit(commit=None, path=None)

Build an Activity.

Parameters

• commit – Commit to process. (default: HEAD)

• path – Process a specific CWL file.

property project

Return the Project instance.

property remote

Return host, owner and name of the remote if it exists.

renku_home

Define a name of the Renku folder (default: .renku).

property renku_metadata_path

Return a Path instance of Renku metadata file.

renku_path

Store a Path instance of the Renku folder.

resolve_in_submodules(commit, path)

Resolve filename in submodules.

subclients(parent_commit)

Return mapping from submodule to client.

property submodules

Return list of submodules it belongs to.

with_commit(commit)

Yield the state of the repo at a specific commit.

with_metadata(read_only=False, name=None)

Yield an editable metadata object.

with_workflow_storage()

Yield a workflow storage.

property workflow_names

Return index of workflow names.

property workflow_path

Return a Path instance of the workflow folder.

renku.core.management.repository.default_path()

Return default repository path.

renku.core.management.repository.path_converter(path)

Converter for path in PathMixin.

Git Internals

Wrap Git client.

class renku.core.management.git.GitCore

Wrap Git client.

property candidate_paths

Return all paths in the index and untracked files.

commit(commit_only=None, commit_empty=True, raise_if_empty=False, commit_message=None)

Automatic commit.

property dirty_paths

Get paths of dirty files in the repository.

ensure_clean(ignore_std_streams=False)

Make sure the repository is clean.

ensure_unstaged(path)

Ensure that path is not part of git staged files.

ensure_untracked(path)

Ensure that path is not part of git untracked files.

find_attr(*paths)

Return map with path and its attributes.

find_ignored_paths(*paths)

Return ignored paths matching .gitignore file.

property modified_paths

Return paths of modified files.

remove_unmodified(paths, autocommit=True)

Remove unmodified paths and return their names.

repo

Store an instance of the Git repository.

setup_credential_helper()

Setup git credential helper to cache if not set already.

transaction(clean=True, commit=True, commit_empty=True, commit_message=None, commit_only=None, ignore_std_streams=False, raise_if_empty=False, up_to_date=False)

Perform Git checks and operations.

worktree(path=None, branch_name=None, commit=None, merge_args='--ff-only')

Create new worktree.

Git utilities.

class renku.core.models.git.GitURL(href, pathname=None, protocol='ssh', hostname='localhost', username=None, password=None, port=None, owner=None, name=None, regex=None)

Parser for common Git URLs.

property image

Return image name.

classmethod parse(href)

Derive basic informations.

class renku.core.models.git.Range(start, stop)

Represent parsed Git revision as an interval.

classmethod rev_parse(git, revision)

Parse revision string.

renku.core.models.git.filter_repo_name(repo_name)

Remove the .git extension from the repo name.