Metadata-Version: 2.1
Name: dbnomics-fetcher-ops
Version: 0.4.12
Summary: Manage DBnomics fetchers
License: AGPL-3.0-or-later
Author: Christophe Benz
Author-email: christophe.benz@nomics.world
Requires-Python: >=3.8,<4.0
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: boltons (>=21.0.0,<22.0.0)
Requires-Dist: daiquiri (>=3.0.1,<4.0.0)
Requires-Dist: dbnomics-solr (>=1.1.2,<2.0.0)
Requires-Dist: pydantic (>=1.8.1,<2.0.0)
Requires-Dist: python-dotenv (>=0.20.0,<0.21.0)
Requires-Dist: python-gitlab (>=2.4.0,<3.0.0)
Requires-Dist: requests (>=2.24.0,<3.0.0)
Requires-Dist: ruamel.yaml (>=0.16.10,<0.17.0)
Requires-Dist: typer (>=0.4.1,<0.5.0)
Requires-Dist: validators (>=0.20.0,<0.21.0)
Requires-Dist: xdg (>=5.1.1,<6.0.0)
Description-Content-Type: text/markdown

# DBnomics fetcher ops

Manage DBnomics fetchers.

## Usage

### Install

```bash
pip install dbnomics-fetcher-ops
```

### Configure a fetcher

Configure:

- GitLab private token: use `--gitlab-private-token` option or `GITLAB_PRIVATE_TOKEN` environment variable. The private token can be a [personal access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html). It must have the `api` scope.

Run:

```bash
dbnomics-fetchers -v configure scsmich --dry-run
# If everything seems OK, remove the --dry-run flag:
dbnomics-fetchers -v configure scsmich
```

### List fetchers

```bash
dbnomics-fetchers -v list
```

### Run fetcher pipelines

```bash
# Replace PROVIDER_SLUG by the real value:
dbnomics-fetchers -v run --provider-slug PROVIDER_SLUG

# To run a pipeline for each fetcher:
dbnomics-fetchers -v list --slug | xargs -I {} dbnomics-fetchers -v run --provider-slug {}
```

## Development

This repository uses [Poetry](https://python-poetry.org/).

```bash
# git clone repo or fork
cd dbnomics-fetcher-ops
poetry install
cp .env.example .env
```

Run commands with:

```bash
poetry run dbnomics-fetchers COMMAND
```

To use ipdb:

```bash
poetry shell
# Find venv dir with "which python"
ipdb3 /path/to/venv/bin/dbnomics-fetchers ...
```

