Metadata-Version: 2.1
Name: castor-extractor
Version: 0.2.3
Summary: Extract your metadata assets.
Home-page: https://www.castordoc.com/
License: EULA
Author: Castor
Author-email: support@castordoc.com
Requires-Python: >=3.7,<3.11
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3
Provides-Extra: all
Provides-Extra: bigquery
Provides-Extra: looker
Provides-Extra: metabase
Provides-Extra: qlik
Provides-Extra: redshift
Provides-Extra: snowflake
Provides-Extra: tableau
Requires-Dist: cachetools (>=4.2.4,<5.0.0)
Requires-Dist: certifi (==2021.10.8)
Requires-Dist: charset-normalizer (>=2.0.7,<3.0.0)
Requires-Dist: click (>=8.0,<8.1)
Requires-Dist: google-api-core (>=2.1.1,<3.0.0)
Requires-Dist: google-auth (>=1.6.3,<3.0.0)
Requires-Dist: google-cloud-bigquery (>=2.0.0,<3.0.0)
Requires-Dist: google-cloud-bigquery-storage (>=2.0.0,<3.0.0)
Requires-Dist: google-cloud-core (>=2.1.0,<3.0.0)
Requires-Dist: google-cloud-storage (>=1.42.3,<2.0.0)
Requires-Dist: google-crc32c (>=1.3.0,<2.0.0)
Requires-Dist: google-resumable-media (>=2.0.3)
Requires-Dist: googleapis-common-protos (>=1.53.0,<7.0.0)
Requires-Dist: idna (>=2.5,<2.9)
Requires-Dist: looker-sdk (>=22.4.0,<=23.0.0); extra == "looker" or extra == "all"
Requires-Dist: msal (>=1.20.0,<2.0.0)
Requires-Dist: protobuf (>=2.0.0,<4.0.0)
Requires-Dist: psycopg2-binary (>=2.0.0,<3.0.0); extra == "metabase" or extra == "redshift" or extra == "all"
Requires-Dist: pyarrow (>=6.0.0,<7.0.0)
Requires-Dist: pyasn1 (>=0.4.8,<1.0.0)
Requires-Dist: pyasn1-modules (>=0.2.8,<1.0.0)
Requires-Dist: pycryptodome (>=3.0.0,<4.0.0)
Requires-Dist: python-dateutil (>=2.0.0,<=3.0.0)
Requires-Dist: pytz (==2021.3)
Requires-Dist: requests (>=2.0.0,<3.0.0)
Requires-Dist: rsa (>=4.0.0,<5.0.0)
Requires-Dist: setuptools (>=65.6.3,<66.0.0)
Requires-Dist: six (>=1.16.0,<2.0.0)
Requires-Dist: snowflake-connector-python (>=2.6.2,!=2.7.10); extra == "snowflake" or extra == "all"
Requires-Dist: snowflake-sqlalchemy (!=1.2.5,<2.0.0); extra == "snowflake" or extra == "all"
Requires-Dist: sqlalchemy (>=1.4,<1.5)
Requires-Dist: sqlalchemy-bigquery (>=1.0.0,<=2.0.0); extra == "bigquery" or extra == "all"
Requires-Dist: sqlalchemy-redshift (==0.8.2); extra == "redshift" or extra == "all"
Requires-Dist: tableauserverclient (==0.17.0); extra == "tableau" or extra == "all"
Requires-Dist: tqdm (>=4.0.0,<5.0.0)
Requires-Dist: typing-extensions (>=4.0.0)
Requires-Dist: urllib3 (>=1.0.0,<2.0.0)
Requires-Dist: websocket-client (>=0,<1); extra == "qlik" or extra == "all"
Project-URL: Documentation, https://docs.castordoc.com/castor-package/castor-extractor
Description-Content-Type: text/markdown

# Castor Extractor <img src="https://app.castordoc.com/images/castor_icon_dark.svg" width=30 />

This library contains utilities to extract your metadata assets into `JSON` or `CSV` files, on your local machine.
After extraction, those files can be pushed to Castor for ingestion.

- Visualization assets are typically:
  - `dashboards`
  - `users`
  - `folders`
  - ...

- Warehouse assets are typically:
  - `databases`
  - `schemas`
  - `tables`
  - `columns`
  - `queries`
  - ...

It also embeds utilities to help you push your metadata to Castor:

- `File Checker` to validate your [generic](https://docs.castordoc.com/integrations/data-warehouses/generic-warehouse) CSV files before pushing to Castor
- `Uploader` to push extracted files to our Google-Cloud-Storage (GCS)

# Table of contents

- [Castor Extractor ](#castor-extractor-)
- [Table of contents](#table-of-contents)
- [Installation](#installation)
  - [Create castor-env](#create-castor-env)
  - [PIP install](#pip-install)
  - [Create the output directory](#create-the-output-directory)
- [Contact](#contact)

# Installation

Requirements: **python3.8+**
<img src="https://upload.wikimedia.org/wikipedia/commons/c/c3/Python-logo-notext.svg" width=20 />

## Create castor-env

We advise to create a dedicated [Python environment](https://docs.python.org/3/library/venv.html).

Here's an example using `Pyenv` and Python `3.8.12`:

- Install Pyenv

```bash
brew install pyenv
brew install pyenv-virtualenv
```

- [optional] Update your `.bashrc` if you encounter this [issue](https://stackoverflow.com/questions/45577194/failed-to-activate-virtualenv-with-pyenv/45578839)

```bash
eval "$(pyenv init -)"
eval "$(pyenv init --path)"
eval "$(pyenv virtualenv-init -)"
```

- [optional] Install python 3.8+

```bash
pyenv versions # check your local python installations

pyenv install -v 3.8.12 # if none of the installed versions satisfy requirements 8+
```

- Create your virtual env

```bash
pyenv virtualenv 3.8.12 castor-env # create a dedicated env
pyenv shell castor-env # activate the environment

# optional checks
python --version # should be `3.8.12`
pyenv version # should be `castor-env`
```

## PIP install

⚠️ `castor-env` must be created AND activated first.

```bash
pyenv shell castor-env
(castor-env) $ # this means the environment is now active
```

ℹ️ please upgrade `PIP` before installing Castor.

```
pip install --upgrade pip
```

Run the following command to install `castor-extractor`:

```
pip install castor-extractor
```

Depending on your use-case, you can also install one of the following `extras`:

```
pip install castor-extractor[looker]
pip install castor-extractor[tableau]
pip install castor-extractor[metabase]
pip install castor-extractor[qlik]
pip install castor-extractor[bigquery]
pip install castor-extractor[redshift]
pip install castor-extractor[snowflake]
```

## Create the output directory

```bash
mkdir /tmp/castor
```

You will provide this path in `extraction` scripts as following:

```
castor-extract-bigquery --output=/tmp/castor
```

Alternatively, you can also set the following `ENV` in your `bashrc`:

```bash
export CASTOR_OUTPUT_DIRECTORY="/tmp/castor"
````

# Contact

For any questions or bug report, contact us at [support@castordoc.com](mailto:support@castordoc.com)

[Castor](https://castordoc.com) helps you find, understand, use your data assets

