Metadata-Version: 2.1
Name: metaphor-connectors
Version: 0.11.98
Summary: A collection of Python-based 'connectors' that extract metadata from various sources to ingest into the Metaphor app.
Home-page: https://metaphor.io
License: Apache-2.0
Author: Metaphor
Author-email: dev@metaphor.io
Requires-Python: >=3.7,<3.11
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Provides-Extra: all
Provides-Extra: bigquery
Provides-Extra: dbt
Provides-Extra: looker
Provides-Extra: metabase
Provides-Extra: postgresql
Provides-Extra: power-bi
Provides-Extra: redshift
Provides-Extra: snowflake
Provides-Extra: synapse
Provides-Extra: tableau
Provides-Extra: throughtspot
Provides-Extra: unity-catalog
Requires-Dist: GitPython (>=3.1.0,<4.0.0); extra == "all" or extra == "looker"
Requires-Dist: PyYAML (>=6.0,<7.0)
Requires-Dist: asyncpg (>=0.25.0,<0.26.0); extra == "all" or extra == "postgresql" or extra == "redshift"
Requires-Dist: aws-assume-role-lib (>=2.10.0,<3.0.0)
Requires-Dist: boto3 (>=1.24.20,<2.0.0)
Requires-Dist: botocore (>=1.27.20,<2.0.0)
Requires-Dist: canonicaljson (>=1.4.0,<2.0.0)
Requires-Dist: coverage (>=6.5.0,<7.0.0)
Requires-Dist: databricks-cli (>=0.17.3,<0.18.0); extra == "all" or extra == "unity-catalog"
Requires-Dist: fastjsonschema (>=2.15.1,<3.0.0)
Requires-Dist: google-cloud-bigquery (>=2.34.4,<3.0.0); extra == "all" or extra == "bigquery"
Requires-Dist: google-cloud-logging (>=3.1.0,<4.0.0); extra == "all" or extra == "bigquery"
Requires-Dist: lkml (>=1.2.0,<2.0.0); extra == "all" or extra == "looker"
Requires-Dist: looker-sdk (>=22.4.0,<23.0.0); extra == "all" or extra == "looker"
Requires-Dist: metaphor-models (>=0.17.2,<0.18.0)
Requires-Dist: metaphor-sqllineage (>=1.3.6,<2.0.0); extra == "all" or extra == "redshift"
Requires-Dist: msal (>=1.17.0,<2.0.0); extra == "all" or extra == "power-bi" or extra == "synapse"
Requires-Dist: pydantic (>=1.9.0,<1.10.0)
Requires-Dist: pymssql (>=2.2.7,<3.0.0); extra == "all" or extra == "synapse"
Requires-Dist: python-dateutil (>=2.8.1,<3.0.0)
Requires-Dist: requests (>=2.27.1,<3.0.0)
Requires-Dist: smart-open (>=5.0.0,<6.0.0)
Requires-Dist: snowflake-connector-python (>=2.7.0,<2.8.0); extra == "all" or extra == "snowflake"
Requires-Dist: sql-metadata (>=2.6.0,<3.0.0); extra == "all" or extra == "bigquery" or extra == "looker" or extra == "metabase" or extra == "power-bi" or extra == "snowflake"
Requires-Dist: tableauserverclient (>=0.17.0,<0.18.0); extra == "all" or extra == "tableau"
Requires-Dist: thoughtspot-rest-api-sdk (>=1.11.0,<1.12.0); extra == "all" or extra == "throughtspot"
Project-URL: Repository, https://github.com/MetaphorData/connectors
Description-Content-Type: text/markdown

<a href="https://metaphor.io"><img src="https://github.com/MetaphorData/connectors/raw/main/logo.png" width="300" /></a>

# Metaphor Connectors

[![Coverage](https://coveralls.io/repos/github/MetaphorData/connectors/badge.svg?branch=main)](https://coveralls.io/github/MetaphorData/connectors?branch=main)
[![CodeQL](https://github.com/MetaphorData/connectors/workflows/CodeQL/badge.svg)](https://github.com/MetaphorData/connectors/actions/workflows/codeql-analysis.yml)
[![PyPI Version](https://img.shields.io/pypi/v/metaphor-connectors)](https://pypi.org/project/metaphor-connectors/)
![Python version 3.7+](https://img.shields.io/badge/python-3.7%2B-blue)
![PyPI Downloads](https://img.shields.io/pypi/dm/metaphor-connectors)
[![License](https://img.shields.io/github/license/MetaphorData/connectors)](https://github.com/MetaphorData/connectors/blob/master/LICENSE)

This repository contains a collection of Python-based "connectors" that extract metadata from various sources to ingest into the [Metaphor](https://metaphor.io) platform.

## Installation

This package requires Python 3.7+ installed. You can verify the version on your system by running the following command,

```shell
python -V  # or python3 on some systems
```

Once verified, you can install the package using [pip](https://docs.python.org/3/installing/index.html),

```shell
pip install "metaphor-connectors[all]"  # or pip3 on some systems
```

This will install all the connectors and required dependencies. You can also choose to install only a subset of the dependencies by installing the specific [extra](https://packaging.python.org/tutorials/installing-packages/#installing-setuptools-extras), e.g.

```shell
pip install "metaphor-connectors[snowflake]"
```

Similarly, you can also install the package using `requirements.txt` or `pyproject.toml`.

## Connectors

Each connector is placed under its own directory under [metaphor](./metaphor) and extends the `metaphor.common.BaseExtractor` class.

| Connector Name                                                | Metadata                                 |
|---------------------------------------------------------------|------------------------------------------|  
| [airflow_plugin](metaphor/airflow_plugin/README.md)           | Lineage                                  |
| [bigquery](metaphor/bigquery/README.md)                       | Schema, description, statistics, queries |
| [bigquery.lineage](metaphor/bigquery/lineage/README.md)       | Lineage                                  |
| [bigquery.profile](metaphor/bigquery/profile/README.md)       | Data profile                             |
| [dbt](metaphor/dbt/README.md)                                 | dbt model, test, lineage                 |
| [dbt.cloud](metaphor/dbt/cloud/README.md)                     | dbt model, test, lineage                 |
| [glue](metaphor/glue/README.md)                               | Schema, description                      |
| [looker](metaphor/looker/README.md)                           | Looker view, explore, dashboard, lineage |
| [manual.data_quality](metaphor/manual/data_quality/README.md) | Data quality                             |
| [manual.governance](metaphor/manual/governance/README.md)     | Ownership, tags, description             |
| [manual.lineage](metaphor/manual/lineage/README.md)           | Lineage                                  |
| [manual.metadata](metaphor/manual/metadata/README.md)         | Custom metadata                          |
| [metabase](metaphor/metabase/README.md)                       | Dashboard, lineage                       |
| [postgresql](metaphor/postgresql/README.md)                   | Schema, description, statistics          |
| [postgresql.profile](metaphor/postgresql/profile/README.md)   | Data profile                             |
| [postgresql.usage](metaphor/postgresql/usage/README.md)       | Usage                                    |
| [power_bi](metaphor/power_bi/README.md)                       | Dashboard, lineage                       |
| [redshift](metaphor/redshift/README.md)                       | Schema, description, statistics, queries |
| [redshift.lineage](metaphor/redshift/lineage/README.md)       | Lineage                                  |
| [redshift.profile](metaphor/redshift/profile/README.md)       | Data profile                             |
| [snowflake](metaphor/snowflake/README.md)                     | Schema, description, statistics, queries |
| [snowflake.lineage](metaphor/snowflake/lineage/README.md)     | Lineage                                  |
| [snowflake.profile](metaphor/snowflake/profile/README.md)     | Data profile                             |
| [synapse](metaphor/synapse//README.md)                        | Schema, queries                          |
| [tableau](metaphor/tableau/README.md)                         | Dashboard, lineage                       |
| [thought_spot](metaphor/thought_spot/README.md)               | Dashboard, lineage                       |
| [unity_catalog](metaphor/unity_catalog/README.md)             | Schema, description                      |

## Development

See [Development Environment](docs/develop.md) for more instructions on how to setup your local development environment.

## Custom Connectors

See [Adding a Custom Connector](docs/custom.md) for instructions and a full example on how to create your custom connectors.

