Metadata-Version: 2.1
Name: pricehist
Version: 1.0.0
Summary: Fetch and format historical price data
Home-page: https://gitlab.com/chrisberkhout/pricehist
License: MIT
Keywords: historical,prices,plaintext,accounting,csv,gnucash,ledger,hledger,beancount
Author: Chris Berkhout
Author-email: chris@chrisberkhout.com
Requires-Python: >=3.9,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Requires-Dist: cssselect (>=1.1.0,<2.0.0)
Requires-Dist: curlify (>=2.2.1,<3.0.0)
Requires-Dist: lxml (>=4.6.2,<5.0.0)
Requires-Dist: requests (>=2.25.1,<3.0.0)
Project-URL: Repository, https://gitlab.com/chrisberkhout/pricehist
Description-Content-Type: text/markdown

# pricehist

A command-line tool for fetching and formatting historical price data, with
support for multiple data sources and output formats.

[![Pipeline status](https://gitlab.com/chrisberkhout/pricehist/badges/master/pipeline.svg)](https://gitlab.com/chrisberkhout/pricehist/-/commits/master)
[![Coverage report](https://gitlab.com/chrisberkhout/pricehist/badges/master/coverage.svg)](https://gitlab.com/chrisberkhout/pricehist/-/commits/master)
[![PyPI version](https://badge.fury.io/py/pricehist.svg)](https://badge.fury.io/py/pricehist)
[![Downloads](https://pepy.tech/badge/pricehist)](https://pepy.tech/project/pricehist)
[![License](https://img.shields.io/pypi/l/pricehist)](https://gitlab.com/chrisberkhout/pricehist/-/blob/master/LICENSE)
[![Code style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

## Installation

Install via pip or
[pipx](https://pypa.github.io/pipx/).

```
pipx install pricehist
```

## Sources

- **`alphavantage`**: [Alpha Vantage](https://www.alphavantage.co/)
- **`coindesk`**: [CoinDesk Bitcoin Price Index](https://www.coindesk.com/coindesk-api)
- **`coinmarketcap`**: [CoinMarketCap](https://coinmarketcap.com/)
- **`ecb`**: [European Central Bank Euro foreign exchange reference rates](https://www.ecb.europa.eu/stats/exchange/eurofxref/html/index.en.html)
- **`yahoo`**: [Yahoo! Finance](https://finance.yahoo.com/)

## Output formats

- **`beancount`**: [Beancount](http://furius.ca/beancount/)
- **`csv`**: [Comma-separated values](https://en.wikipedia.org/wiki/Comma-separated_values)
- **`gnucash-sql`**: [GnuCash](https://www.gnucash.org/) SQL
- **`ledger`**: [Ledger](https://www.ledger-cli.org/) and [hledger](https://hledger.org/)

## How to

### Fetch prices

Fetch prices by choosing a source, a pair and, optionally, a time interval.

```
pricehist fetch ecb EUR/AUD -s 2021-01-04 -e 2021-01-08
```
```
date,base,quote,amount,source,type
2021-01-04,EUR,AUD,1.5928,ecb,reference
2021-01-05,EUR,AUD,1.5927,ecb,reference
2021-01-06,EUR,AUD,1.5824,ecb,reference
2021-01-07,EUR,AUD,1.5836,ecb,reference
2021-01-08,EUR,AUD,1.5758,ecb,reference
```

The default output format is CSV, which is suitable for use in spreadsheets and
with other tools. For example, you can generate a price chart from the command
line as follows (or using [an alias](https://gitlab.com/-/snippets/2163031)).

```
pricehist fetch coindesk BTC/USD -s 2021-01-01 | \
  sed 1d | \
  cut -d, -f1,4 | \
  gnuplot -p -e '
    set datafile separator ",";
    set xdata time;
    set timefmt "%Y-%m-%d";
    set format x "%b\n%Y";
    plot "/dev/stdin" using 1:2 with lines title "BTC/USD"
  '
```

![BTC/USD prices](example-gnuplot.png)

### Show usage information

Add `-h` to any command to see usage information.

```
pricehist fetch -h
```
```
usage: pricehist fetch SOURCE PAIR [-h] [-vvv] [-t TYPE] [-s DATE | -sx DATE] [-e DATE | -ex DATE]
[-o beancount|csv|gnucash-sql|ledger] [--invert] [--quantize INT]
[--fmt-base SYM] [--fmt-quote SYM] [--fmt-time TIME] [--fmt-decimal CHAR] [--fmt-thousands CHAR]
[--fmt-symbol rightspace|right|leftspace|left] [--fmt-datesep CHAR] [--fmt-csvdelim CHAR]

positional arguments:
  SOURCE                   the source identifier
  PAIR                     pair, usually BASE/QUOTE, e.g. BTC/USD

optional arguments:
  -h, --help               show this help message and exit
  -vvv, --verbose          show all log messages
  -t TYPE, --type TYPE     price type, e.g. close
  -s DATE, --start DATE    start date, inclusive (default: source start)
  -sx DATE, --startx DATE  start date, exclusive
  -e DATE, --end DATE      end date, inclusive (default: today)
  -ex DATE, --endx DATE    end date, exclusive
  -o FMT, --output FMT     output format (default: csv)
  --invert                 invert the price, swapping base and quote
  --quantize INT           round to the given number of decimal places
  --fmt-base SYM           rename the base symbol in output
  --fmt-quote SYM          rename the quote symbol in output
  --fmt-time TIME          set a particular time of day in output (default: 00:00:00)
  --fmt-decimal CHAR       decimal point in output (default: '.')
  --fmt-thousands CHAR     thousands separator in output (default: '')
  --fmt-symbol LOCATION    commodity symbol placement in output (default: rightspace)
  --fmt-datesep CHAR       date separator in output (default: '-')
  --fmt-csvdelim CHAR      field delimiter for CSV output (default: ',')
```

### Choose and customize the output format

As the output format you can choose one of `beancount`, `csv`, `ledger` or
`gnucash-sql`.

```
pricehist fetch ecb EUR/AUD -s 2021-01-04 -e 2021-01-08 -o ledger
```
```
P 2021-01-04 00:00:00 EUR 1.5928 AUD
P 2021-01-05 00:00:00 EUR 1.5927 AUD
P 2021-01-06 00:00:00 EUR 1.5824 AUD
P 2021-01-07 00:00:00 EUR 1.5836 AUD
P 2021-01-08 00:00:00 EUR 1.5758 AUD
```

Formatting options let you control certain details of the output.

```
pricehist fetch ecb EUR/AUD -s 2021-01-04 -e 2021-01-08 -o ledger \
  --fmt-time '' --fmt-datesep / --fmt-base € --fmt-quote $ --fmt-symbol left
```
```
P 2021/01/04 € $1.5928
P 2021/01/05 € $1.5927
P 2021/01/06 € $1.5824
P 2021/01/07 € $1.5836
P 2021/01/08 € $1.5758
```

### Load prices into GnuCash

You can generate SQL for a GnuCash database and apply it immediately with one
of the following commands.

```
pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | sqlite3 Accounts.gnucash
pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | mysql -u username -p -D databasename
pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | psql -U username -d databasename -v ON_ERROR_STOP=1
```

Beware that the GnuCash project itself does not support integration at the
database level, so there is a risk that the SQL generated by `pricehist` will
be ineffective or even damaging for some version of GnuCash.

In practice, this strategy has been used successfully by other projects.
Reading the SQL and keeping regular database backups is recommended.

### Show source information

The `source` command shows information about a source.

```
pricehist source alphavantage
```
```
ID          : alphavantage
Name        : Alpha Vantage
Description : Provider of market data for stocks, forex and cryptocurrencies
URL         : https://www.alphavantage.co/
Start       : 1995-01-01
Types       : close, open, high, low, adjclose, mid
Notes       : Alpha Vantage has data on...
```

Available symbols can be listed for most sources, either as full pairs or as
separate base and quote symbols that will work in certain combinations.

```
pricehist source ecb --symbols
```
```
EUR/AUD    Euro against Australian Dollar
EUR/BGN    Euro against Bulgarian Lev
EUR/BRL    Euro against Brazilian Real
EUR/CAD    Euro against Canadian Dollar
EUR/CHF    Euro against Swiss Franc
...
```

It may also be possible to search for symbols.

```
pricehist source alphavantage --search Tesla
```
```
TL0.DEX       Tesla, Equity, XETRA, EUR
TL0.FRK       Tesla, Equity, Frankfurt, EUR
TSLA34.SAO    Tesla, Equity, Brazil/Sao Paolo, BRL
TSLA          Tesla Inc, Equity, United States, USD
TXLZF         Tesla Exploration Ltd, Equity, United States, USD
```

### Inspect source interactions

You can see extra information by adding the verbose option (`--verbose` or
`-vvv`), including `curl` commands that reproduce each request to a source.

```
pricehist fetch coindesk BTC/USD -s 2021-01-01 -e 2021-01-05 -vvv
```
```
DEBUG Began pricehist run at 2021-08-12 14:38:26.630357.
DEBUG Starting new HTTPS connection (1): api.coindesk.com:443
DEBUG https://api.coindesk.com:443 "GET /v1/bpi/historical/close.json?currency=USD&start=2021-01-01&end=2021-01-05 HTTP/1.1" 200 319
DEBUG curl -X GET -H 'Accept: */*' -H 'Accept-Encoding: gzip, deflate' -H 'Connection: keep-alive' -H 'User-Agent: python-requests/2.25.1' --compressed 'https://api.coindesk.com/v1/bpi/historical/close.json?currency=USD&start=2021-01-01&end=2021-01-05'
DEBUG Available data covers the interval [2021-01-01--2021-01-05], as requested.
date,base,quote,amount,source,type
2021-01-01,BTC,USD,29391.775,coindesk,close
2021-01-02,BTC,USD,32198.48,coindesk,close
2021-01-03,BTC,USD,33033.62,coindesk,close
2021-01-04,BTC,USD,32017.565,coindesk,close
2021-01-05,BTC,USD,34035.0067,coindesk,close
DEBUG Ended pricehist run at 2021-08-12 14:38:26.709428.
```

Running a logged `curl` command shows exactly what data is returned by the
source.

```
pricehist fetch coindesk BTC/USD -s 2021-01-01 -e 2021-01-05 -vvv 2>&1 \
  | grep '^DEBUG curl' | sed 's/^DEBUG //' | bash | jq .
```
```json
{
  "bpi": {
    "2021-01-01": 29391.775,
    "2021-01-02": 32198.48,
    "2021-01-03": 33033.62,
    "2021-01-04": 32017.565,
    "2021-01-05": 34035.0067
  },
  "disclaimer": "This data was produced from the CoinDesk Bitcoin Price Index. BPI value data returned as USD.",
  "time": {
    "updated": "Jan 6, 2021 00:03:00 UTC",
    "updatedISO": "2021-01-06T00:03:00+00:00"
  }
}
```

### Use as a library

You may find `pricehist`'s source classes useful in your own scripts.

```
$ python
Python 3.9.6 (default, Jun 30 2021, 10:22:16)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pricehist.series import Series
>>> from pricehist.sources.ecb import ECB
>>> series = ECB().fetch(Series("EUR", "AUD", "reference", "2021-01-04", "2021-01-08"))
>>> series.prices
[Price(date='2021-01-04', amount=Decimal('1.5928')), Price(date='2021-01-05', amount=Decimal('1.5927')), Price(date='2021-01-06', amount=Decimal('1.5824')), Price(date='2021-01-07', amount=Decimal('1.5836')), Price(date='2021-01-08', amount=Decimal('1.5758'))]
```

A subclass of `pricehist.exceptions.SourceError` will be raised for any error.

## Terminology

A **source** is an upstream service that can provide a series of prices.

Each **series** of prices is for one pair and price type.

The [**pair**](https://en.wikipedia.org/wiki/Currency_pair) is made up of a
base and a quote, each given as a symbol. Sometimes you will give the base
only, and the quote will be determined with information from the source. The
available pairs, the symbols used in them and the available price types all
depend on the particular source used.

The **base** is the currency or commodity being valued. Each price expresses
the value of one unit of the base.

The **quote** is the unit used to express the value of the base.

A **symbol** is a code or abbreviation for a currency or commodity.

The **prices** in a series each have a date and an amount.

The **amount** is the number of units of the quote that are equal to one unit
of the base.

Consider the following command.

```
pricehist fetch coindesk BTC/USD --type close
```

- **`coindesk`** is the ID of the CoinDesk Bitcoin Price Index source.
- **`BTC`** is the symbol for Bitcoin, used here as the base.
- **`USD`** is the symbol for the United States Dollar, used here as the quote.
- **`BTC/USD`** is the pair Bitcoin against United States Dollar.
- **`close`** is the price type for the last price of each day.

A BTC/USD price of the amount 29,391.775 can be written as
"BTC/USD = 29391.78" or "BTC 29391.78 USD", and means that one Bitcoin is
worth 29,391.78 United States Dollars.

## Initial design choices

To keep things simple, `pricehist` provides only univariate time series of
daily historical prices. It doesn't provide other types of market, financial or
economic data, real-time prices, or other temporal resolutions. Multiple or
multivariate series require multiple invocations.

## Potential features

In the future, `pricehist` may be extended to cover some of the following
features:

- **Time of day**: Sources sometimes provide specific times for each day's
  high/low prices and these could be preserved for output. This would require
  changes to how dates are handled internally, clarification of time zone
  handling and extension of the time formatting option.
- **Alternate resolutions**: Some sources can provide higher or lower
  resolution data, such as hourly or weekly. These could be supported where
  available. For other cases an option could be provided for downsampling data
  before output.
- **Real-time prices**: These generally come from different source endpoints
  than the historical data. Real-time prices will usually have a different
  price type, such as `last`, `bid` or `ask`. Support for real-time prices
  would allow adding sources that don't provide historical data. Start and end
  times are irrelevant when requesting real-time prices. A "follow" option
  could continuously poll for new prices.
- **Related non-price data**: Trading volume, spreads, split and dividend
  events and other related data could be supported. The base/quote/type model
  used for prices would work for some of this. Other things may require
  extending the model.
- **Multivariate series**: Would allow, for example, fetching
  high/low/open/close prices in a single invocation.
- **`format` command**: A command for rewriting existing CSV data into one of
  the other output formats.

## Alternatives

Beancount's [`bean-price`](https://github.com/beancount/beanprice) tool fetches
prices and addresses other workflow concerns in a Beancount-specific manner,
generally requiring a Beancount file as input.

The [Piecash](https://piecash.readthedocs.io/) library is a pythonic interface
to GnuCash files stored in SQL which has a
[`Commodity.update_prices`](https://piecash.readthedocs.io/en/master/api/piecash.core.commodity.html?highlight=update_prices#piecash.core.commodity.Commodity.update_prices)
method for fetching historical prices.
The GnuCash wiki documents [wrapper scripts](https://wiki.gnucash.org/wiki/Stocks/get_prices)
for the [Finance::QuoteHist](https://metacpan.org/pod/Finance::QuoteHist) Perl
module.

