Metadata-Version: 2.1
Name: keras-lmu
Version: 0.4.2
Summary: Keras implementation of Legendre Memory Units
Home-page: https://www.nengo.ai/keras-lmu
Author: Applied Brain Research
Author-email: info@appliedbrainresearch.com
License: Free for non-commercial use
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: Free for non-commercial use
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python 
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Topic :: Scientific/Engineering 
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.6
Provides-Extra: all
Provides-Extra: docs
Provides-Extra: optional
Provides-Extra: tests
License-File: LICENSE.rst

KerasLMU: Recurrent neural networks using Legendre Memory Units
---------------------------------------------------------------

`Paper <https://papers.nips.cc/paper/9689-legendre-memory-units-continuous-time-representation-in-recurrent-neural-networks.pdf>`_

This is a Keras-based implementation of the
Legendre Memory Unit (LMU). The LMU is a novel memory cell for recurrent neural
networks that dynamically maintains information across long windows of time using
relatively few resources. It has been shown to perform as well as standard LSTM or
other RNN-based models in a variety of tasks, generally with fewer internal parameters
(see `this paper
<https://papers.nips.cc/paper/9689-legendre-memory-units-continuous-time-representation-in-recurrent-neural-networks.pdf>`_ for more details). For the Permuted Sequential MNIST (psMNIST) task in particular, it has been demonstrated to outperform the current state-of-the-art results. See the note below for instructions on how to get access to this model.

The LMU is mathematically derived to orthogonalize its continuous-time history – doing
so by solving *d* coupled ordinary differential equations (ODEs), whose phase space
linearly maps onto sliding windows of time via the Legendre polynomials up to degree
*d* − 1 (the example for *d* = 12 is shown below).

.. image:: https://i.imgur.com/Uvl6tj5.png
   :target: https://i.imgur.com/Uvl6tj5.png
   :alt: Legendre polynomials

A single LMU cell expresses the following computational graph, which takes in an input
signal, **x**, and couples a optimal linear memory, **m**, with a nonlinear hidden
state, **h**. By default, this coupling is trained via backpropagation, while the
dynamics of the memory remain fixed.

.. image:: https://i.imgur.com/IJGUVg6.png
   :target: https://i.imgur.com/IJGUVg6.png
   :alt: Computational graph

The discretized **A** and **B** matrices are initialized according to the LMU's
mathematical derivation with respect to some chosen window length, **θ**.
Backpropagation can be used to learn this time-scale, or fine-tune **A** and **B**,
if necessary.

Both the kernels, **W**, and the encoders, **e**, are learned. Intuitively, the kernels
learn to compute nonlinear functions across the memory, while the encoders learn to
project the relevant information into the memory (see `paper
<https://papers.nips.cc/paper/9689-legendre-memory-units-continuous-time-representation-in-recurrent-neural-networks.pdf>`_ for details).

.. note::

   The ``paper`` branch in the ``lmu`` GitHub repository includes a pre-trained
   Keras/TensorFlow model, located at ``models/psMNIST-standard.hdf5``, which obtains
   a psMNIST result of **97.15%**. Note that the network is using fewer internal
   state-variables and neurons than there are pixels in the input sequence.
   To reproduce the results from `this paper
   <https://papers.nips.cc/paper/9689-legendre-memory-units-continuous-time-representation-in-recurrent-neural-networks.pdf>`_,
   run the notebooks in the ``experiments`` directory within the ``paper`` branch.

Nengo Examples
--------------

* `LMUs in Nengo (with online learning)
  <https://www.nengo.ai/nengo/examples/learning/lmu.html>`_
* `Spiking LMUs in Nengo Loihi (with online learning)
  <https://www.nengo.ai/nengo-loihi/examples/lmu.html>`_
* `LMUs in NengoDL (reproducing SotA on psMNIST)
  <https://www.nengo.ai/nengo-dl/examples/lmu.html>`_

Citation
--------

.. code-block::

   @inproceedings{voelker2019lmu,
     title={Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks},
     author={Aaron R. Voelker and Ivana Kaji\'c and Chris Eliasmith},
     booktitle={Advances in Neural Information Processing Systems},
     pages={15544--15553},
     year={2019}
   }

***************
Release history
***************

.. Changelog entries should follow this format:

   version (release date)
   ======================

   **section**

   - One-line description of change (link to Github issue/PR)

.. Changes should be organized in one of several sections:

   - Added
   - Changed
   - Deprecated
   - Removed
   - Fixed

0.4.2 (May 17, 2022)
====================

*Compatible with TensorFlow 2.1 - 2.9*

**Added**

- Added support for TensorFlow 2.9. (`#48`_)

.. _#48: https://github.com/nengo/keras-lmu/pull/48

0.4.1 (February 10, 2022)
=========================

*Compatible with TensorFlow 2.1 - 2.8*

**Added**

- Added support for TensorFlow 2.8. (`#46`_)
- Allow for optional bias on the memory component with the ``use_bias`` flag. (`#44`_)
- Added regularizer support for kernel, recurrent kernel, and bias. (`#44`_)

.. _#44: https://github.com/nengo/keras-lmu/pull/44
.. _#46: https://github.com/nengo/keras-lmu/pull/46

0.4.0 (August 16, 2021)
=======================

*Compatible with TensorFlow 2.1 - 2.7*

**Added**

- Setting ``kernel_initializer=None`` now removes the dense input kernel. (`#40`_)
- The ``keras_lmu.LMUFFT`` layer now supports ``memory_d > 1``. ``keras_lmu.LMU`` now
  uses this implementation for all values of ``memory_d`` when feedforward conditions
  are satisfied (no hidden-to-memory or memory-to-memory connections,
  and the sequence length is not ``None``). (`#40`_)
- Added ``trainable_theta`` option, which will allow the ``theta`` parameter to be
  learned during training. (`#41`_)
- Added ``discretizer`` option, which controls the method used to solve for the ``A``
  and ``B`` LMU matrices. This is mainly useful in combination with
  ``trainable_theta=True``, where setting ``discretizer="euler"`` may improve the
  training speed (possibly at the cost of some accuracy). (`#41`_)
- The ``keras_lmu.LMUFFT`` layer can now use raw convolution internally (as opposed to
  FFT-based convolution). The new ``conv_mode`` option exposes this. The new
  ``truncate_ir`` option allows truncating the impulse response when running with a
  raw convolution mode, for efficiency. Whether FFT-based or raw convolution is faster
  depends on the specific model, hardware, and amount of truncation. (`#42`_)

**Changed**

- The ``A`` and ``B`` matrices are now stored as constants instead of non-trainable
  variables. This can improve the training/inference speed, but it means that saved
  weights from previous versions will be incompatible. (`#41`_)
- Renamed ``keras_lmu.LMUFFT`` to ``keras_lmu.LMUFeedforward``. (`#42`_)

**Fixed**

- Fixed dropout support in TensorFlow 2.6. (`#42`_)

.. _#40: https://github.com/nengo/keras-lmu/pull/40
.. _#41: https://github.com/nengo/keras-lmu/pull/41
.. _#42: https://github.com/nengo/keras-lmu/pull/42

0.3.1 (November 16, 2020)
=========================

**Changed**

- Raise a validation error if ``hidden_to_memory`` or ``input_to_hidden`` are True
  when ``hidden_cell=None``. (`#26`_)

**Fixed**

- Fixed a bug with the autoswapping in ``keras_lmu.LMU`` during training. (`#28`_)
- Fixed a bug where dropout mask was not being reset properly in the hidden cell.
  (`#29`_)

.. _#26: https://github.com/nengo/keras-lmu/pull/26
.. _#28: https://github.com/nengo/keras-lmu/pull/28
.. _#29: https://github.com/nengo/keras-lmu/pull/29


0.3.0 (November 6, 2020)
========================

**Changed**

- Renamed module from ``lmu`` to ``keras_lmu`` (so it will now be imported via
  ``import keras_lmu``), renamed package from ``lmu`` to
  ``keras-lmu`` (so it will now be installed via ``pip install keras-lmu``), and
  changed any references to "NengoLMU" to "KerasLMU" (since this implementation is
  based in the Keras framework rather than Nengo). In the future the ``lmu`` namespace
  will be used as a meta-package to encapsulate LMU implementations in different
  frameworks. (`#24`_)

.. _#24: https://github.com/abr/lmu/pull/24

0.2.0 (November 2, 2020)
========================

**Added**

- Added documentation for package description, installation, usage, API, examples,
  and project information. (`#20`_)
- Added LMU FFT cell variant and auto-switching LMU class. (`#21`_)
- LMUs can now be used with any Keras RNN cell (e.g. LSTMs or GRUs) through the
  ``hidden_cell`` parameter. This can take an RNN cell (like
  ``tf.keras.layers.SimpleRNNCell`` or ``tf.keras.layers.LSTMCell``) or a feedforward
  layer (like ``tf.keras.layers.Dense``) or ``None`` (to create a memory-only LMU).
  The output of the LMU memory component will be fed to the ``hidden_cell``.
  (`#22`_)
- Added ``hidden_to_memory``, ``memory_to_memory``, and ``input_to_hidden`` parameters
  to ``LMUCell``, which can be used to enable/disable connections between components
  of the LMU. They default to disabled. (`#22`_)
- LMUs can now be used with multi-dimensional memory components. This is controlled
  through a new ``memory_d`` parameter of ``LMUCell``. (`#22`_)
- Added ``dropout`` parameter to ``LMUCell`` (which applies dropout to the input)
  and ``recurrent_dropout`` (which applies dropout to the ``memory_to_memory``
  connection, if it is enabled). Note that dropout can be added in the hidden
  component through the ``hidden_cell`` object. (`#22`_)

**Changed**

- Renamed ``lmu.lmu`` module to ``lmu.layers``. (`#22`_)
- Combined the ``*_encoders_initializer``parameters of ``LMUCell`` into a single
  ``kernel_initializer`` parameter. (`#22`_)
- Combined the ``*_kernel_initializer`` parameters of ``LMUCell`` into a single
  ``recurrent_kernel_initializer`` parameter. (`#22`_)

**Removed**

- Removed ``Legendre``, ``InputScaled``, ``LMUCellODE``, and ``LMUCellGating``
  classes. (`#22`_)
- Removed the ``method``, ``realizer``, and ``factory`` arguments from ``LMUCell``
  (they will take on the same default values as before, they just cannot be changed).
  (`#22`_)
- Removed the ``trainable_*`` arguments from ``LMUCell``. This functionality is
  largely redundant with the new functionality added for enabling/disabling internal
  LMU connections. These were primarily used previously for e.g. setting a connection to
  zero and then disabling learning, which can now be done more efficiently by
  disabling the connection entirely. (`#22`_)
- Removed the ``units`` and ``hidden_activation`` parameters of ``LMUCell`` (these are
  now specified directly in the ``hidden_cell``. (`#22`_)
- Removed the dependency on ``nengolib``. (`#22`_)
- Dropped support for Python 3.5, which reached its end of life in September 2020.
  (`#22`_)

.. _#20: https://github.com/abr/lmu/pull/20
.. _#21: https://github.com/abr/lmu/pull/21
.. _#22: https://github.com/abr/lmu/pull/22

0.1.0 (June 22, 2020)
=====================

Initial release of KerasLMU 0.1.0! Supports Python 3.5+.

The API is considered unstable; parts are likely to change in the future.

Thanks to all of the contributors for making this possible!
