Metadata-Version: 2.1
Name: amulog
Version: 0.2.4
Summary: A system log management tool with automatically generated log templates.
Home-page: https://github.com/cpflat/amulog/
Author: Satoru Kobayashi
Author-email: sat@nii.ac.jp
License: The 3-Clause BSD License
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.6
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
License-File: LICENSE

########################################
AMULOG (A Manager for Unstructured LOGs)
########################################

.. image:: https://img.shields.io/pypi/v/amulog
   :alt: PyPI release
   :target: https://pypi.org/project/amulog/

.. image:: https://img.shields.io/pypi/pyversions/amulog
   :alt: Python support
   :target: https://pypi.org/project/amulog/

.. image:: https://img.shields.io/badge/License-BSD%203--Clause-blue.svg
   :alt: BSD 3-Clause License
   :target: https://opensource.org/licenses/BSD-3-Clause

.. image:: https://travis-ci.org/cpflat/amulog.svg?branch=master
   :alt: Travis CI
   :target: https://travis-ci.com/github/cpflat/amulog


Amulog is a tool to support system log management.
The main function is to classify log messages with automatically generated log templates (formats and variable locations),
and to store the data in a database.
This system works on python3.

* Source: https://github.com/cpflat/amulog
* Bug Reports: https://github.com/cpflat/amulog/issues
* Author: `Satoru Kobayashi <https://github.com/cpflat/>`_
* License: `BSD-3-Clause <https://opensource.org/licenses/BSD-3-Clause>`_


Main features
=============

- Support multiple databases: sqlite and mysql
- Smart log segmentation with `log2seq <https://github.com/cpflat/log2seq>`_
- Multiple template generation algorithms such as: Drain, SHISO, LenMa, FT-tree, Dlog, etc.
- Support Online (incremental) and Offline (hindsight) use
- Suspend and resume the template generation process
- Import and Export log templates if you need
- Edit log templates manually if you need
- Search API with datetime, hostname and log template IDs


Tutorial
========

Install
-------

::

    $ pip install amulog


Generate config
---------------

For the first step, save following config as :code:`test.conf` on an empty directory.

::

    [general]
    src_path = logfile.txt
    src_recur = false
    logging = auto.log

    [database]
    database = sqlite3
    sqlite3_filename = log.db

    [log_template]
    lt_methods = drain
    indata_filename = ltgen.dump

Then modify :code:`general.src_path` option to a logfile you want to load.
(If you want to use multiple files, change :code:`general.src_recur` into true and specify directory name to :code:`general.src_path`.)


Generate database
-----------------

Try following command to generate database:

::

    $ python -m amulog db-make -c test.conf


Check database
--------------

::

    $ python -m amulog show-db-info -c test.conf

shows status of the generated database.

::

    $ python -m amulog show-lt -c test.conf

shows all generated log templates in the given logfile.

::

    $ python -m amulog show-log -c test.conf ltid=2

shows all log messages corresponding to log template ID 2.


Resume generating database
--------------------------

Try following command to resume generating database:

::

    $ python -m amulog db-add -c test.conf logfile2.txt


Export and Import templates
---------------------------

Following command exports all log templates in the database:

::

    $ python3 -m amulog show-db-import -c test.conf > exported_tpl.txt

You can modify the exported templates manually.
Note that some special letters (:code:`\\`, :code:`@`, :code:`*`) are escaped in the exported templates.

To import the templates, save following config as :code:`test2.conf`.

::

    [general]
    src_path = logfile.txt
    src_recur = false
    logging = new_auto.log

    [database]
    database = sqlite3
    sqlite3_filename = new_log.db

    [log_template]
    lt_methods = import
    indata_filename = new_ltgen.dump

    [log_template_import]
    def_path = exported_tpl.txt

Then, try generating database again:

::

    python -m amulog db-make -c test2.conf


Further usage
-------------

see help with following command:

::

    python -m amulog -h


Reference
=========

This tool is demonstrated at `CNSM2020 <http://dl.ifip.org/db/conf/cnsm/cnsm2020/>`_.
If you use this code, please consider citing:

::

    @inproceedings{Kobayashi_CNSM2020,
      author = {Kobayashi, Satoru and Yamashiro, Yuya and Otomo, Kazuki and Fukuda, Kensuke},
      booktitle = {Proceedings of the 16th International Conference on Network and Service Management (CNSM'20)},
      title = {amulog: A General Log Analysis Framework for Diverse Template Generation Methods},
      pages={1-5},
      year = {2020}
    }


