Metadata-Version: 2.1
Name: contrail-scrape
Version: 0.1.8
Summary: crawler for all the APIs of Juniper Contrail nodes
Home-page: https://github.com/aniruddhamonker/contrail-scrape
Author: Aniruddh Amonker
Author-email: aamonker@juniper.net
License: MIT
Description: # contrail-scrape
        Contrail-scrape is an introspect and API scraping tool for contrail versions 3.2 and later.
        It also has a built in analyzer component which is a modified IST script for offline analysis of introspect data.
        contrail-scrape uses threads for concurrency and the number of threads can be adjusted based on the cluster environment and API utilization.
        
        In the current version, contrail-scrape supports introspect data collection of the following contrail nodes
        + control(contrail-control, dns)
        + config(api, schema, discovery, svc-monitor, device-manager)
        + vrouter
        + analytics(api, collector, alarmgen, query)
        
        Besides introspect collection, contrail-scrape also supports API scraping of analytics-api's uves and uve alarms.
        
        ## Platforms & Supportibility
        + Requires python version 3.5 and later.
        + Requires a linux system pre-installed with python3-pip.
        + Runs on Ubuntu 16.04 and later
        + Runs on Red Hat RHEL 7.5 and later
        
        ## Installation
        contrail-scrape is packaged and available for installation using pip.
        
        ##### $pip install contrail-scrape
        
        If installation inside virtualenv is desired:
        
        $mkdir ~/contrail-scrape
        
        $python3 -m venv ~/contrail-scrape --system-site-packages
        
        $source ~/contrail-scrape/bin/activate
        
        ##### $pip install contrail-scrape
        
        ## Usage
        ### Help
        ```
        $contrail-scrape --help
        
        A tool to scrape APIs and Introspects of various Contrail nodes
        
        optional arguments:
          -h, --help            show this help message and exit
          --yaml-config FILE    path to YAML configuration file
          --vrouter HOST        contrail-vrouter
          --config HOST         contrail-config
          --control HOST        contrail-control
          --analytics HOST      contrail-analytics-api introspect
          --alarmgen HOST       contrail-analytics-alarmgen
          --discovery HOST      contrail-discovery
          --schema HOST         contrail-schema
          --svc-monitor HOST    contrail-svc-monitor
          --collector HOST      contrail-analytics-collector
          --dev-mgr HOST        contrail device manager
          --dns HOST            contrail-dns
          --query HOST          contrail-query-engine
          --analytics-api HOST  contrail analytics uves and alarms
          --version             print the version number
          --debug               turn on debug mode
          --threads THREADS     control the number of threads, default=50
          --api-timeout API_TIMEOUT
                                set API requests timeout in secs, default=10sec
        ```
        
        ### Usage Examples
        #### Usage 1
        ```
        $contrail-scrape --control 10.85.188.101 --config 10.85.188.102 --analytics 10.85.188.103 --vrouter 10.85.180.79
        
        API Scraping Progress: 100%|████████████████████████████████████| 55/55 [01:11<00:00,  1.30s/thread]
        
        Finishing introspection of all nodes
        No Errors reported
        Begining archive process..
        ```
        
        #### Usage 2 (Using YAML input)
        ```
        $contrail-scrape --yaml-config ~/contrail-scrape/hosts.yaml
        
        API Scraping Progress: 100%|██████████████████████████████████| 186/186 [07:50<00:00,  2.53s/thread]
        
        Finishing introspection of all nodes with Errors
        Please check log file /var/log/scrape-2020-11-12-18:58:59-PST.log for details
        Begining archive process..
        ```
        
        sample hosts.yaml : https://gist.github.com/aniruddhamonker/f49074b50b09ec89a0aec35505204b77
        
        #### Usage 3 (custom introspect ports)
        ```
        $contrail-scrape --vrouter 10.85.180.80:8096 --vrouter 10.85.180.79:8096
        
        API Scraping Progress: 100%|████████████████████████████████████| 42/42 [01:30<00:00,  2.16s/thread]
        
        Finishing introspection of all nodes
        No Errors reported
        Begining archive process..
        ```
        
        #### Usage 4 (Analytics-api uves and alarms)
        ```
        $contrail-scrape --analytics-api 10.85.188.101
        
        Analytics-api Progress: 100%|█████████████████████████████████████| 28/28 [00:56<00:00,  2.02s/uves]
        
        Finishing api scraping of all analytics nodes
        No Errors reported
        Begining archive process..
        ```
        
        # Analyzing Introspect Data
        All the introspect data collected by the contrail-scrape is pretty formated as xml. The data is saved into a single tar.gz archive. The file is saved under local directory from where contrail-scrape is run.
        ```
        # ls -l contrail-scrape*
        -rwxrwxrwx 1 aniruddh aniruddh 242752 Nov 12 20:00 contrail-scrape.tgz
        ```
        
        Once the file is extracted, the data is saved under directory named **scrape** and structured into multiple sub-directories based on the type of introspect node.
        Each of the sub-directory corresponds to introspect type for each type of node(control, config, etc..)
        
        ```
        # tree -L 2 scrape/
        scrape/
        ├── 10.85.188.101-analytics
        │   ├── alarmgen_ctrl
        │   ├── analytics_api_info
        │   ├── cpuinfo
        │   ├── redis
        │   ├── sandesh_alarm
        │   ├── sandesh_ctrl
        │   ├── sandesh_trace
        │   └── sandesh_uve
        ├── 10.85.188.101-config
        │   ├── cpuinfo
        │   ├── discovery_client_stats
        │   ├── greenlets
        │   ├── sandesh_alarm
        │   ├── sandesh_ctrl
        │   ├── sandesh_trace
        │   └── sandesh_uve
        ------skipped----------
        ```
        check https://gist.github.com/aniruddhamonker/81ed43dac81e77d4e7fd9be2e451b402 for the entire structure
        
        ## Contrail-scrape-ist
        Although the data is pre-formatted as xml and can be directly printed from the corresponding introspect files, contrail-scrape also packages a modified IST script **contrail-scrape-ist** to assist with offline analysis and troubleshooting.
        
        Ref IST: https://github.com/vcheny/contrail-introspect-cli/blob/master/ist.py
        
        ```
        # contrail-scrape-ist --help
        usage: contrail-scrape-ist [-h] [--version] [--debug]
                                   {alarmgen,analytics,collector,config,control,devmgr,discovery,dns,query,schema,svc-monitor,vrouter}
                                   ...
        
        A script to make Contrail Introspect output CLI friendly.
        
        positional arguments:
          {alarmgen,analytics,collector,config,control,devmgr,discovery,dns,query,schema,svc-monitor,vrouter}
            alarmgen            contrail-alarm-gen
            analytics           contrail-analytics-api
            collector           contrail-collector
            config              contrail-api
            control             contrail-control
            devmgr              contrail-device-manager
            discovery           contrail-discovery
            dns                 contrail-dns
            query               contrail-query-engine
            schema              contrail-schema
            svc-monitor         contrail-svc-monitor
            vrouter             contrail-vrouter-agent
        
        optional arguments:
          -h, --help            show this help message and exit
          --version             Script version
          --debug               Verbose mode
         ```
        
        ### Usage Contrail-scrape-ist
        To start analyzing data using contrail-scrape-ist, change the working directory to the introspect node that needs to be analyzed.
        ```
        $cd scrape/
        scrape$ ls -l
        total 0
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.180.79-kernel
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.180.80-dpdk
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.188.101-analytics
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.188.101-config
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.188.101-control
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.188.102-analytics
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.188.102-config
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.188.102-control
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.188.103-analytics
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.188.103-config
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.188.103-control
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.188.104-kernel
        drwxrwxrwx 1 root root 4096 Nov 12 20:18 10.85.188.105-kernel
        ```
        The following example shows the usage of contrail-scrape-ist script to analyze data of a contrail control node.
        The instructions to analyze the data from any other node are identical.
        ```
        scrape$cd 10.85.188.101-control
        scrape/10.85.188.101-control$ contrail-scrape-ist control --help
        
        positional arguments:
          {status,cpu,trace,uve,nei,ri,route,mcast,bgp_stats,xmpp,ifmap,sc,config,rt}
            status              Node/component status
            cpu                 CPU load info
            trace               Sandesh trace buffer
            uve                 Sandesh UVE cache
            nei                 Show BGP/XMPPP neighbors
            ri                  Show routing instances
            route               Show route info
            mcast               Show multicast managers
            bgp_stats           Show BGP server stats
            xmpp                Show XMPP info
            ifmap               Show IFMAP info
            sc                  Show ServiceChain info
            config              Show related config info
            rt                  Show RtGroup info
        
        optional arguments:
          -h, --help            show this help message and exit
        
        scrape/10.85.188.101-control$ contrail-scrape-ist control status
        Loading from introspect xml:  /root/scrape/10.85.188.101-control/sandesh_uve/NodeStatus
        module_id: contrail-control
        state: Functional
        description
        +-----------+-------------+-----------------------+--------+--------------------------------------+
        | type      | name        | server_addrs          | status | description                          |
        +-----------+-------------+-----------------------+--------+--------------------------------------+
        | IFMap     | IFMapServer |   172.18.101.101:8443 | Up     | Connection with IFMap Server (irond) |
        | Collector | n/a         |   172.18.101.101:8086 | Up     | Established                          |
        | Discovery | Collector   |   172.18.101.100:5998 | Up     | SubscribeResponse                    |
        | Discovery | IfmapServer |   172.18.101.100:5998 | Up     | SubscribeResponse                    |
        | Discovery | xmpp-server |   172.18.101.100:5998 | Up     | Publish Response - HeartBeat         |
        +-----------+-------------+-----------------------+--------+--------------------------------------+
        
        scrape/10.85.188.101-control$ contrail-scrape-ist control nei
        Loading from introspect xml:  /root/scrape/10.85.188.101-control/bgp_peer/BgpNeighborReq
        +--------------------------------------+----------------+----------+----------+-----------+-------------+-----------------+------------+-----------------------------+
        | peer                                 | peer_address   | peer_asn | encoding | peer_type | state       | send_state      | flap_count | flap_time                   |
        +--------------------------------------+----------------+----------+----------+-----------+-------------+-----------------+------------+-----------------------------+
        | 61f75496-e2d0-4611-8f95-252490bfb5cf | 10.10.10.100   | 65513    | BGP      | external  | Active      | not advertising | 0          | n/a                         |
        | 99ae7f6e-b5a3-4120-a173-de30944d636f | 10.0.1.3       | 65478    | BGP      | external  | Active      | not advertising | 0          | n/a                         |
        | 76437055-f294-4640-ba8c-d9744c12d649 | 4.4.4.100      | 60101    | BGP      | external  | Active      | not advertising | 0          | n/a                         |
        | d173cfb2-5752-4a99-bf8e-861e3c98a3d1 | 4.4.4.100      | 60101    | BGP      | external  | Active      | not advertising | 0          | n/a                         |
        | camaro                               | 192.168.0.204  | 60100    | BGP      | internal  | Established | in sync         | 0          | n/a                         |
        | cont102                              | 172.18.101.102 | 60100    | BGP      | internal  | Established | in sync         | 1          | 2020-Nov-12 11:07:13.036137 |
        | cont103                              | 172.18.101.103 | 60100    | BGP      | internal  | Established | in sync         | 0          | n/a                         |
        | vmx                                  | 192.168.0.250  | 60100    | BGP      | internal  | Established | in sync         | 0          | n/a                         |
        | e7ef3ea8-8ed3-4b4d-b48e-72a84cef50d2 | 66.0.0.4       | 60066    | BGP      | external  | Active      | not advertising | 0          | n/a                         |
        | 7a93d6d0-77c8-4bee-adca-ab4ef1f14bce | 10.0.1.3       | 65478    | BGP      | external  | Active      | not advertising | 0          | n/a                         |
        | bcomp79                              | 172.18.79.79   | 0        | XMPP     | internal  | Established | in sync         | 0          | n/a                         |
        | bcomp80                              | 172.18.102.80  | 0        | XMPP     | internal  | Established | in sync         | 2          | 2020-Nov-11 13:45:55.916819 |
        | comp104                              | 172.18.101.104 | 0        | XMPP     | internal  | Established | in sync         | 2          | 2020-Nov-11 13:45:55.958485 |
        | comp105                              | 172.18.101.105 | 0        | XMPP     | internal  | Established | in sync         | 2          | 2020-Nov-11 13:45:56.069561 |
        | comp106                              | 172.18.102.106 | 0        | XMPP     | internal  | Established | in sync         | 2          | 2020-Nov-11 13:45:55.966067 |
        +--------------------------------------+----------------+----------+----------+-----------+-------------+-----------------+------------+-----------------------------+
        
        ```
        The output also displays the target introspect file that is being used to populate the above data. This is helpful if any data required from an introspect is missing in the output displayed by contrail-scrape-ist script.
        
        see gist: https://gist.github.com/aniruddhamonker/ce4f6943c65ec5b0bf5e9e8069c65dbd for more examples on using contrail-scrape-ist
        
Platform: UNKNOWN
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.8
Description-Content-Type: text/markdown
