This repository has automation content to test Arista devices.

Overview

GitHub test workflow

Network tests automation

About this repository

This repository has automation content to test Arista devices.

To run tests programatically on your EOS devices, once you are done with the requirements described below, you simply need to indicate your devices name in a text file and to indicate the tests you would like to run in a YAML file.

This repository has also content to collect commands output on devices, to clear counters on devices, and to test the devices reachability.

This repository uses Python scripts and eAPI (EOS API). You can find examples of EOS automation with eAPI in this repository.

Requirements

Requirements on your laptop

Python 3 (at least 3.3) and some packages that are not part of the standard Python library.

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic
python -V
Python 3.9.6

The required packages are in the requirements.txt file. Clone the repository and install the requirements:

pip install -r requirements.txt

Verify

pip list | grep 'jsonrpclib-pelix\|PyYAML\|colorama\|prettytable'
colorama                   0.3.7
jsonrpclib-pelix           0.4.2
prettytable                2.5.0
PyYAML                     5.4.1

Requirements on the switches

  • Enable eAPI:
switch1#configure
switch1(config)#management api http-commands
switch1(config-mgmt-api-http-cmds)#protocol https port 443
switch1(config-mgmt-api-http-cmds)#no shutdown
switch1(config-mgmt-api-http-cmds)#vrf MGMT
switch1(config-mgmt-api-http-cmds-vrf-MGMT)#no shutdown
switch1(config-mgmt-api-http-cmds-vrf-MGMT)#exit
switch1(config-mgmt-api-http-cmds)#exit
switch1(config)#
switch1#sh run int ma1
interface Management1
   description oob_management
   vrf MGMT
   ip address 10.73.1.105/24
switch1#

Now the swicth accepts HTTPS requests on port 443 in the MGMT VRF containing a list of CLI commands.

  • Verify:
switch1(config)#sh management http-server
SSL Profile:        none
FIPS Mode:          No
QoS DSCP:           0
Log Level:          none
CSP Frame Ancestor: None
TLS Protocols:      1.0 1.1 1.2
   VRF        Server Status      Enabled Services
---------- --------------------- ----------------
   MGMT       HTTPS: port 443    http-commands

switch1>show management api http-commands
Enabled:            Yes
HTTPS server:       running, set to use port 443
HTTP server:        shutdown, set to use port 80
Local HTTP server:  shutdown, no authentication, set to use port 8080
Unix Socket server: shutdown, no authentication
VRFs:               MGMT
Hits:               83
Last hit:           2631 seconds ago
Bytes in:           11348
Bytes out:          335951
Requests:           53
Commands:           64
Duration:           9.242 seconds
SSL Profile:        none
FIPS Mode:          No
QoS DSCP:           0
Log Level:          none
CSP Frame Ancestor: None
TLS Protocols:      1.0 1.1 1.2
   User          Requests       Bytes in       Bytes out    Last hit
------------- -------------- -------------- --------------- ------------------
   arista        2              305            1235         639908 seconds ago
   ansible       51             11043          334716       2631 seconds ago

URLs
-------------------------------------
Management1 : https://10.73.1.105:443

switch1>

Quick test

Run this python script to validate the requirements and the device reachability. Use your device credentials and IP address.

from jsonrpclib import Server
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
username = "arista"
password = "arista"
ip = "10.100.164.145"
url = "https://" + username + ":" + password + "@" + ip + "/command-api"
switch = Server(url)
result=switch.runCmds(1,['show version'], 'text')
print(result[0]['output'])

List of available tests

The tests are defined in the python module functions.py in the python package tests_eos. Each function returns True or False (or None when it can not run properly).

The documentation directory has the tests documentation:

We indicate the tests we would like to run in a YAML file. Some tests require an input. In that case we provide it using the same YAML file. Here's an example tests.yaml.

Repository usage

  • Clone this repository.
  • Install the requirements (see above)

To test devices reachability

vi devices.txt
./check-devices-reachability.py --help
./check-devices-reachability.py -i devices.txt -u username

To run tests on devices

  • Update the devices inventory devices.txt with the devices IP address or hostnames.
  • Update the file tests.yaml to indicate the tests you would like to run. Some tests require an input. In that case, provide it using the same YAML file.
  • Run the python script check-devices.py.
  • Check the result in the output file.
vi devices.txt
vi tests.yaml
./check-devices.py --help
./check-devices.py -i devices.txt -t tests.yaml -o output.txt -u username
cat output.txt

To collect commands output from EOS devices

  • Update the devices inventory devices.txt with your devices IP address or hostnames.
  • Update the EOS commands list eos-commands.yaml you would like to collect from the devices in text or JSON format.
  • Run the python script collect-eos-commands.py.
  • Check the output in the output directory.
vi devices-list.text
vi eos-commands.yaml
./collect-eos-commands.py --help
./collect-eos-commands.py -i devices.txt -c eos-commands.yaml -o outdir -u username
ls outdir

To clear counters on EOS devices

vi devices-list.text
./clear_counters.py --help
./clear_counters.py -i devices.txt -u username

Demo

$ ./check-devices.py -u ansible -i devices.txt -o output.txt -t tests.yaml
Device password:
Enable password (if any):
Can not connect to device 2.2.2.2
Running tests on device 10.73.1.101 ...
Running tests on device 10.73.1.102 ...
Running tests on device 10.73.1.106 ...
Test results are saved on output.txt
$ cat output.txt
Fri Jan 28 09:48:54 2022
devices inventory file was devices.txt
devices username was ansible
list of unreachable devices is
2.2.2.2
tests file was tests.yaml

***** Results *****

+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|    devices    |  01.01  |  01.02  |  01.03  |  01.04  |  02.01  |  02.02  |  02.03  |  02.04  |  02.05  |  02.06  |  02.07  |  02.08  |  03.01  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|  10.73.1.101  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |   Pass  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |
|  10.73.1.102  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |
|  10.73.1.106  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |   Pass  |   Pass  |   Fail  |   Pass  |   Pass  |   Pass  |   Skip  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|    devices    |  03.02  |  03.03  |  03.04  |  03.05  |  04.01  |  04.02  |  05.01  |  05.02  |  06.01  |  07.01  |  08.01  |  08.02  |  08.03  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|  10.73.1.101  |   Fail  |   Pass  |   Fail  |   Skip  |   Pass  |   Fail  |   Skip  |   Skip  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
|  10.73.1.102  |   Fail  |   Pass  |   Fail  |   Skip  |   Pass  |   Fail  |   Skip  |   Skip  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
|  10.73.1.106  |   Fail  |   Pass  |   Fail  |   Skip  |   Pass  |   Fail  |   Skip  |   Skip  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|    devices    |  08.04  |  08.05  |  09.01  |  09.02  |  09.03  |  09.04  |  10.01  |  11.01  |  11.02  |  12.01  |  13.01  |  14.01  |  14.02  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|  10.73.1.101  |   Pass  |   Pass  |   Skip  |   Skip  |   Skip  |   Skip  |   Fail  |   Fail  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
|  10.73.1.102  |   Pass  |   Pass  |   Skip  |   Skip  |   Skip  |   Skip  |   Fail  |   Fail  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
|  10.73.1.106  |   Pass  |   Pass  |   Fail  |   Pass  |   Fail  |   Fail  |   Fail  |   Pass  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|    devices    |  15.01  |  16.01  |  16.02  |  16.03  |  16.04  |  16.05  |  16.06  |  16.07  |  17.01  |  18.01  |  18.02  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|  10.73.1.101  |   Pass  |   Fail  |   Pass  |   Fail  |   Skip  |   Fail  |   Skip  |   Fail  |   Fail  |   Pass  |   Fail  |
|  10.73.1.102  |   Pass  |   Fail  |   Pass  |   Fail  |   Skip  |   Fail  |   Skip  |   Fail  |   Fail  |   Skip  |   Pass  |
|  10.73.1.106  |   Pass  |   Fail  |   Pass  |   Pass  |   Skip  |   Pass  |   Skip  |   Fail  |   Fail  |   Skip  |   Fail  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+

***** Tests *****
01.01    {"name": "verify_eos_version", "versions": ["4.25.4M", "4.26.1F"]}
01.02    {"name": "verify_terminattr_version", "versions": ["v1.13.6", "v1.8.0"]}
01.03    "verify_eos_extensions"
01.04    "verify_field_notice_44_resolution"
02.01    {"name": "verify_uptime", "min": 86400}
02.02    "verify_reload_cause"
02.03    "verify_coredump"
02.04    "verify_agent_logs"
02.05    "verify_syslog"
02.06    "verify_cpu_utilization"
02.07    "verify_memory_utilization"
02.08    "verify_filesystem_utilization"
03.01    {"name": "verify_transceivers_manufacturers", "manufacturers": ["Not Present", "Arista Networks", "Arastra, Inc."]}
03.02    "verify_system_temperature"
03.03    "verify_transceiver_temperature"
03.04    "verify_environment_cooling"
03.05    "verify_environment_power"
04.01    "verify_zerotouch"
04.02    "verify_running_config_diffs"
05.01    {"name": "verify_unified_forwarding_table_mode", "mode": 3}
05.02    {"name": "verify_tcam_profile", "profile": "vxlan-routing"}
06.01    "verify_adverse_drops"
07.01    "verify_ntp"
08.01    "verify_interface_utilization"
08.02    "verify_interface_errors"
08.03    "verify_interface_discards"
08.04    "verify_interface_errdisabled"
08.05    {"name": "verify_interfaces_status", "minimum": 4}
09.01    "verify_portchannels"
09.02    "verify_illegal_lacp"
09.03    "verify_mlag_status"
09.04    "verify_mlag_interfaces"
10.01    {"name": "verify_loopback_count", "number": 3}
11.01    "verify_vxlan"
11.02    "verify_vxlan_config_sanity"
12.01    "verify_svi"
13.01    "verify_spanning_tree_blocked_ports"
14.01    {"name": "verify_routing_protocol_model", "model": "multi-agent"}
14.02    {"name": "verify_routing_table_size", "min": 2, "max": 20}
15.01    "verify_bfd"
16.01    "verify_bgp_ipv4_unicast_state"
16.02    "verify_bgp_ipv6_unicast_state"
16.03    "verify_bgp_evpn_state"
16.04    "verify_bgp_rtc_state"
16.05    {"name": "verify_bgp_evpn_count", "number": 2}
16.06    {"name": "verify_bgp_rtc_count", "number": 2}
16.07    {"name": "verify_bgp_ipv4_unicast_count", "number": 2}
17.01    {"name": "verify_ospf", "number": 3}
18.01    {"name": "verify_igmp_snooping_vlans", "configuration": "disabled", "vlans": [10, 12]}
18.02    {"name": "verify_igmp_snooping_global", "configuration": "enabled"}

Repository structure

devices.txt file

The file devices.txt is the devices inventory.

The devices inventory is a text file with the devices IP address or hostnames (if resolvable with DNS or your hosts file). This file has one device per ligne.

eos-commands.yaml file

The file eos-commands.yaml is a YAML file used to indicated the list of commands output we would like to collect from EOS devices in text or json format.

tests.yaml file

The file tests.yaml is a YAML file used to indicated the tests we would like to run. It is also used to indicated the parameters used by the tests. Each test has an identifier which is then used in the tests report. The tests are defined in the directory tests_eos.

tests_eos directory

The directory tests_eos is a python package.

The python functions to test EOS devices are defined the python module functions.py in the python package tests_eos.

generate_functions_documentation.py file

The script generate_functions_documentation.py is used to generate the functions documentation in markdown format. It requires the installation of the package lazydocs that is indicated in the file requirements-dev.txt

The functions to test EOS devices are coded in the python module functions.py in the python package tests_eos. These functions have docstrings. The docstrings are used by the script generate_functions_documentation.py to generate the functions documentation in markdown format in the directory documentation.

documentation directory

The documentation directory has the tests documentation in markdown format:

check-devices-reachability.py file

The python script check-devices-reachability.py is used to test devices reachability with eAPI.

The python script check-devices-reachability.py takes as input a text file with the devices IP address or hostnames (when resolvable), and tests devices reachability with eAPI and prints the unreachable devices on the console.

clear_counters.py file

The python script clear_counters.py is used to clear counters on EOS devices.

It takes as input a text file with the devices IP address or hostnames (when resolvable) and clears counters on these devices.

collect-eos-commands.py file

The python script collect-eos-commands.py is used to collect commands output from EOS devices.

The python script collect-eos-commands.py:

  • Takes as input:
    • A text file with the devices IP address or hostnames (when resolvable).
    • A YAML file with the list of EOS commands we would like to collect in text or JSON format.
  • Collects the EOS commands from the devices, and saves the result in files.

check-devices.py file

The python script check-devices.py is used to run tests on devices.

The python script check-devices.py:

  • Imports the python functions defined in the directory tests_eos.
    • These functions defined the tests.
  • Takes as input:
    • A text file with the devices IP address or hostnames (when resolvable).
    • A YAML file with the list of the tests we would like to use and their parameters.
  • Runs the tests, and prints the result on the console and saves the result in a file.

unit_test.py file

The python script unit_test.py is used to test the functions defined in the directory test_eos without using actual EOS devices. It requires the installation of the package pytest that is indicated in the file requirements-dev.txt

mock_data directory

The mock_data directory has data used by the python script unit_test.py to test the functions defined in the directory test_eos without using actual EOS devices.

Contribution guide

Contributions are welcome. The contribution guide is CONTRIBUTING.md

Credits

Thank you to Paul Lavelle, Colin MacGiollaEain, Matthieu Tache, Angélique Phillipps, Thomas Grimonet, Onur Gashi and Khelil Sator for their contributions and guidances.

Comments
  • feat(anta.inventory): Add support for DNS names, eAPI port and device alias names

    feat(anta.inventory): Add support for DNS names, eAPI port and device alias names

    When using a containerlab topology on macOS, we need to use port forwarding to localhost to access eAPI endpoints. The current inventory only supports IP addresses. The PR is to support hostnames (DNS names instead of IP), ports and alias names in the inventory.

    The following inventory works with this PR:

    anta_inventory:
      hosts:
      - name: spine1
        host: localhost
        port: 44301
      - name: spine2
        host: localhost
        port: 44302
      - name: leaf1
        host: localhost
        port: 44311
      - name: leaf1
        host: localhost
        port: 44312
      - name: leaf3
        host: localhost
        port: 44313
      - name: leaf4
        host: localhost
        port: 44314
    
    opened by mtache 3
  • fix(check-devices): Update --timeout management + add logging management

    fix(check-devices): Update --timeout management + add logging management

    • Fix an issue where timeout was not correctly set.
    • Fix timeout defined as an str where a float was expected
    • Add -log or --logging to set log level for inventory and result_manager

    Default mode

    check-devices.py -i .personal/avd-lab.yml -c .personal/ceos-catalog.yml --tags arista --username admin --password admin123 --table --timeout 1
    ╭──────────────────────── Settings ─────────────────────────╮
    │ Running check-devices with:                               │
    │               - Inventory: .personal/avd-lab.yml          │
    │               - Tests catalog: .personal/ceos-catalog.yml │
    ╰───────────────────────────────────────────────────────────╯
                     All tests results                  
    ┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
    ┃ Device IP ┃ Test Name ┃ Test Status ┃ Message(s) ┃
    ┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
    └───────────┴───────────┴─────────────┴────────────┘
    

    With logging enabled

    ❯ check-devices.py -i .personal/avd-lab.yml -c .personal/ceos-catalog.yml --tags arista --username admin --password admin123 --table --timeout 1 --loglevel info
    ╭──────────────────────── Settings ─────────────────────────╮
    │ Running check-devices with:                               │
    │               - Inventory: .personal/avd-lab.yml          │
    │               - Tests catalog: .personal/ceos-catalog.yml │
    ╰───────────────────────────────────────────────────────────╯
    [14:08:41] WARNING  Service not running on device 10.73.1.254 with:                                                                                                                                                                                                                                                                                    __init__.py:191
               WARNING  connection return: f('admin:[email protected]/command-api', 302, 'Found', <http.client.HTTPMessage object at 0x1062952b0>)                                                                                                                                                                                                      __init__.py:192
               WARNING  Service not running on device 10.73.1.254                                                                                                                                                                                                                                                                                          __init__.py:222
    [14:08:42] WARNING  Service not running on device 10.100.164.150 with:                                                                                                                                                                                                                                                                                 __init__.py:191
               WARNING  connection return: ftimed out                                                                                                                                                                                                                                                                                                      __init__.py:192
    [14:08:43] WARNING  Service not running on device 10.100.164.150                                                                                                                                                                                                                                                                                       __init__.py:222
               INFO     Inventory .personal/avd-lab.yml loaded                                                                                                                                                                                                                                                                                        check-devices.py:202
               INFO     starting running test on inventory ...                                                                                                                                                                                                                                                                                        check-devices.py:208
               INFO     testing done !                                                                                                                                                                                                                                                                                                                check-devices.py:226
               INFO     retrieve list of result using output_format list                                                                                                                                                                                                                                                                                   __init__.py:106
                     All tests results                  
    ┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
    ┃ Device IP ┃ Test Name ┃ Test Status ┃ Message(s) ┃
    ┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
    └───────────┴───────────┴─────────────┴────────────┘
    
    bug 
    opened by titom73 3
  • feat(reporter): Implement initial reporter using rich

    feat(reporter): Implement initial reporter using rich

    Reporting

    Provides easy reporting module

    • Get all tests results:
    ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Device IP    ┃ Test Name           ┃ Test Status ┃ Message(s)                                                                                   ┃
    ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │ 192.168.0.10 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.11 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.12 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.13 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.14 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.15 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.15 │ verify_uptime       │ success     │                                                                                              │
    │ 192.168.0.15 │ verify_reload_cause │ error       │ 'response'                                                                                   │
    │ 192.168.0.15 │ verify_coredump     │ success     │                                                                                              │
    │ 192.168.0.15 │ verify_agent_logs   │ success     │                                                                                              │
    └──────────────┴─────────────────────┴─────────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘
    
    • Get tests summarized
    ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Test Case          ┃ # of success ┃ # of failure ┃ # of errors ┃ List of failed or error nodes                                                                    ┃
    ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │ verify_eos_version │ 0            │ 6            │ 0           │ ['192.168.0.10', '192.168.0.11', '192.168.0.12', '192.168.0.13', '192.168.0.14', '192.168.0.15'] │
    └────────────────────┴──────────────┴──────────────┴─────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
    
    • Get hosts summarized
    ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Host IP      ┃ # of success ┃ # of failure ┃ # of errors ┃ List of failed ortest case                    ┃
    ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │ 192.168.0.15 │ 3            │ 1            │ 1           │ ['verify_eos_version', 'verify_reload_cause'] │
    └──────────────┴──────────────┴──────────────┴─────────────┴───────────────────────────────────────────────┘
    

    Testing

    python .github/scripts/anta-tester.py -h
    [17:03:13] INFO     ANTA program is loaded                                                                                                                                                                                                                                                                                                                 anta-tester.py:99
    usage: anta-tester.py [-h] [--inventory INVENTORY] [--catalog CATALOG] [--username USERNAME] [--password PASSWORD] [--search_ip SEARCH_IP] [--search_test SEARCH_TEST] [--verbose] [--list] [--full] [--devices] [--testcases] [--table] [--save] [--save_file SAVE_FILE]
    
    ANTA test & demo script
    
    optional arguments:
      -h, --help            show this help message and exit
      --inventory INVENTORY, -i INVENTORY
                            ANTA Inventory file
      --catalog CATALOG, -c CATALOG
                            ANTA Tests cagtalog
      --username USERNAME, -u USERNAME
                            EOS Username
      --password PASSWORD, -p PASSWORD
                            EOS Password
      --search_ip SEARCH_IP
                            search result for host
      --search_test SEARCH_TEST
                            search result for test
      --verbose, -v         Set script to verbose mode
      --list, -l            Display internal data
      --full                Display all test cases results
      --devices             Provides summary of test results per device
      --testcases           Provides summary of test results per test case
      --table               Result represented in tables
      --save                Save result to file
      --save_file SAVE_FILE
                            Generated report file
    
    framework-enhancement 
    opened by titom73 2
  • collect sheduled show tech

    collect sheduled show tech

    #!/usr/bin/env python3
    
    ## This script collects all the tech-support files stored on Arista switches flash and copies them locally
    
    import ssl
    import paramiko
    from jsonrpclib import Server
    from scp import SCPClient
    from time import strftime, gmtime
    from argparse import ArgumentParser
    from getpass import getpass
    import os
    from sys import exit
    from socket import setdefaulttimeout
    
    port = 22
    ssl._create_default_https_context = ssl._create_unverified_context
    date = strftime("%d %b %Y %H:%M:%S", gmtime())
    
    def device_directories (device, root_dir):
        cwd = os.getcwd()
        show_tech_directory = cwd + '/' + root_dir
        device_directory = show_tech_directory + '/' + device
        for directory in [show_tech_directory, device_directory]:
            if not os.path.exists(directory):
                os.makedirs(directory)
        result = show_tech_directory, device_directory
        return result
    
    def createSSHClient (device, port, username, password):
        client = paramiko.SSHClient()
        client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        client.connect(device, port, username, password)
        return client
    
    def main():
        parser = ArgumentParser(
            description='Collect all the tech-support files'
            )
        parser.add_argument(
            '-i',
            help='Text file containing a list of switches',
            dest='file',
            required=True
            )
        parser.add_argument(
            '-u',
            help='Devices username',
            dest='username',
            required=True
            )
        parser.add_argument(
            '-o',
            help='Output directory',
            dest='output_directory',
            required=True
            )
        args = parser.parse_args()
        args.password = getpass(prompt='Device password: ')
    
        try:
            with open(args.file, 'r') as file:
                devices = file.readlines()
        except:
            print('Error opening ' + args.file)
            exit(1)
    
        for i,device in enumerate(devices):
            devices[i] = device.strip()
    
        # Deletes unreachable devices from devices list
    
        unreachable = []
        
        print('Connecting to devices .... please be patient ... ')
        
        for device in devices:
            try:
                setdefaulttimeout(5)
                url = 'https://%s:%s@%s/command-api' %(args.username, args.password, device)
                switch = Server(url)
                response = switch.runCmds(1, ['enable'])
            except:
                unreachable.append(device)
        
        for item in unreachable:
            devices.remove(item)
            print("Can not connect to device " + item)
    
        # Collects all the tech-support files stored on Arista switches flash and copies them locally
    
        for device in devices:
            url = "https://" + args.username + ":" + args.password + "@" + device + "/command-api"
            try:
                ## Create one zip file named all_files.zip on the device with the all the show tech-support files in it
                cmds=['enable','bash timeout 30 zip /mnt/flash/schedule/all_files.zip /mnt/flash/schedule/tech-support/*']
                switch = Server(url)
                switch.runCmds(1,cmds, 'text')
                ## Connect to the device using SSH
                ssh = createSSHClient(device, port, args.username, args.password)
                ## Create directories 
                output_dir = device_directories (device, args.output_directory)
                ## Get the zipped file all_files.zip using SCP and save it locally
                my_path = output_dir[1] + '/' + date + '.zip'
                ssh = createSSHClient(device, port, args.username, args.password)
                scp = SCPClient(ssh.get_transport())
                scp.get("/mnt/flash/schedule/all_files.zip",local_path = my_path)
                scp.close()
                ## Delete the created zip file on the device
                cmds=['enable','bash timeout 30 rm /mnt/flash/schedule/all_files.zip']
                switch.runCmds(1,cmds, 'text')
            except:
                print('You are unlucky today! ' + device + ' does not like this script')
    
    if __name__ == '__main__':
        main()
        ```
    opened by ksator 2
  • Implement

    Implement "AntaTest" and "AntaAsyncTest" as protocols

    Sharing some ideas that could improve the framework... Need a way to enforce the format of the test functions to ensure all functions are written correctly (return boolean, etc) and the user can introduce more functions on runtime. Both normal/sequential and async modes could be supported, running the async tests in parallel on a "AntaAsyncTestRunner" and the other functions in a "AntaTestRunner". A test manager could handle the runners and dispatch the tests to each of them delegating the execution. So having the tests modelled in a flexible way could be the first step towards a more generic execution framework that can be customized/extended by the user on runtime.

    opened by gusmb 1
  • split the module `tests` of the package `anta` into multiples modules

    split the module `tests` of the package `anta` into multiples modules

    The module is too large (>1600 lines). we can split it into diff modules:

    • routing
    • hardware
    • interfaces
    • ... This will require an update of the check-devices.py
    opened by ksator 1
  • Scripts can't run due to 'jsonrpclib' incompatibility

    Scripts can't run due to 'jsonrpclib' incompatibility

    I am having an issue running the scripts, as jsonrpclib doesn't work with Python3:

    testvm:~/network_tests_automation# ./check-devices-reachability.py --help
    Traceback (most recent call last):
      File "./check-devices-reachability.py", line 5, in <module>
        from jsonrpclib import Server
    ModuleNotFoundError: No module named 'jsonrpclib'
    

    I've installed PIP dependencies as instructed, and even added 'pip install jsonrpclib' separately:

    test-vm:~/network_tests_automation# /usr/bin/env python3 -V
    Python 3.6.9
    
    testvm:~# pip list | grep jsonrpc
    jsonrpclib (0.1.7)
    jsonrpclib-pelix (0.4.3.2)
    
    

    There seems to be a known issue with this library and Python3: https://stackoverflow.com/questions/38664102/python-jsonrpclib-not-working-after-upgrade-to-python-3-5-2

    Any ideas how to get around this?

    opened by ctyrider 1
  • feat(anta.inventory): use aioeapi to handle a device session

    feat(anta.inventory): use aioeapi to handle a device session

    Using asyncio framework will help improving performances when running tests on a large network with a high number of tests. First step is to refactor the anta.inventory module to use an async library, this PR tries an attempt with aio-eapi. This PR must also refactor all the tests as coroutines and modify the scripts to run the asyncio event loop.


    Modifications to anta.inventory

    The session attribute of InventoryDevice is now a aioeapi.Device object. auto_connect is removed from the constructor: this would cause to run an asyncio event loop in an object constructor and prevent fine control from a ANTA user script on the asyncio scheduling.

    _is_device_online() uses aioeapi.Device.check_connection() to only tries to open a port on the device. _ refresh_device_fact() updates all the device flags and reduce the number of API calls compared to the previous implementation.

    connect_inventory() is a coroutine that needs to be called in a ANTA user script to establish connection towards the devices.

    async def main():
       await anta_inventory.connect_inventory()
    

    When using a containerlab topology on macOS, we need to use port forwarding to localhost to access eAPI endpoints. The current inventory only supports IP addresses. This PR supports hostnames (DNS names instead of IP), ports and alias names in the inventory.

    The following inventory works with this PR:

    anta_inventory:
      hosts:
      - name: spine1
        host: localhost
        port: 44301
      - name: spine2
        host: localhost
        port: 44302
      - name: leaf1
        host: localhost
        port: 44311
      - name: leaf1
        host: localhost
        port: 44312
      - name: leaf3
        host: localhost
        port: 44313
      - name: leaf4
        host: localhost
        port: 44314
    
    opened by mtache 2
  • Add support for custom defined tests

    Add support for custom defined tests

    Description

    In order to allow users to build their own test collection, it would be nice to add option to support catalog pointing to custom libraries.

    Today, we only look at anta.tests package but we could extend YAML format to support both anta.tests and a custom package with the following format:

    <package.name>:
          - <test function>:
               <function options>
    

    An example would be:

    anta.tests.software:
      - verify_eos_version: # Verifies the device is running one of the allowed EOS version.
          versions: # List of allowed EOS versions.
            - 4.25.4M
            - 4.26.1F
    inetsix.tests.mydesign:
      - check_topology:
    

    Potential impacts:

    • anta.loader.parse_catalog
    framework-enhancement 
    opened by titom73 0
  • Add attributes 'label' and 'description' with function decorator

    Add attributes 'label' and 'description' with function decorator

    Hello,

    Currently the only precision about the tests is the name of the function itself. It is therefore necessary to read the code of the function to understand how it works and to have more details.

    An idea would be to add to each test function a python decorator with two attributes: label and description.

    Example with this sample piece of code:

    def decorator(**kwargs):
        def wrapper(function):
            for k, v in kwargs.items():
                setattr(function, k, v)
            return function
        return wrapper
    
    @decorator(label='EOS version', description='Verifies the device is running one of the allowed EOS version')
    def verify_eos_version():
        pass
    
    print(f'label: {verify_eos_version.label}\ndescription: {verify_eos_version.description}')
    
    >>> label: EOS version
    >>> description: Verifies the device is running one of the allowed EOS version
    

    Now, we can access easily this information, without having to add two fields in a .yaml file, for example. Thank you very much.

    opened by noam-zeitoun 2
  • Implement

    Implement "AntaTest" and "AntaAsyncTest" as protocols

    Sharing some ideas that could improve the framework... Need a way to enforce the format of the test functions to ensure all functions are written correctly (return boolean, etc) and the user can introduce more functions on runtime. Both normal/sequential and async modes could be supported, running the async tests in parallel on a "AntaAsyncTestRunner" and the other functions in a "AntaTestRunner". A test manager could handle the runners and dispatch the tests to each of them delegating the execution. So having the tests modelled in a flexible way could be the first step towards a more generic execution framework that can be customized/extended by the user on runtime.

    framework-enhancement 
    opened by gusmb 2
  • Map tags to inventory devices and test catalog

    Map tags to inventory devices and test catalog

    Increase flexibility by using tags on inventory devices and the test catalog, so that catalog tests can run on relevant devices only. For instance, in s typical EVPN scenario there.might be some tests that should run only in spines and some other tests only on leaf switches, having also some tests that run on all devices. Using tags or filters on tests and inventory devices could achieve this easily and add flexibility to the way tests execute. For inventory networks/ranges, tags would apply to the whole range/network...

    framework-enhancement 
    opened by gusmb 1
Releases(v0.3.0)
  • v0.3.0(Sep 26, 2022)

    v0.3.0

    Features

    • 836f995 [anta.reporter] Add skipped column in aggregated results (#110)
    • 8c3404b [inventory] Add tag support to filter test execution (#112)
    • 40f6977 [result_manager] Support JSON output and save option (#113)

    Bug fixes

    • c010bae [check-devices] Update --timeout management + add logging management (#117)
    • cfd9d59 [fix] Missing docstring for tags (#114)

    Documentation

    Other changes

    • 39f80e4 [make] Build system: Prepare version management (#120)
    • 3262f09 [mkdocs] Build system: Fix incorrect requirement path
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0-alpha(Jul 22, 2022)

    Add the Inventory module in the anta package

    Migrated the following scripts to use the Inventory module :

    check-devices-reachability.py
    clear-counters.py
    collect-eos-commands.py
    collect-sheduled-show-tech.py
    evpn-blacklist-recovery.py
    

    This release is not in PyPi

    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Jul 2, 2022)

    Includes the following package:

    • anta with 53 functions to test EOS

    Includes the following scripts:

    • check-devices-reachability.py
    • check-devices.py
    • clear-counters.py
    • collect-eos-commands.py
    • collect-sheduled-show-tech.py
    • evpn-blacklist-recovery.py

    This release is not in PyPi

    Source code(tar.gz)
    Source code(zip)
Owner
Netdevops Community
Arista Network Automation Community (Community projects only, no direct affiliation)
Netdevops Community
The Good Old Days. | Testing Out A New Module-

The-Good-Old-Days. The Good Old Days. | Testing Out A New Module- Installation Asciimatics supports Python versions 2 & 3. For the precise list of tes

Syntax. 2 Jun 08, 2022
Just a small test with lists in cython

Test for lists in cython Algorithm create a list of 10^4 lists each with 10^4 floats values (namely: 0.1) - 2 nested for iterate each list and compute

Federico Simonetta 32 Jul 23, 2022
A Simple Unit Test Matcher Library for Python 3

pychoir - Python Test Matchers for humans Super duper low cognitive overhead matching for Python developers reading or writing tests. Implemented in p

Antti Kajander 15 Sep 14, 2022
A python bot using the Selenium library to auto-buy specified sneakers on the nike.com website.

Sneaker-Bot-UK A python bot using the Selenium library to auto-buy specified sneakers on the nike.com website. This bot is still in development and is

Daniel Hinds 4 Dec 14, 2022
A pytest plugin to skip `@pytest.mark.slow` tests by default.

pytest-skip-slow A pytest plugin to skip @pytest.mark.slow tests by default. Include the slow tests with --slow. Installation $ pip install pytest-ski

Brian Okken 19 Jan 04, 2023
AllPairs is an open source test combinations generator written in Python

AllPairs is an open source test combinations generator written in Python

Robson Agapito Correa 5 Mar 05, 2022
0hh1 solver for the web (selenium) and also for mobile (adb)

0hh1 - Solver Aims to solve the '0hh1 puzzle' for all the sizes (4x4, 6x6, 8x8, 10x10 12x12). for both the web version (using selenium) and on android

Adwaith Rajesh 1 Nov 05, 2021
Fills out the container extension form automatically. (Specific to IIT Ropar)

automated_container_extension Fills out the container extension form automatically. (Specific to IIT Ropar) Download the chrome driver from the websit

Abhishek Singh Sambyal 1 Dec 24, 2021
The successor to nose, based on unittest2

Welcome to nose2 nose2 is the successor to nose. It's unittest with plugins. nose2 is a new project and does not support all of the features of nose.

736 Dec 16, 2022
masscan + nmap 快速端口存活检测和服务识别

masnmap masscan + nmap 快速端口存活检测和服务识别。 思路很简单,将masscan在端口探测的高速和nmap服务探测的准确性结合起来,达到一种相对比较理想的效果。 先使用masscan以较高速率对ip存活端口进行探测,再以多进程的方式,使用nmap对开放的端口进行服务探测。 安

starnightcyber 75 Dec 19, 2022
Mixer -- Is a fixtures replacement. Supported Django, Flask, SqlAlchemy and custom python objects.

The Mixer is a helper to generate instances of Django or SQLAlchemy models. It's useful for testing and fixture replacement. Fast and convenient test-

Kirill Klenov 871 Dec 25, 2022
This is a web test framework based on python+selenium

Basic thoughts for this framework There should have a BasePage.py to be the parent page and all the page object should inherit this class BasePage.py

Cactus 2 Mar 09, 2022
A Library for Working with Sauce Labs

Robotframework - Sauce Labs Plugin This is a plugin for the SeleniumLibrary to help with using Sauce Labs. This library is a plugin extension of the S

joshin4colours 6 Oct 12, 2021
Fidelipy - Semi-automated trading on fidelity.com

fidelipy fidelipy is a simple Python 3.7+ library for semi-automated trading on fidelity.com. The scope is limited to the Trade Stocks/ETFs simplified

Darik Harter 8 May 10, 2022
Automated Security Testing For REST API's

Astra REST API penetration testing is complex due to continuous changes in existing APIs and newly added APIs. Astra can be used by security engineers

Flipkart Incubator 2.1k Dec 31, 2022
Selenium Manager

SeleniumManager I'm fed up with always having to struggle unnecessarily when I have to use Selenium on a new machine, so I made this little python mod

Victor Vague 1 Dec 24, 2021
catsim - Computerized Adaptive Testing Simulator

catsim - Computerized Adaptive Testing Simulator Quick start catsim is a computerized adaptive testing simulator written in Python 3.4 (with modificat

Nguyễn Văn Anh Tuấn 1 Nov 29, 2021
Kent - Fake Sentry server for local development, debugging, and integration testing

Kent is a service for debugging and integration testing Sentry.

Will Kahn-Greene 100 Dec 15, 2022
A Proof of concept of a modern python CLI with click, pydantic, rich and anyio

httpcli This project is a proof of concept of a modern python networking cli which can be simple and easy to maintain using some of the best packages

Kevin Tewouda 17 Nov 15, 2022
Pytest-rich - Pytest + rich integration (proof of concept)

pytest-rich Leverage rich for richer test session output. This plugin is not pub

Bruno Oliveira 170 Dec 02, 2022