ReST based network device broker

The problem with the current commit/save_config is that every device class in netmiko inherits BaseConnection which defines both a commit and save_config. Its up to child classes to implement these. The problem is that since its using an if/elif statement the save_config method never gets called for device that have implemented it (NXOS, IOS, etc).

The change here just catches the exception if the netmiko device class doesn't implement commit then tries to run save_config

These changes work fine on live devices (IOSXR, NXOS) but the pytests would fail all the time. I spent a long time on it and it seems pytest will fail when it tries to save_config. Not really sure why its failing on that but as a fix I changed the uri's to /setconfig/dry-run for the tests which bypasses these save functions. Tests pass now but just wanted to let you know.

This PR addresses some issues I was facing when attempting to push config using ncclient. Below is a summary of the issues I had when using this library.

• editconfig would return Unserializable return value from redis jobs
• lack of jinja2 templating when using ncclient

Here is a summary of the changes I am proposing.

• Move render_json boolean out of the args body making it an optional component. This allows me to render json from the reponses of getconfig and setconfig when using the ncclient library.
• render_json works for both getconfig and setconfig respones in the ncclient library.
• Made args in NcclientSetConfig model optional.
• Made j2config in NcclientSetConfig model optional.
• If you rendered j2config for ncclient it will add it to the kwargs['args']['config'] dict which gets passed to ncclient.
• ncclient_drvr.ncclien:editconfig formats the response as xml
• added a ncclient template for pytest
• added a new method for setconfig mark for pytest test_setconfig_ncclient_j2()
• fixed references to render_json in test files.

Side note: Do you have issues with github actions failing randomly for no reason? I built it several times locally and they passed but when I ran the github actions they would fail on different errors each time and eventually it passed after running more than once?

e.g. this "completes" just fine but never shows up in "gettemplates" output. Not sure if it's just the index rewrite that's failing or what.

{
	"key": "",
	"driver": "cisgo_foo",
	"command": "show asdf events"
}

When i deploy the project on a physical machine with the pinned worker model, the network worker like "rq:worker:11.0.2.2_4a06654f-0a23-4ee3-8ae6-dceb8d2b6f54" will be orphan process when the pinned worker process down, which should be killed.

I looked through the source code，the RQ module use the os.fork to create a sub process in fork_work_horse, if the subprocess exit after execution，that's right. However, the subprocess is pinned_worker_constructor, it also has a subprocess pinned_worker which is loop, not exit, when the pinned_worker_constructor completed, it will be exited, so the pinned_worker will be orphan process. The orphan process will be sending heartbeat message after default_worker_ttl, so the pinned worker state has only one filed "last_heartbeat".

Out of process will be caused by the pinned worker multiple restarts. The container environment doesn't has the problem due to container only monitor the init process which will also take over the orphan process.

Adding a capabilities flag to /getconfig/ncclient so we can get netconf capabilities of the target devices.

I figured this was better than adding an entire new endpoint.

Templates are currently loaded on container creation and containers have to be re-built if the templates are edited. It would be ideal if the template directory was reloaded automatically after a certain time period or when a template has been changed.

It seems like when a script runs into an exception, it gets caught by s_exec in netpalm/backend/plugins/calls/scriptrunner/script.py and returns the exception string to the script_exec function, which then has no way to know it failed. This makes scripts always be reported as successful. Would it be better to either remove the try/except in s_exec or maybe have it re-raise the exception or something? Am I just missing something here?

Modified "hello_world" script to fail and raise an exception:

def run(**kwargs):
        args = kwargs.get("kwargs")
        world = args.get("hello")
        return non_existent_world

Current result:

<...>
   "task_status": "finished",
    "task_result": {},
    "task_errors": []
  }

Modified scriptrunner/script.py function:

    def s_exec(self):
        module = importlib.import_module(self.script_name)
        runscrp = getattr(module, "run")
        res = runscrp(kwargs=self.arg)
        return res

Result with modified s_exec():

<...>
    "task_status": "failed",
    "task_result": null,
    "task_errors": [
      "name 'non_existent_world' is not defined"
    ]

I think I missed something on the install I can run a getconfig command but the task is telling me it can not connect to the device. I am connecting to a cisco 3750 using cisco_ios. I can ssh in to that unit just fine from the machine that Docker is running on...what did I miss?

Add interfaces service template to ease the pulling of interface details with the end goal of creating a csv which one can easily import into the Netbox bulk interface importer.

Current idea is to start with very basics and maybe just wrap this in a little CLI which can simply output the csv which one can paste into netbox. Further enhancements could include directly calling the netbox api to add said interfaces or something like that.

Notes

Netbox interface importer example: https://netboxdemo.com/dcim/interfaces/import

Login: netbox/netbox

Fields:

device, name, type, enabled, mac_address, mtu, description, mode 

• device - device name which the interface belongs to
• name - interface name
• type - physical medium (SFP, 10/100, 1G, etc.)
• enabled - boolean on/off
• mac_address - mac address
• mtu - mtu size
• description - description
• mode - 802.1Q Mode (access, tagged, tagged (all))

Used environment/collection variables for: the netpalm host itself the x-api-key header device creds that were 'admin/admin' (leaving alone those for the IOS XE REST sandbox)

Used path variables as show in screenshot for: /task/:task_id /service/:service_id

Path variables make it a bit easier/cleaner to paste values from other commands into. I, at least, find it difficult to make certain I'm not overwriting too many characters when pasting directly into the url, etc.

NOTE: Postman seems to prefer using tab indentation. Existing "bodies" were mix of tabs/spaces, so I settled on tabs.
This isn't too hard to re-do, wrote a short utility to make it easier in the future if needed (i.e. need to create a bunch of new endpoints w/o manually futzing w/ variable definitions, etc)

Please add rollback option for any task. This can be solved by two ways-

1. auto generating the delete config for service template which can be triggered when asked to rollback.
2. A rollback api can be developed which can just copy the rollback config to current running configuration device( generated during task execution by napalm)

This adds support to build netpalm in a vagrant vm.

I primarily need this for issues accessing resources over a VPN connection while developing my app.

Edit: I think this would also be useful for people that just want to spin it up quick and test as well.

Add the ability to pass an array to "ttp_template" key in netmiko args. It will match any given command in order with the template. If there's a command without a template, it should be put in the end of the command array, and it will get the raw data.

Example 1, will match command with template, in order (command1 -> template1, command2 -> template2) command: [ 'command1', 'command2' ] ttp_template: [ 'template1', 'template2' ]

Example 2, will match command with template, and command without template will return the raw data (command1 -> template1, command2 -> raw) command: [ 'command1', 'command2' ] ttp_template: [ 'template1' ]

If command and ttp_template are a string, they will be encapsulated in an array and will be matched as before.

At the moment it's not possible to pass an array of commands to getconfig endpoint, and match multiple ttp templates to the output, meaning that you can't effectively parse data correctly. When you try to do this, only the first command get's parsed correctly.

Would this be too hard to implement?

I checked the requirements.txt that genie is in the list of modules to be installed. But If I try to use it in a get Config request I get an error that it is not installed.

"args": {
   "use_genie": "true"
  },

"exception_args": [
          "\nGenie and PyATS are not installed. Please PIP install both Genie and PyATS:\npip install genie\npip install pyats\n"
        ]

Would certainly be nice to have a few new extensibles endpoints:

1. Bulk upload. Reload using the /reload-extensibles endpoint once at the end of processing the payload.
2. Delete extensibles. Bulk delete, maybe?