Howdy,

I just created a simple test to check for a few services running:

def test_passwd_file(File):
  passwd = File("/etc/passwd")
  assert passwd.contains("root")
  assert passwd.user == "root"

def test_nimsoft(Service):
  nimsoft = Service("nimbus")
  assert nimsoft.is_running

def test_uxauthd(Service):
  uxauthd = Service("uxauthd")
  assert uxauthd.is_running

def test_sshd(Service):
  sshd = Service("sshd")
  assert sshd.is_running

If I attempt to check a host via ssh, I'm told the services are failed. But if I copy the file to the remote host, install testinfra, and run it locally, everything passes:

$ testinfra --connection=ssh [email protected] --sudo test_integrations.py 
======================= test session starts =
platform linux -- Python 3.5.0+, pytest-2.8.3, py-1.4.31, pluggy-0.3.1
rootdir: /home/xian/work/validater, inifile: 
plugins: testinfra-1.0.0.0a16
collected 4 items 

test_integrations.py .FFF

================== FAILURES =======
_____________________________ test_nimsoft[ssh://evprappbpwl01.domain.name] ____

Service = <service None>

    def test_nimsoft(Service):
      nimsoft = Service("nimbus")
>     assert nimsoft.is_running
E     assert <service nimbus>.is_running

test_integrations.py:8: AssertionError
____________________ test_uxauthd[ssh://evprappbpwl01.domain.name] 

Service = <service None>

    def test_uxauthd(Service):
      uxauthd = Service("uxauthd")
>     assert uxauthd.is_running
E     assert <service uxauthd>.is_running

test_integrations.py:12: AssertionError
_______________________ test_sshd[ssh://evprappbpwl01.domain.name] 

Service = <service None>

    def test_sshd(Service):
      sshd = Service("sshd")
>     assert sshd.is_running
E     assert <service ssh>.is_running

test_integrations.py:16: AssertionError
========================= 3 failed, 1 passed in 2.51 seconds 

Versus running it on the host

[[email protected] ~]$ testinfra --sudo test_integrations.py
=========== test session starts =============================
platform linux2 -- Python 2.6.6, pytest-2.8.3, py-1.4.31, pluggy-0.3.1
rootdir: /home/hcladmin, inifile:
plugins: testinfra-1.0.0.0a16
collected 4 items

test_integrations.py ....

======================= 4 passed in 0.78 seconds ===========

Thanks!

This module could provide network tests like ping / open tcp udp connections / dns resolution.

Example:

assert host.network.resolve('google.com', type='NS', server='8.8.8.8') == set(['ns{}.google.com'.format(i) for i in range(1, 5)])
assert host.network.resolve('example.com', type='A').ttl = 1982
# ---> this could be implemented by using a smart object that resolve to a host set with additional attributes
assert host.network.reachable('tcp://google.com:443')
assert host.network.reachable('icmp://google.com')
assert host.network.not_reachable('tcp://google.com:22', timeout=3)

I think the implementation could be a mix of dig, nc, ping.

Please feel free to explain what you want to do with such module.

I'm trying out version 3.0.1 with --connection=ansible and it breaks on ansible_runner. The same command works fine with 2.1.0:

pytest -vvv --connection=ansible --ansible-inventory=hosts test_remote.py 
collected 0 items / 1 errors                                                  

=================================== ERRORS ====================================
_______________________ ERROR collecting test_remote.py _______________________
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/pluggy/hooks.py:289: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/pluggy/manager.py:68: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/pluggy/manager.py:62: in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/_pytest/python.py:238: in pytest_pycollect_makeitem
    res = list(collector._genfunctions(name, obj))
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/_pytest/python.py:414: in _genfunctions
    self.ihook.pytest_generate_tests(metafunc=metafunc)
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/pluggy/hooks.py:289: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/pluggy/manager.py:68: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/pluggy/manager.py:62: in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/testinfra/plugin.py:113: in pytest_generate_tests
    ansible_inventory=metafunc.config.option.ansible_inventory,
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/testinfra/host.py:143: in get_hosts
    for backend in testinfra.backend.get_backends(hosts, **kwargs):
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/testinfra/backend/__init__.py:91: in get_backends
    for name in klass.get_hosts(host, **kw):
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/testinfra/backend/ansible.py:59: in get_hosts
    return AnsibleRunner.get_runner(inventory).get_hosts(host)
../../../../.virtualenvs/ceph-volume/lib/python3.6/site-packages/testinfra/utils/ansible_runner.py:124: in get_hosts
    groupmatch = fnmatch.fnmatch(group, pattern)
../../../../.virtualenvs/ceph-volume/lib/python3.6/fnmatch.py:35: in fnmatch
    pat = os.path.normcase(pat)
../../../../.virtualenvs/ceph-volume/lib/python3.6/posixpath.py:54: in normcase
    s = os.fspath(s)
E   TypeError: expected str, bytes or os.PathLike object, not NoneType
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!

With 2.1.0 is fine:

pytest -vvv --connection=ansible --ansible-inventory=hosts test_remote.py
============================= test session starts =============================
platform linux2 -- Python 2.7.16, pytest-4.4.1, py-1.8.0, pluggy-0.10.0 -- /tmp/27/bin/python2
cachedir: .pytest_cache
rootdir: /home/alfredo/python/
plugins: testinfra-2.1.0
collected 2 items                                                             

test_remote.py::test_release_file[ansible://node2] PASSED               [ 50%]
test_remote.py::test_release_file[ansible://node3] PASSED               [100%]

The test file:

def test_release_file(host):
    release_file = host.file("/etc/os-release")
    assert release_file.contains('CentOS')
    assert release_file.contains('VERSION="7 (Core)"')

And the hosts file:

[nginx]
node2
node3

Running a simple test with the testinfra wrapper generates a pytest warning in the console output. The tests pass/fail correctly and the command exits with a zero status code, but the warning is a little concerning. The warning is:

WP1 None Modules are already imported so can not be re-written: testinfra

Testinfra 1.42 Pytest 3.03 Python 2.7.12 macOS 10.12

$ cat mytest.py
def test_passwd_file(File):
    passwd = File("/etc/passwd")
    assert passwd.contains("root")
    assert passwd.user == "root"
    assert passwd.group == "wheel"
    assert passwd.mode == 0o644

(venv)
$ testinfra -v mytest.py
==================================================================== test session starts =====================================================================
platform darwin -- Python 2.7.12, pytest-3.0.3, py-1.4.31, pluggy-0.4.0 -- /Users/me/testing/venv/bin/python2.7
cachedir: .cache
rootdir: /Users/me/testing, inifile:
plugins: testinfra-1.4.2
collected 1 items

mytest.py::test_passwd_file[local] PASSED

=================================================================== pytest-warning summary ===================================================================
WP1 None Modules are already imported so can not be re-written: testinfra
======================================================== 1 passed, 1 pytest-warnings in 0.04 seconds =========================================================```

Here is the performance I see for 2 simple tests on different backends. It seems that ansible is much, much, much slower ! Admittedly ansible needs more time to update its inventory, but the task themselves are slow as if it were connecting to the machine at every single test.

Here is some sample output: 1/ test with IP provided 2/ test with ansible

<!> Also output is not the same for some unknown reason

$ testinfra [email protected],[email protected],[email protected],[email protected],[email protected] main.py ================================================= test session starts ================================================= platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 rootdir: /home/emayssat/workspaces/emayssat/devtools-vpc/devops/aws/release/tests, inifile: plugins: testinfra-1.0.0.0a21 collected 10 items

main.py ..........

============================================== 10 passed in 7.62 seconds ============================================== (aws)~/workspaces/emayssat/devtools-vpc/devops/aws/release/tests [email protected] $ make execute_tests AWS_PROFILE=7009 testinfra -v --connection=ansible --ansible-inventory=inventories/ec2_7009 --hosts=key_Release001 main.py ================================================= test session starts ================================================= platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /home/emayssat/.virtualenvs/aws/bin/python cachedir: .cache rootdir: /home/emayssat/workspaces/emayssat/devtools-vpc/devops/aws/release/tests, inifile: plugins: testinfra-1.0.0.0a21 collected 10 items

main.py::test_passwd_file[ansible://52.8.21.124] PASSED main.py::test_surrogate_file[ansible://52.8.21.124] PASSED main.py::test_passwd_file[ansible://54.67.33.158] PASSED main.py::test_surrogate_file[ansible://54.67.33.158] PASSED main.py::test_passwd_file[ansible://52.9.29.73] PASSED main.py::test_surrogate_file[ansible://52.9.29.73] PASSED main.py::test_passwd_file[ansible://54.67.6.175] PASSED main.py::test_surrogate_file[ansible://54.67.6.175] PASSED main.py::test_passwd_file[ansible://54.67.81.102] PASSED main.py::test_surrogate_file[ansible://54.67.81.102] PASSED

============================================= 10 passed in 82.07 seconds ============================================== (aws)~/workspaces/emayssat/devtools-vpc/devops/aws/release/tests [email protected] $ make execute_tests AWS_PROFILE=7009 testinfra -v --connection=ansible --ansible-inventory=inventories/ec2_7009 --hosts=key_Release001 main.py ================================================= test session starts ================================================= platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /home/emayssat/.virtualenvs/aws/bin/python cachedir: .cache rootdir: /home/emayssat/workspaces/emayssat/devtools-vpc/devops/aws/release/tests, inifile: plugins: testinfra-1.0.0.0a21 collected 10 items

main.py::test_passwd_file[ansible://52.8.21.124] PASSED main.py::test_surrogate_file[ansible://52.8.21.124] PASSED main.py::test_passwd_file[ansible://54.67.33.158] PASSED main.py::test_surrogate_file[ansible://54.67.33.158] PASSED main.py::test_passwd_file[ansible://52.9.29.73] PASSED main.py::test_surrogate_file[ansible://52.9.29.73] PASSED main.py::test_passwd_file[ansible://54.67.6.175] PASSED main.py::test_surrogate_file[ansible://54.67.6.175] PASSED main.py::test_passwd_file[ansible://54.67.81.102] PASSED main.py::test_surrogate_file[ansible://54.67.81.102] PASSED

============================================= 10 passed in 47.62 seconds ============================================== (aws)~/workspaces/emayssat/devtools-vpc/devops/aws/release/tests [email protected] $ make execute_tests AWS_PROFILE=7009 testinfra -v --connection=ansible --ansible-inventory=inventories/ec2_7009 --hosts=key_Release001 main.py ================================================= test session starts ================================================= platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /home/emayssat/.virtualenvs/aws/bin/python cachedir: .cache rootdir: /home/emayssat/workspaces/emayssat/devtools-vpc/devops/aws/release/tests, inifile: plugins: testinfra-1.0.0.0a21 collected 10 items

main.py::test_passwd_file[ansible://52.8.21.124] PASSED main.py::test_surrogate_file[ansible://52.8.21.124] PASSED main.py::test_passwd_file[ansible://54.67.33.158] PASSED main.py::test_surrogate_file[ansible://54.67.33.158] PASSED main.py::test_passwd_file[ansible://52.9.29.73] PASSED main.py::test_surrogate_file[ansible://52.9.29.73] PASSED main.py::test_passwd_file[ansible://54.67.6.175] PASSED main.py::test_surrogate_file[ansible://54.67.6.175] PASSED main.py::test_passwd_file[ansible://54.67.81.102] PASSED main.py::test_surrogate_file[ansible://54.67.81.102] PASSED

============================================= 10 passed in 54.28 seconds ============================================== (aws)~/workspaces/emayssat/devtools-vpc/devops/aws/release/tests

Switch to ansible-runner instead of directly interfacing with ansible

Based on @jctanner gist at https://gist.github.com/jctanner/512ec07171d95e61a5b7fd5f9103a301

I hope we can collaborate around something like this. I'm making a common pull request but please update ... our infra can test pull requests from the main repo

Not ready to merge, obviously, but this reliably works for the backend tests (and my local tests) with both local and remote ansible that go via the shell module.

The two tests that fail are not using shell output. As mentioned inline; I can't seem to get the json response from ansible-runner reliably. I get the feeling ansible-runner is sometimes dropping some of the output of the ansible process in it's stdout descriptor it returns. This is a problem for the things not calling the shell module.

Instead of adding a keyword argument for each supported option, allow users to pass arguments directly to the ansible command line.

Support for passing extra vars has been proposed in https://github.com/philpep/testinfra/pull/462.

This approach offers support for extra vars, as well as become user, verbosity, running in diff mode, etc.

Supercedes https://github.com/philpep/testinfra/pull/410

fixes https://github.com/philpep/testinfra/issues/401 fixes https://github.com/ansible/molecule/issues/1727

There's a couple reasons for this pullrequest ...

1. The ansible python api is unstable, and upkeep for testinfra will continue to be a pain point.
2. ansible-runner is how Ansible Tower and AWX communicate with ansible and will be perpetually maintained
3. testinfra is Apache licensed and imports ansible which is GPL licensed. This violates GPL for most people's interpretation of GPL https://opensource.stackexchange.com/a/1641

NOTE: Molecule can not yet use this version until an incompatibility with colorama is fixed https://github.com/ansible/molecule/pull/2001

this pull request is an attempt at implementing docker module, which allows you to test whether or not certain docker containers are running on a particular connection backend.

this pull request references issue #264

when upgrading to the latest version of testinfra with the lastest version of py.test (see above) it seems parametrization is broken:

def test_work(File):
    assert True

testinfra test.py

================================================================================================ test session starts ================================================================================================
platform linux2 -- Python 2.7.6, pytest-3.0.2, py-1.4.31, pluggy-0.3.1
rootdir: /, inifile: 
plugins: testinfra-1.4.1
collected 0 items / 1 errors 

====================================================================================================== ERRORS =======================================================================================================
_____________________________________________________________________________________________ ERROR collecting /test.py _____________________________________________________________________________________________
usr/local/lib/python2.7/dist-packages/testinfra/plugin.py:128: in pytest_generate_tests
    "_testinfra_backend", params, ids=ids, scope="module")
usr/local/lib/python2.7/dist-packages/_pytest/python.py:837: in parametrize
    raise ValueError(msg % (saferepr(id_value), type(id_value).__name__))
E   ValueError: ids must be list of strings, found: 'local' (type: unicode)
============================================================================================== pytest-warning summary ===============================================================================================
WP1 None Modules are already imported so can not be re-written: testinfra
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================================================== 1 pytest-warnings, 1 error in 0.22 seconds =====================================================================================

when downgrading py.test to 3.0.1 everything works fine

when parameters are left away it will also work:

def test_work():
    assert True

we could reproduce the issue on the following os using native python (python2) with testrinfra installed via pip:

• debian 7,8
• ubuntu 1204,1404,1604
• centos 5,6,7
• sles 11.4,12.1

Is there a way to access ansible inventory inside the test itself?

I am putting together a set of tests that run across multiple environments. I would like some of the tests to be explicit about the domain or URL it is testing. This information is stored in ansible's inventory. The test below feels wrong (hostname -d).

@pytest.mark.parametrize('host, scheme', [('api.pub.{}', 'https'), ('api.loc1.{}', 'http'),
])
def test_keystone_service_list(Command, host, scheme):
    domain = Command('hostname -d').stdout
    cmd = openstack('service list',
    password='keystone_admin_password',
    auth_host=host.format(domain),
    auth_host_scheme=scheme)
    out = Command.check_output(cmd)

    assert re.search(r'image', out)

Ideally, I'd like to run this same test across multiple environments, swapping out the inventory.

@pytest.mark.parametrize('host, scheme', [
    ('api.pub.{}.format(VAR_FROM_INVENTORY)', 'https'),
    ('api.loc1.{}.format(VAR_FROM_INVENTORY)', 'http'),
])
...

Pytest has deprecated markers for hooks configuration. See: https://docs.pytest.org/en/latest/deprecations.html#configuring-hook-specs-impls-using-markers

Each envocation of pytest with python warning enabled would produce the following message: PytestDeprecationWarning: The hookimpl pytest_configure uses old-style configuration options (marks or attributes).

Please use the pytest.hookimpl(trylast=True) decorator instead

This commit fixes that issue.

When using this today, I struggled to find how this works in the documentation. This hopefully gives some pointers for people how to structure their pattern. I wanted to find out how to search for "term1" or "term2" which in gnu grep (or extended grep in other grep versions) would be "term1|term2" but didn't include this as this depends too much on the grep used. Feel free to change or ask for improvements. Not fully satisfied yet and did this via web so not sure if flake8 is happy.

This change is adding Zoned Block Device support to LinuxBlockDevice in BlockDevice Module.

Currently cat-ing a /sysfs/block/<dev name>/queue/<zoned related file>s is used query a drive for ZBD options.

Added properties "zoned", "zoned_type" and populated LinuxBlockDevice._data with ZBD configuration values.

Added get_zoned_param() method for querying the zoned device (ZBD-related) configuration values.

Configuration values names allowed as parameters in get_zoned_param() queries:

• "zoned_type"
• "chunk_sectors"
• "nr_zones"
• "zone_append_max_bytes"
• "max_open_zones"
• "max_actives_zones".

So I am looking to get rid of an install process in a bash script outside of config mgmt handling.

Could I just create a single python script using testinfra, take advantage of its execution (ie invoke), output, analysis, assertion capabilities and not use it as a separate script. Essentially in the Arrange, Act, Assert pattern, Can I just do the real thing in Arrange and Assert?

Example I dont want to just test a service is running in my testinfra script. I want to start it. Then I can handle complex idempotency concerns beyond config mgmt. Then write complex assertions.

Is this frowned upon? Isn't this a way better than starting a service in bash and checking its running, service listening, accepting conns, etc? I dont want to unit test my script outside itself because the functions all change real inf, and mocking is futile. Plus it would be nice to just get the execution of testinfra as the deployer.

Might this be the new way?

Details

I am wanting to write a custom TestInfra module for something that isn't provided by TestInfra by default. So, I wanted to test this out by doing something trivial. So, I looked at the testinfra/modules/podman.py file and thought I had identified how this should work, but I am not doing it correctly. Here is what my module is doing.

"""TestInfra Plugin Module
"""

from testinfra.modules.base import Module


class Myname(Module):
    """Myname Class. It inherits the TestInfra Module class.
    """

    def __init__(self, name):
        """Class Constructor
        """
        self.name = name
        super().__init__()

    @property
    def is_dudley(self):
        """Verify that the string is Dudley
        """
        string: str = "Dudley"

        return string

I then used Poetry to install the module. Then I use the following test file:

"""Test the TestInfra Module
"""


def test_is_dudley(host):
    """Test if is_dudley returns Dudley.
    """
    dudley = host.myname("phillip")
    assert dudley.is_dudley == "Dudley"

Then I ran the following command to attempt to run the module.

poetry run py.test

Then I got the following error.

➜  pytest-testinfra-dudley
> poetry run py.test
================ test session starts ================
platform linux -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/home/filbot/bluekc/development/pytest-testinfra-dudley
plugins: testinfra-6.8.0
collected 1 item

tests/test_pytest_testinfra_dudley.py F                                                                                                                                                                       [100%]

=============== FAILURES ========================
_______________ test_is_dudley[local] ____________________

host = <testinfra.host.Host local>

    def test_is_dudley(host):
        """Test if is_dudley returns Dudley.
        """
>       dudley = host.myname("phillip")

tests/test_pytest_testinfra_dudley.py:8:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <testinfra.host.Host local>, name = 'myname'

    def __getattr__(self, name):
        if name in testinfra.modules.modules:
            module_class = testinfra.modules.get_module_class(name)
            obj = module_class.get_module(self)
            setattr(self, name, obj)
            return obj
>       raise AttributeError(
            "'{}' object has no attribute '{}'".format(self.__class__.__name__, name)
        )
E       AttributeError: 'Host' object has no attribute 'myname'

../../../.cache/pypoetry/virtualenvs/pytest-testinfra-dudley-djwyNVfS-py3.10/lib/python3.10/site-packages/testinfra/host.py:120: AttributeError
=============== short test summary info ==============
FAILED tests/test_pytest_testinfra_dudley.py::test_is_dudley[local] - AttributeError: 'Host' object has no attribute 'myname'
=============== 1 failed in 0.03s =================

Question

Is there any documentation on how to extend TestInfra with a custom module plugin? I'm not doing something correctly.