Same as #316 just remove all the commits by a new branch instead. Removing dependency from MicroOS Create LiveISO image type with persistent storage Updated vagrant image and add VirtualBox support.

Signed-off-by: Alex Lau (AvengerMoJo) [email protected]

I'm not really sure if it's actually a bug and Python 3.7+ is required (and the issue can just be closed). Just in case it's not, I like to raise the issue here. So... on my local environment I noticed the following issue when running tox:

tox
py3 installed: aiofiles==0.6.0,attrs==20.3.0,click==7.1.2,dataclasses==0.8,fastapi==0.63.0,h11==0.12.0,importlib-metadata==3.10.0,iniconfig==1.1.1,packaging==20.9,pluggy==0.13.1,py==1.10.0,pydantic==1.7.3,pyfakefs==4.4.0,pyparsing==2.4.7,pytest==6.2.2,pytest-asyncio==0.14.0,pytest-datadir==1.3.1,pytest-mock==3.5.1,starlette==0.13.6,toml==0.10.2,typing-extensions==3.7.4.3,uvicorn==0.13.3,websockets==8.1,zipp==3.4.1
py3 runtests: PYTHONHASHSEED='2579776963'
py3 runtests: commands[0] | pytest gravel/
============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.6.12, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /.../src/aquarium/src, configfile: tox.ini
plugins: datadir-1.3.1, pyfakefs-4.4.0, asyncio-0.14.0, mock-3.5.1
collected 17 items                                                                                                                                                                                                                          

gravel/tests/unit/cephadm/test_cephadm.py .......                                                                                                                                                                                     [ 41%]
gravel/tests/unit/controllers/test_config.py ...                                                                                                                                                                                      [ 58%]
gravel/tests/unit/controllers/test_gstate.py .F                                                                                                                                                                                       [ 70%]
gravel/tests/unit/controllers/orch/test_ceph.py .....                                                                                                                                                                                 [100%]

================================================================================================================= FAILURES ==================================================================================================================
_______________________________________________________________________________________________________________ test_tickers ________________________________________________________________________________________________________________

gstate = <gravel.controllers.gstate.GlobalState object at 0x7fdae58bfb00>

    @pytest.mark.asyncio
    async def test_tickers(gstate):
        from gravel.controllers.gstate import Ticker
    
        class TestTicker(Ticker):
            def __init__(self, name):
                super().__init__(name, 1.0)
                self.has_ticked = False
    
            async def _do_tick(self) -> None:
                self.has_ticked = True
    
            async def _should_tick(self) -> bool:
                return not self.has_ticked
    
        ticker = TestTicker("test")
        assert "test" in gstate.tickers
    
>       await gstate._do_ticks()  # pyright: reportPrivateUsage=false

gravel/tests/unit/controllers/test_gstate.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <gravel.controllers.gstate.GlobalState object at 0x7fdae58bfb00>

    async def _do_ticks(self) -> None:
        for desc, ticker in self.tickers.items():
            logger.debug(f"tick {desc}")
>           asyncio.create_task(ticker.tick())
E           AttributeError: module 'asyncio' has no attribute 'create_task'

gravel/controllers/gstate.py:147: AttributeError
========================================================================================================== short test summary info ==========================================================================================================
FAILED gravel/tests/unit/controllers/test_gstate.py::test_tickers - AttributeError: module 'asyncio' has no attribute 'create_task'
======================================================================================================= 1 failed, 16 passed in 0.36s ========================================================================================================
ERROR: InvocationError: '/.../src/aquarium/src/.tox/py3/bin/pytest gravel/'

Because setup-dev.sh is ran on the developers machine, it is not okay to use the system packages which may conflict with the requirements.txt aquarium is trying to install (for example, the system packages may already supply a dependency which will mean that pip will then not install it into the venv).

Instead we should ensure the venv is created without system packages. This is what is eventually copied into the virtual machine. Before we copy it in though we need to re-enable the system packages for librados to work, so we do that manually with sed.

Signed-off-by: Joshua Hesketh [email protected]

Replace the kiwi package install with pip instead. Update the config.sh remove baseCleanMount replace the vagrant setup with baseVagrantSetup

Signed-off-by: Alex Lau (AvengerMoJo) [email protected]

This is work towards being able to have an actual package (rpm, deb, ...) of Aquarium (see https://github.com/aquarist-labs/aquarium/issues/91).

I've added a Makefile with a dist target, so you can now run make dist to get a tarball named something like aquarium-0.1.0-0-gc28eb76.tar.gz, which includes a built glass and all the necessary bits of gravel. It also includes all our python dependencies (per src/requirements.txt) in /usr/share/aquarium/venv, so we don't have to worry about not having packaged versions of these things on whatever distros we deploy on.

This doesn't change the current dev workflow at all. You still build an image by running ./tools/build-image.sh, but that script will in turn call make dist to get a tarball to install in the image. In future (once I've got a spec file and rpm build sorted out), this can be changed so that the kiwi config uses that actual package, instead of the tarball.

That said, if you don't want to build an image, but would rather try a manual Aquarium install on some random Linux, you could run make dist, then take the tarball and extract it on the system you want to deploy on. Might be a bit of fun ;-)

Note: the version string used in the tarball name is based on the latest annotated tag in the form 'vX.Y.Z', which means I've suddenly introduced the notion of actual version numbers and releases to the project. Sorry.

Fixes: https://github.com/aquarist-labs/aquarium/issues/277 Signed-off-by: Tim Serong [email protected]

Signed-off-by: Kyr Shatskyy [email protected]

Checklist

• [ ] References issues, create if required
• [ ] Updates documentation if necessary
• [ ] Includes tests for new functionality or reproducer for bug

Inspired by sesdev.

Hopefully will consolidate things a bit more for us, and especially newcomers, than a bunch of scripts spread throughout.

Stretch goal, eventually, will be to also get image builds going through this script.

Signed-off-by: Joao Eduardo Luis <[email protected]>

This is a bit rough. There's all sorts of sharp corners you can cut your fingers on. Also, I haven't actually removed the etcd bootstrap bits yet, and I haven't touched any of the tests. Still, it does actually work. Well... Almost. I just tried adding a second node, and it's failing with "assert ntp_addr" in gravel/controllers/nodes/deployment.py's join() for some reason. I'm not sure yet if that's due to this change or not. Still, feedback on the general approach would be greatly appreciated :-)

Signed-off-by: Tim Serong [email protected]

Checklist

• [ ] References issues, create if required
• [ ] Updates documentation if necessary
• [ ] Includes tests for new functionality or reproducer for bug

MicroOS using cutomized liveiso persistent script that is not easy to debug. Since we are not using all the distribution profile in the MicroOS kiwi configuration. I rewrite the kiwi config with the liveiso being persistent by the driver available storage

The default credentials set up during deployment are: admin/aquarium

Fixes: https://github.com/aquarist-labs/aquarium/issues/421 Fixes: https://github.com/aquarist-labs/aquarium/issues/23 Fixes: https://github.com/aquarist-labs/aquarium/issues/26

Signed-off-by: Volker Theile [email protected]

Checklist

• [x] References issues, create if required
• [ ] Updates documentation if necessary
• [ ] Includes tests for new functionality or reproducer for bug

This change is purely to do with formatting and also likely purely subjective.

However, there have been a few times in reviews that people have had different styles and that has caused either confusion or a desire to consolidate.

Black takes all these arguments away by being the dictator in what is correct. Read more about it here: https://black.readthedocs.io/en/stable/index.html

As this is very subjective, we should make sure people are onboard with this styling before merging it.

Seen while testing #758. Tried to join a new node to the cluster, and got this:

From a quick grep through src/gravel/controllers/deployment/join.py, it looks like we might be trying to default to https:// when calling /api/deploy/join/request on the existing host, but that's never going to work until #759 goes in.

Seen while testing #758. When joining a new node to an existing cluster, the Networking page asks if you want to use the default image, or a custom image. Why allow a different image to be specified for nodes that are joining? Shouldn't the image to use be a cluster-wide setting, and only need to be specified on the first node, when you create the cluster? Or am I missing something?

Seen while playing with #758. The Storage page shows a list of disks you can select to use for storage, and if you want to use all of them, you have to click each one individually. It'd be nice if there were a way to select all disks with one click (maybe a checkbox at the top in the header line?)

While testing #758 I had a similar experience to #707 when specifying a bogus container image name:

This results in a brief "Http failure response for [...]/api/deploy/create: 400 Bad Request" when you hit the "Install" button:

journalctl -u aquarium just says:

Dec 21 07:52:40 node1 uvicorn[1446]: INFO:     2021-12-21 07:52:40 -- mgr -- Attempt to create a new deployment.
Dec 21 07:52:42 node1 uvicorn[1446]: ERROR:    2021-12-21 07:52:42 -- create -- Unable to create deployment: pre-checks failed:
Dec 21 07:52:42 node1 uvicorn[1446]: ERROR:    2021-12-21 07:52:42 -- mgr -- Error creating a new deployment: Pre-checks failed, unable to create.

The debug log isn't really any more enlightening:

DEBUG:    2021-12-21 07:56:18 -- deploy -- Create new deployment.
INFO:     2021-12-21 07:56:18 -- mgr -- Attempt to create a new deployment.
DEBUG:    2021-12-21 07:56:18 -- create -- Preflight checks: registry='registry.opensuse.org' image='filesystems/ceph/master/.../this-does-not-exist' secure=True
DEBUG:    2021-12-21 07:56:18 -- connectionpool -- Starting new HTTPS connection (1): registry.opensuse.org:443
DEBUG:    2021-12-21 07:56:19 -- connectionpool -- https://registry.opensuse.org:443 "GET /v2/ HTTP/1.1" 200 2
ERROR:    2021-12-21 07:56:19 -- create -- Unable to create deployment: pre-checks failed: 
ERROR:    2021-12-21 07:56:19 -- mgr -- Error creating a new deployment: Pre-checks failed, unable to create.

Looking at registry_check() in src/gravel/controllers/containers.py, there seems to be plenty of good error messages attached to the various exceptions that are raised, but these don't seem to make it back into the log file, or to the UI.

Also, semi-related: should we really be splitting the Image URL into "Image registry URL" and "Image Name" in the UI? Wouldn't it be more intuitive to just have the image URL as a single field?

Signed-off-by: Mykola Golub [email protected]

Checklist

• [ ] References issues, create if required
• [ ] Updates documentation if necessary
• [ ] Includes tests for new functionality or reproducer for bug