Very much draft PR, but submitting to get early feedback:

• Chart structure - does using deployments make sense? the structure of the values.yaml make sense?
• Directory structure: Wasn't sure what the best structure? maybe install/helm instead of kubernetes/helm?
• I created a ./kubernetes/setup-kind.sh as a starting point for an end-to-end test. I plan to add an e2e test in the github actions
• Helm unit tests. I'm not familiar with python options; i saw a declarative one here. I never used it before, so wanted to get consensus before i add tests.
• Doc is very much WIP
• Releasing - how do you want this chart released? i see that you currently don't use github releases. This can also be in follow-up PR.

Description

Collect tags and enable tagging methods for Route53 Hosted Zones.

To-Dos

• [x] Add test coverage for new or updated functionality
• [x] Lint and test with tox
• [ ] Document new or updated functionality (someengineering/resoto.com#XXXX)

Code of Conduct

By submitting this pull request, I agree to follow the code of conduct.

Description

A user's atime should be the time the account was last used, a users mtime should be the time the password was last updated. The new AWS collector is missing this info.

Version

main

Environment

No response

Steps to Reproduce

No response

Logs

No response

Additional Context

No response

Motivation

Currently, the Helm Chart needs to be pulled from the GitHub repository.

Desired Behavior

The Helm Chart should be published in a repository.

Additional Context

No response

Code of Conduct

• [X] I agree to follow the Code of Conduct

Description

This avoids the need to specify every context in order to have it imported.

Added a new argument rather than a magic value like all for --k8s-context to avoid any potential conflicts.

To-Dos

• [x] Add test coverage for new or updated functionality (to the extent that there's existing coverage to extend)
• [x] Lint and test with tox
• N/A ~Document new or updated functionality (someengineering/resoto.com#XXXX)~

Code of Conduct

By submitting this pull request, I agree to follow the code of conduct.

Description

When collecting a large number of accounts *1 the graph merge fails with any version newer than v2.0.0a13.

Using https://github.com/someengineering/resoto/pull/608 (on a v2.0.0a15 image) to dump the collected graph and model:

-rw-r--r--   1 marv  staff   306M Feb  7 23:31 resoto-graph-zasag9am.ndjson
-rw-r--r--   1 marv  staff   103K Feb  7 23:27 resoto-model-bag5nlua.json

Trying the steps @lloesche documented, I was unable to reproduce the behaviour locally (macOS, current main, graphdb in Docker). The graphs imports fine, albeit very slowly (which might just be due to the graphdb being undersized etc).

Trying to understand whats going on from the provided stack trace:

1. Import process starts at 19:49:07
2. Import process dies at 19:49:40 - 33 seconds later.

This kind of leads me to believe that Queue#get(True, 30) hits its 30 second timeout. Without diving too deep yet, the underlying locking mechanism seems like it might be platform/os dependent and therefore explains why I'm not seeing the same behaviour locally.

Unfortunately I wont be able to supply the collected graph for debugging.

Version

v2.0.0a14+

Environment

Linux, Resoto one-for-all container image running on GKE

Steps to Reproduce

1. Collect a large graph, let resotoworker send it to resotocore for merge, grep logs of the spawned import process

Logs

22-02-07 19:49:01|resotoworker| INFO|85|worker-collect_cleanup  resotocore Event Handler called
22-02-07 19:49:01|resotocore| INFO|84|MainThread  127.0.0.1 [07/Feb/2022:19:49:01 +0000] "POST /graph/resoto HTTP/1.1" 200 414 "-" "python-requests/2.27.1"
22-02-07 19:49:04|resotocore| INFO|84|MainThread  127.0.0.1 [07/Feb/2022:19:49:02 +0000] "PATCH /model HTTP/1.1" 200 138466 "-" "python-requests/2.27.1"
22-02-07 19:49:05|resotocore| INFO|84|MainThread  Received merge_graph request
======== Running on http://0.0.0.0:8900 ========
(Press CTRL+C to quit)
22-02-07 19:49:07|resotocore| INFO|497|MainThread  Import process started: 497
22-02-07 19:49:07|resotocore| INFO|497|MainThread  Create ArangoHTTPClient with timeout=900 and verify=True
22-02-07 19:49:33|resotocore| INFO|84|MainThread  127.0.0.1 [07/Feb/2022:19:49:33 +0000] "GET /metrics HTTP/1.1" 200 30633 "-" "Prometheus/2.32.1"
22-02-07 19:49:40|resotocore|ERROR|497|MainThread  Update process interrupted. Preemptive Exit. 
Traceback (most recent call last):
  File "/usr/local/resoto-venv-pypy3/lib/pypy3.8/site-packages/core/model/db_updater.py", line 155, in run
    result = asyncio.run(self.setup_and_merge())
  File "/usr/local/pypy/lib/pypy3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/pypy/lib/pypy3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/pypy/lib/pypy3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/usr/local/pypy/lib/pypy3.8/asyncio/tasks.py", line 280, in __step
    result = coro.send(None)
  File "/usr/local/resoto-venv-pypy3/lib/pypy3.8/site-packages/core/model/db_updater.py", line 145, in setup_and_merge
    result = await self.merge_graph(db)
  File "/usr/local/resoto-venv-pypy3/lib/pypy3.8/site-packages/core/model/db_updater.py", line 125, in merge_graph
    nxt = self.next_action()
  File "/usr/local/resoto-venv-pypy3/lib/pypy3.8/site-packages/core/model/db_updater.py", line 120, in next_action
    return self.read_queue.get(True, 30)  # type: ignore
  File "/usr/local/pypy/lib/pypy3.8/multiprocessing/queues.py", line 108, in get
    raise Empty
queue.Empty
22-02-07 19:49:40|resotocore| INFO|497|MainThread  Shutdown initiated for current process.

Additional Context

*1 - about 2000-ish GCP projects, only few select resource types resulting in a 306M resoto-graph .ndjson file

Description

Added a CDK construct for a one command resoto deployment to AWS.

To-Dos

• [ ] Store container data on a different volume

• [ ] Document new or updated functionality (someengineering/resoto.com#XXXX)

Code of Conduct

By submitting this pull request, I agree to follow the code of conduct.

Description

When running workflows run collect, the collector for DigitalOcean fais with An unhandled error occurred while collecting team XXXXXX (where XXXXX is the our team ID).

Version

2.4.1

Environment

Docker compose running on macOS 10.15

Steps to Reproduce

1. Setup the worker as described here: https://resoto.com/docs/getting-started/configure-cloud-provider-access/digitalocean
2. Run workflows run collect
3. Check the log to see the error

Logs

resoto-resotoworker-1        | {"timestamp": "2022-09-16T11:53:39", "level": "INFO", "message": "Collecting k8s_clusters in digitalocean_team XXXXXX", "pid": 136, "thread": "digitalocean", "process": "resotoworker"}
resoto-resotoworker-1        | {"timestamp": "2022-09-16T11:53:39", "level": "INFO", "message": "Collecting snapshots in digitalocean_team XXXXXX", "pid": 136, "thread": "digitalocean", "process": "resotoworker"}
resoto-resotoworker-1        | {"timestamp": "2022-09-16T11:53:45", "level": "ERROR", "message": "An unhandled error occurred while collecting team XXXXXXX", "pid": 136, "thread": "digitalocean", "process": "resotoworker", "exception": "Traceback (most recent call last):\n  File \"/usr/local/resoto-venv-python3/lib/python3.10/site-packages/resoto_plugin_digitalocean/__init__.py\", line 66, in collect_team\n    dopc.collect()\n  File \"/usr/local/resoto-venv-python3/lib/python3.10/site-packages/resoto_plugin_digitalocean/collector.py\", line 243, in collect\n    collector()\n  File \"<decorator-gen-24>\", line 2, in collect_snapshots\n  File \"/usr/local/resoto-venv-python3/lib/python3.10/site-packages/prometheus_client/context_managers.py\", line 81, in wrapped\n    return func(*args, **kwargs)\n  File \"/usr/local/resoto-venv-python3/lib/python3.10/site-packages/resoto_plugin_digitalocean/collector.py\", line 740, in collect_snapshots\n    self.collect_resource(\n  File \"/usr/local/resoto-venv-python3/lib/python3.10/site-packages/resoto_plugin_digitalocean/collector.py\", line 372, in collect_resource\n    log.debug(f\"Parent resource for {resource_instance.rtdname} automatically set to {pr.rtdname}\")\nAttributeError: 'list' object has no attribute 'rtdname'"}
resoto-resotoworker-1        | {"timestamp": "2022-09-16T11:53:45", "level": "INFO", "message": "Collector of plugin digitalocean finished in 140.2884s", "pid": 136, "thread": "collector_digitalocean", "process": "resotoworker"}

Additional Context

No response

Description

As I was editing the Discord how-to guide, I noticed that the current Discord alerts look a little unpolished:

We should allow for users to omit the alert message entirely if they only want to specify the title. Also, the message text should be contained within the embed object for a cleaner look (and also so that the message doesn't look like a title).

To-Dos

• [ ] Add test coverage for new or updated functionality
• [ ] Lint and test with tox

Code of Conduct

By submitting this pull request, I agree to follow the code of conduct.

Description

Allow configuration of workflow schedules.

To-Dos

• [x] Add test coverage for new or updated functionality
• [x] Lint and test with tox

Code of Conduct

By submitting this pull request, I agree to follow the code of conduct.

Description

In logs section I've pasted a text version of this screenshot.

The screenshot shows how the text renders with colors in Ubuntu Terminal.

The messages that start with hashes such as "# Show all available kinds." are almost impossible to read because the background is dark blue and the text is slightly darker blue.

Version

2.0.0a16

Environment

Ubuntu 20.04.3 LTS

Steps to Reproduce

1. Start an Ubuntu Terminal session.
2. Start resh using docker run.
3. Run a help command such as help kind.
4. Struggle to see the darker blue text on a dark blue background.

Logs

> help kind
kind: Retrieves information about the graph data kinds.                                                  

┌───────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ kind [-p property_path] [name]                                                                        │
└───────────────────────────────────────────────────────────────────────────────────────────────────────┘

kind gives information about the available graph data kinds.                                             


                                                 Options                                                 

 • -p [Optional] property_path: lookup the kind for the defined property path. This will do a reverse    
   lookup and search all kinds for the specified property path.                                          


                                               Parameters                                                

 • name [Optional]: show available information about the kind with provided name.                        


                                                Examples                                                 

┌───────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                                       │
│ # Show all available kinds.                                                                           │
│ > kind                                                                                                │
│ access_key                                                                                            │
│ .                                                                                                     │
│ .                                                                                                     │
│ zone                                                                                                  │
│                                                                                                       │
│ # Show details about a specific kind.                                                                 │
│ > kind graph_root                                                                                     │
│ name: graph_root                                                                                      │
│ bases:                                                                                                │
│ - graph_root                                                                                          │
│ properties:                                                                                           │
│ - description: The name of this node.                                                                 │
│   kind: string                                                                                        │
│   name: name                                                                                          │
│   required: false                                                                                     │
│ - description: All attached tags of this node.                                                        │
│   kind: dictionary[string, string]                                                                    │
│   name: tags                                                                                          │
│   required: false                                                                                     │
│                                                                                                       │
│ # Lookup the type of the given property path in the model.                                            │
│ > kind -p reported.tags.owner                                                                         │
│ name: string                                                                                          │
│ runtime_kind: string                                                                                  │
└───────────────────────────────────────────────────────────────────────────────────────────────────────┘
>

Additional Context

No response

Motivation

In the past the resource didn't have group_ip_permissions and group_ip_permissions_egress so the pre_delete method fetched them. Now that they are being collected and part of the node that extra fetch during deletion is no longer necessary.

Desired Behavior

Use the data already present in the node, instead of fetching it again.

Additional Context

No response

Description

Currently RRSets are only collected but can't be deleted: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/route53.html#Route53.Client.change_resource_record_sets

Version

3.0.0

Environment

No response

Steps to Reproduce

No response

Logs

No response

Additional Context

No response

Description

Load balancer backends not filled in for DigitalOcean and K8S

Version

3.0.0

Environment

No response

Steps to Reproduce

No response

Logs

No response

Additional Context

No response

Motivation

I'd like to be able to save a version of my infrastructure graph, export it and import it into another Resoto installation.

Desired Behavior

> graph snapshot <snapshot_name> [graph_name]
> graph export <filename> [graph_name]
> graph import <filename> <graph_name>
> graph delete <graph_name>
> graph rename <graph_name> <new_graph_name>
> model export <filename>
> model import <filename>

Additional Context

No response

Motivation

I often find myself wanting to look at the contents of a random element of a search result. Most of the time I do

search ... | tail -1 | dump

Or the head equivalent, but if I want a third sample I need to copy'paste it from the list output.

Desired Behavior

search ... | random | dump   # returns 1 random element
search ... | random 3 | dump   # returns 3 random elements

Additional Context

No response

Motivation

kinds | grep certificate would be great, since kinds is a very long list

Desired Behavior

grep filters for string

Additional Context

No response