Since I added a couple of tests, would be nice to actually run them presumably on GH actions if we can.

Also, I used pytest because it's much nicer than nose (is nose even still used nowadays)?

This is an issue where new modules need to be added - but does it make sense to actually require this setting, i.e. shouldn't it be based on what modules there are in the discovery mode? And then allow (in future) config to --exclude particular modules (or white list), but by default use all available modules?

This comes with some notes, and items which might want to be changed:

1. New 'resource' (was called modules) go into the resources folder. EAch file should contain One (or more) classes
2. Should implement the Resource class - I've steered clear of "aws" terminology here so that in future resources can be for anything - I've used 'terraform' for the terraform and 'real' for things that actually exist (or don't)... (debate naming strategy)
3. must decorate the class with the register decorator so that it links the state file to the class.

Nothing else is needed - the class instance management is handled by the scanner, and it will find all classes in that folder.

What I think is very much up for debate is this particular class interface - I don't think you can get away from the 'process state' (which is process an element in the state, not process the entire state file) being called multiple times, but perhaps the fetch_real_resources shouldn't take 'region' but a config, and in that config if its AWS it looks for aws_regions or something?

from ..resource import Resource, register

@register("aws-name-in-state-file")
class MyResource(Resource):
    _local_real_things = {}
    _local_terraform_things = {} 

    def fetch_real_resources(self, region):
        # This may be called multiple times for each region in the scan list
        # i.e. append
        raise NotImplementedError()

    def process_state_resource(state_resource):
        # This function is called potentilly multiple times as each resource
        # is discovered by the state scanner i.e. append results to local store
        raise NotImplementedError()

    def compare(self, depth):
        # this function should be called once, take the local data and return
        # an array of result elements.
        raise NotImplementedError()

More code cleanup is probably possible, I think I got all the debug entries into debug (don't try running with -vvv unless you're very brave (boto debug output becomes manic - fixing that requires a full logging config i.e. waiting for the configure file ticket).

Fixes #45 If anything is in include-resource it completely overrides exclude-resource so it's like starting from a blank list.

I went for --include-resource instead of --only so that the arguments are logically matched with --exclude-resource

Happy to change if others disagree

Logging is not initialised until the Second argparse, which means logging in the initial import scan is impossible. This means exceptions in the import are completely silent and ignored.

Some arguments need to be in both first and second - first to be parsed and second so the help works. Perhaps all core arguments?

--region is really an aws argument - i.e. --aws_region but how does argparser (which hasn't had the module scanner run) know what the extra arguments are? Does it need to run twice, once before and one after loading?

Implements https://github.com/jmons/riffdog/issues/17

Added

• RDS Cluster basics (cluster, instances, parameter groups.

Fixed

• Minor bug where the ResourceDirectory didnt have self in its str.
• In order to support foldered module structure, the dynamic importing has changed a little.

Giant homepage and docs revamp

• making the home page more welcoming / project home page
• added youtube video link (video needs review as well, see dm's)
• added mailchimp (this may need a patch) for signup to news etc
• added core team

Then

• fixed some of the help docs with some examples - because --help is quite good but brief, perhaps the command line reference morphs to being some tips and tricks type help? Don't want to really duplicate the wording of --help?
• updated release notes to include the 0.1.0 release.

This is a relatively larger refactor

1. It removes support for v3 state file (see #55 )
2. It adds support for a file based state (and folder based state) - these don't have a named arguments but just go on the end of the command.
3. it also adds support for the -b option to be a single folder or path e.g. -b bucketname/folder/ which speeds up dev/testing, especially with the --include-resource
4. It fixes a bug in the alias - where before we had a bug that the --include-resource option did not apply to the scanning in state files, that fix introduced a new one where it excluded aliased. This fix now means that it will be included if the 'root' is in the list.

In terms of testing, it now contains 2 state files, so when testing the core you can now do

$ pip install -e tests/test_resource_pack
$ riffdog -i test_resource_pack tests/test_state_files/

The tox tests install the resource pack & then check that a scan finds the right items etc.

Looks like last merge we accidentally left in some code that was there to help with local devving. Flipping that back out (commenting it) and uncommenting the actual, release ready code that scans buckets dynamically.

Using --exclude is more difficult now there are lots of resources - instead we should consider using an --only element to specify.

I don't like the word --only though so feedback and ideas here appreciated.

Whilst I added explicit support for V3 state files in 0.1.0, it seems that the structure of the individual states in the file are not compatible with V4- I had assumed they were because v4 states have schema_versions set to 0, which I assumed meant the state structure hadn't changed.

But they have.

In an upcoming PR, I have removed support for v3- explicitly with a warning.

I think we need to set an overall output flag and change the table to indicate that there were states which look like tf states but we could not understand - but I think this ties into a statistics output so that a user knows what 'coverage' they are getting of their environment.

Implement a framework on the FoundItem (and then outputs) to store a Dirty Reason list, so that you know why a thing is dirty, not just given a flag.

Consider output, and also perhaps guidance how to use it so outputs are concise.

Consider the edge case:

class Foo:
     ...

class Bar:
    depends_on = [Foo]
    ...

and then $ riffdog --exclude-resource Foo

I think without checking that this means foo's real gets executed even though its states got ignored (?) (or it will error because Foo wasn't registered, not sure where the --exclude kicks in).

At the moment we're heading rapidly towards using just command line arguments, but I think we need to support a config file especially as we head towards more complex arguments such as AWS credentials etc.

What do we want for this? YAML? (and then problems with yaml library imports - the pyyaml parser seems to have a lot of security patches which are mostly irrelevant in our case, but looks bad if we don't roll updates).

It would be nice for this to have the option to generate a pretty output in HTML form. Useful if running this in an automated environment and you want to be able to link to a page/report. Something that is a bit more human friendly.

One suggestion was to use Jinja and have it as optional?