This is the command which create a user and returns credentials for a specified bucket, optionally also creating the bucket as well.

See initial design notes in #1.

I just spotted list-buckets has the same not-quite-newline-delimited JSON output format, which is a bad default. I should fix that too.

Originally posted by @simonw in https://github.com/simonw/s3-credentials/issues> /28#issuecomment-1014838721

The initial reason for creating this tool was that I wanted to be able to create long-lived (never expiring) tokens for the kinds of use-cases described in this post: https://simonwillison.net/2021/Nov/3/s3-credentials/

Expiring credentials are fantastic for all sorts of other use-cases. It would be great if this tool could optionally create those instead of creating long-lived credentials.

This would mean the tool didn't have to create users at all (when used in that mode) - it could create a role and then create temporary access credentials for that role using sts.assume_role(): https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role

It's frustrating when using s3-credentials put-object that you have to specify the key name each time, rather than deriving that from the filename:

s3-credentials put-object simonwillison-cors-allowed-public \
  click_default_group-1.2.2-py3-none-any.whl \
  /tmp/click-default-group/dist/click_default_group-1.2.2-py3-none-any.whl

One way to fix this would be with a s3-credentials put-objects which works like this:

s3-credentials put-objects simonwillison-cors-allowed-public /tmp/click-default-group/dist/click_default_group-1.2.2-py3-none-any.whl

It could accept multiple files (hence the plural name) and could also accept directories and recursively upload their contents.

The current --policy option lets you set a custom policy, but leaves it to you to define one.

I find myself wanting to mix in the following to the policy that I use, for s3-ocr:

https://docs.aws.amazon.com/textract/latest/dg/security_iam_id-based-policy-examples.html#security_iam_async-actions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "textract:StartDocumentTextDetection",
                "textract:StartDocumentAnalysis",
                "textract:GetDocumentTextDetection",
                "textract:GetDocumentAnalysis"
            ],
            "Resource": "*"
        }
    ]
}

Would be nice if there was a neat way to do this.

The tests for this project currently run against mocks - which is good, because I don't like the idea of GitHub Action tests hitting real APIs.

But... this project is about building securely against AWS. As such, automated tests that genuinely exercise a live AWS account (and check that the resulting permissions behave as expected) would be incredibly valuable for growing my confidence that this tool works as advertised.

These tests would need quite a high level of administrative access, because they need to be able to create users, roles etc.

I don't like the idea of storing my own AWS administrator account credentials in a GitHub Actions secret though. I think I'll write these tests such that they can be run outside of GitHub Actions, maybe configured via environment variables that allow other project contributors to run tests against their own accounts.

See https://github.com/simonw/s3-credentials/issues/11#issuecomment-959844042_ for context.

The read-write policy currently uses "Action": "s3:*Object" - and the read-only one uses Action": "s3:GetObject*".

This is pretty gross - surely explicitly listing the allowed actions is better practice?

• [x] #23
• [x] #24
• [x] #25

It would be useful to have an opt-in option for saying "this bucket should be configured as a website" - because setting that up without a tool is quite fiddly.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteAccessPermissionsReqd.html has the details:

When you configure a bucket as a static website, if you want your website to be public, you can grant public read access. To make your bucket publicly readable, you must disable block public access settings for the bucket and write a bucket policy that grants public read access.

See #20 for "block public access" setting, and #19 for bucket policies.

Refs #3. This is implemented... but it doesn't seem to work - when I copy and paste the credentials into Transmit it refuses to connect.

• [x] Get it working
• [x] Add tests
• [x] Add documentation

Originally posted by @simonw in https://github.com/simonw/s3-credentials/pull/39#issuecomment-1014857276

Splitting this into a separate issue mainly so I can clearly document how to use Litestream in the comments here.

Goal is to confirm that S3 credentials created using s3-credentials create ... --prefix litestream-test/ can be used with Litestream to back up a SQLite database to that path within the bucket.

https://github.com/simonw/s3-credentials/blob/main/s3_credentials/policies.py

My suggestions:

• specify individual actions explicitly (no wildcards)
• separate permissions by resource (Buckets vs. Objects)
• Sid is unnecessary

Your read/write policy is good, but instead of *Object, list GetObject and PutObject.

Your read-only policy would be better written like your read/write policy, one section for the bucket permission (ListBucket), one for the object permission (which should be GetObject, no wildcard).

Your write-only policy is great as is.

You may want to add additional permissions to let clients set ACLs. But if it's all simple object-by-object stuff, these very simple policies are great.

Originally posted by @jdub in https://github.com/simonw/s3-credentials/issues/7#issuecomment-958651592

This came up here:

• https://github.com/simonw/public-notes/issues/9#issuecomment-1328567164

It turned out django-storages nees a write policy that includes s3:PutObjectAcl: https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#iam-policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:PutObjectAcl"
            ],
            "Principal": {
                "AWS": "arn:aws:iam::example-AWS-account-ID:user/example-user-name"
            },
            "Resource": [
                "arn:aws:s3:::example-bucket-name/*",
                "arn:aws:s3:::example-bucket-name"
            ]
        }
    ]
}

Looks like I should add s3:GetObjectAcl to the default read policies too.

Hello! I am looking into using s3-credentials for my projects. I use tags to identify resources in different ways, like how it was created or the project it belongs. I was wondering if there is planned support for adding tags to the resources created, or if you would be open for a contribution in that area.

Idea:

• --skip to skip downloading a file if it already exists with the same filename
• --skip-hash to skip downloading a file if it already exists AND the MD5 hash has not changed (more expensive as needs to calculate the local hash)

Originally posted by @simonw in https://github.com/simonw/s3-credentials/issues/78#issuecomment-1248398247

Users with multiple AWS accounts can declare named profiles to manage the different sets of credentials/regions. It would be ideal if s3-credentials accepted a --profile argument, just like the aws comand line tool.

If a bad or expired session token is set, the create command fails with a misleading error that an existing bucket doesn't exist, if --create-bucket isn't specified. If --create-bucket is specified a traceback with more info is given instead:

export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_SESSION_TOKEN="EXPIRED_TOKEN" 

$ s3-credentials create --username USERNAME BUCKET
Error: Bucket does not exist: BUCKET - try --create-bucket to create it

$ s3-credentials create --create-bucket --username USERNAME BUCKET
Traceback (most recent call last):
  File "/home/kimv/work/data_engineering/sbsa_archive/.venv/bin/s3-credentials", line 8, in <module>
    sys.exit(cli())
  File "/home/kimv/work/data_engineering/sbsa_archive/.venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/kimv/work/data_engineering/sbsa_archive/.venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/kimv/work/data_engineering/sbsa_archive/.venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/kimv/work/data_engineering/sbsa_archive/.venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/kimv/work/data_engineering/sbsa_archive/.venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/kimv/work/data_engineering/sbsa_archive/.venv/lib/python3.9/site-packages/s3_credentials/cli.py", line 314, in create
    s3.create_bucket(Bucket=bucket, **kwargs)
  File "/home/kimv/work/data_engineering/sbsa_archive/.venv/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/kimv/work/data_engineering/sbsa_archive/.venv/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ExpiredToken) when calling the CreateBucket operation: The provided token has expired.