Upload files or folders (even with subfolders) to Amazon S3 in a totally automatized way taking advantage of:
-
Amazon S3 Multipart Upload: The uploaded files are processed transparently in parts improving the throughput and the quick recovery from any network issues.
-
Resilent Retry System: Intelligent S3 Upload has been built to detect any error during the uploading process and to perform any retries whenever is necessary.
-
User Friendly Interface: Just check the demo to see with your own eyes how the upload process is performed.
Installation
Clone the repository
git clone https://github.com/polius/intelligent-s3-upload.git
Install the dependencies
python3 -m pip install boto3 --user
python3 -m pip install requests --user
Setup
Before executing the Intelligent S3 Upload, modify the credentials.json file.
{
"aws_access_key_id": "",
"aws_secret_access_key": "",
"region_name": "",
"bucket_name": "",
"bucket_prefix": "",
"storage_class": "",
"skip_s3_existing_files": true,
"server_side_encryption": true,
"slack_url": ""
}
- aws_access_key_id | aws_secret_access_key: Credentials generated by Amazon IAM.
- region_name: The AWS Region Code where the bucket is located.
- bucket_name: The bucket name created by Amazon S3.
- bucket_path: (Optional) The bucket path to store the uploaded objects.
- storage_class: The type of storage to use for the uploaded object. These are the possible values:
| storage_class |
|---|
| STANDARD |
| REDUCED_REDUNDANCY |
| STANDARD_IA |
| ONEZONE_IA |
| INTELLIGENT_TIERING |
| GLACIER |
| DEEP_ARCHIVE |
| OUTPOSTS |
- skip_s3_existing_files: Skip uploading objects if these already exists in S3. Possible values: [ true | false ]
- server_side_encryption: Enable Server-side encryption using the Amazon S3 key (SSE-S3). Possible values: [ true | false ]
- slack_url: (Optional) Enter a Webhook URL to send a message to Slack when a upload finishes.
AWS Policy
To be able to run the script check that your policy meets the following requirements:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:ListBucketMultipartUploads",
"s3:AbortMultipartUpload",
"s3:ListBucket",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::yourbucket/*",
"arn:aws:s3:::yourbucket"
]
}
]
}
Execution
python3 upload.py --path "{PATH}"
Replace the {PATH} string with the absolute file/folder path.
License
This project is licensed under the MIT license. See the LICENSE file for more info.
1 Dec 13, 2021
428 Jan 8, 2023
71 Jul 19, 2022
1 Nov 6, 2021
0 Dec 18, 2021
4 Dec 10, 2021
240 Jan 6, 2023
219 Oct 3, 2022
59 Nov 27, 2022
2 Apr 26, 2022
97 Dec 21, 2022
1 Jan 29, 2022
2 Jun 02, 2022
73 Jan 01, 2023
4 Dec 13, 2021
3 Sep 30, 2022
9 Dec 20, 2022
1.7k Jan 07, 2023
11 Oct 21, 2022
16 Nov 06, 2022
4 Oct 20, 2022
4 Aug 04, 2022
304 Jan 01, 2023
255 Dec 28, 2022
3 Aug 31, 2022
8 Oct 28, 2022
0 Dec 04, 2021
95 Nov 11, 2022
6 May 30, 2022