Asynchronous, fast, pythonic DynamoDB Client

Overview

AsyncIO DynamoDB

CircleCI Code style: black Documentation Status

Asynchronous pythonic DynamoDB client; 2x faster than aiobotocore/boto3/botocore.

Quick start

With httpx

Install this library

pip install "aiodynamo[httpx]" or, for poetry users poetry add aiodynamo -E httpx

Connect to DynamoDB

from aiodynamo.client import Client
from aiodynamo.credentials import Credentials
from aiodynamo.http.httpx import HTTPX
from httpx import AsyncClient

    async with AsyncClient() as h:
        client = Client(HTTPX(h), Credentials.auto(), "us-east-1")

With aiohttp

Install this library

pip install "aiodynamo[aiohttp]" or, for poetry users poetry add aiodynamo -E aiohttp

Connect to DynamoDB

from aiodynamo.client import Client
from aiodynamo.credentials import Credentials
from aiodynamo.http.aiohttp import AIOHTTP
from aiohttp import ClientSession

    async with ClientSession() as session:
        client = Client(AIOHTTP(session), Credentials.auto(), "us-east-1")

API use

        table = client.table("my-table")

        # Create table if it doesn't exist
        if not await table.exists():
            await table.create(
                Throughput(read=10, write=10),
                KeySchema(hash_key=KeySpec("key", KeyType.string)),
            )

        # Create or override an item
        await table.put_item({"key": "my-item", "value": 1})
        # Get an item
        item = await table.get_item({"key": "my-item"})
        print(item)
        # Update an item, if it exists.
        await table.update_item(
            {"key": "my-item"}, F("value").add(1), condition=F("key").exists()
        )

Why aiodynamo

  • boto3 and botocore are synchronous. aiodynamo is built for asynchronous apps.
  • aiodynamo is fast. Two times faster than aiobotocore, botocore or boto3 for operations such as query or scan.
  • aiobotocore is very low level. aiodynamo provides a pythonic API, using modern Python features. For example, paginated APIs are automatically depaginated using asynchronous iterators.
  • Legible source code. botocore and derived libraries generate their interface at runtime, so it cannot be inspected and isn't typed. aiodynamo is hand written code you can read, inspect and understand.
  • Pluggable HTTP client. If you're already using an asynchronous HTTP client in your project, you can use it with aiodynamo and don't need to add extra dependencies or run into dependency resolution issues.

Complete documentation is here

Comments
  • question: Any chance for transaction support?

    question: Any chance for transaction support?

    I'm wondering if there are any plans to support TransactWriteItems or just Transactions in general.

    I can probably work on this if/when I have more time 🤔

    opened by stupoid 16
  • add AWS_PROFILE env var to credentials chain

    add AWS_PROFILE env var to credentials chain

    Adds support for using AWS_PROFILE env var in the credentials chain.
    This behavior is more in line with the behavior expected by users coming from boto3 or aiobotocore. Added a unit test to cover this feature.

    Relates to: Issue #139

    opened by BTripp1986 8
  • Http wrapping issues

    Http wrapping issues

    When I was adding credentials handler to assume role with web identity which we need for our EKS setup I came across few issues why I am for now implementing it as custom code in our own project instead of providing it for the library.

    Http wrappers assume response is json, https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py but as endpoint response is xml (and at least I can't find any mention of possibility to request it as json, but you know AWS docs 💩 ). This is blocker and thought that perhaps there needs to be some refactoring around http handling which takes too long for us.

    Some additional things I noticed

    1. in MetadataCredentials there is fetch_with_retry which couldn't use because it assumes GET but endpoint for assume role is POST. Was thinking should/could this retry option be in base http classes?

    2. Missing timeout in POST, would prefer to have timeout also for this credentials call. As mentioned in https://github.com/HENNGE/aiodynamo/issues/45

    3. http.post(..) requires body as bytes even when it can in aiohttp and httpx also be None. https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py#L28-L30

    opened by jarikujansuu 6
  • Add support for pay_per_request billing mode during table creation

    Add support for pay_per_request billing mode during table creation

    Currently aiodynamo creates tables using the provisioned billing mode. DynamoDB also supports pay_per_request where no throughput config is provided. This PR is backwards compatible, but also adds the support to provide a billing_mode parameter to create_table. The throughput parameter has been changed to optional as the pay_per_request billing mode doesn't use it.

    PS. I'd be very grateful if you could also add the hacktoberfest-accepted label to this PR once it is accepted so that it can be counted as a hacktoberfest PR. Thanks :)

    hacktoberfest-accepted 
    opened by TheEdgeOfRage 6
  • Handle ClientConnectorError in connector

    Handle ClientConnectorError in connector

    Exception text: ClientConnectorError: Cannot connect to host dynamodb.<snip>.amazonaws.com:443 ssl:default [None]

    Exception type: aiohttp.client_exceptions.ClientConnectorError

    Stack:

    aiohttp/connector.py in _wrap_create_connection at line 943
    aiohttp/connector.py in _create_direct_connection at line 980
    aiohttp/connector.py in _create_direct_connection at line 1004
    aiohttp/connector.py in _create_connection at line 858
    aiohttp/connector.py in connect at line 523
    aiohttp/client.py in _request at line 480
    aiohttp/client.py in __aenter__ at line 1012
    aiodynamo/http/aiohttp.py in post at line 31
    aiodynamo/client.py in send_request at line 651
    aiodynamo/client.py in get_item at line 454
    aiodynamo/client.py in get_item at line 124
    
    opened by dimaqq 6
  • Raise asyncio.TimeoutError on httpx.TimeoutException

    Raise asyncio.TimeoutError on httpx.TimeoutException

    Now asyncio.TimeoutError is handled in the client logic. But httpx wrapper raises httpx.TimeoutException. This pull request convert the exception and make the client be able to handle timeouts in httpx.

    ref.) httpx timeout exceptions https://github.com/encode/httpx/blob/master/httpx/_exceptions.py#L7-L11

    ref.) Custom HTTP Client Adaptor document https://github.com/HENNGE/aiodynamo/blob/2e6c4716a3ac9fe5669bbdcaa73e6bbe0f73cfbb/docs/advanced.rst

    opened by gunyarakun 5
  • Count limits

    Count limits

    This was motivated by our real-life use case. We sometimes need to limit counting to preserve latency and Dynamo resources. I tried using query_single_page but it doesn't support count queries, so I just added limit to count.

    opened by Tinche 5
  • Use 3rd-party ddbcereal library for deserialize/serialize.

    Use 3rd-party ddbcereal library for deserialize/serialize.

    Figured I'd put this out here as an option. I've been working on a serializer/deserializer lib to supplement my use of aiobotocore, but it might make sense for this project too. It should be as fast or faster than this one.

    Overall, it reduces/simplifies the code, but the initialization of Client is a tad uglier due to ddbcereal needing a pre-constructed Deserializer and Client being frozen. It could be made pretty by unfreezing it or alternatively, making a seperate client-spawning function that passes the Deserializer to the Client constructor.

    opened by JustinTArthur 5
  • Don't send `StreamSpecification: null`

    Don't send `StreamSpecification: null`

    https://github.com/HENNGE/aiodynamo/blob/81b6f98111ce4bed41aeb8d07747695af836c76a/src/aiodynamo/client.py#L41-L49

    Sends StreamSpecification=None when none is supplied by the caller. Perhaps aiodynamo should not inject this key if the value is None.

    It seems that's OK for AWS SaaS dynamo, dynamodb-local and dynalite, but not ScyllaDB:

    https://github.com/scylladb/scylla/issues/5796

    opened by dimaqq 5
  • How to update map element using a pattern?

    How to update map element using a pattern?

    Described in https://dev.to/matthewvielkind/updating-values-in-dyanmodb-map-attributes-can

    UpdateExpression="SET players.#player_id.score = :score_val",  
    ExpressionAttributeNames={"#player_id": player_id},  
    

    I'm not sure what the API is in this case... Would I need to overload F?

    opened by dimaqq 4
  • Support consistent read

    Support consistent read

    Purpose

    Support consistent read in get_item, query, scan, count and batch get operations.

    Note: Tests do not actually check anything except that DynamoDb calls do not explode if using consistent_read option. Local DynamoDb is always strongly consistent. And might want to reduce copied code in them.

    opened by jarikujansuu 4
  • Empty-set safety

    Empty-set safety

    aiodynamo used to be empty-string-safe when DynamoDB didn't support empty string values. This mercifully got fixed in DynamoDB, but DynamoDB still doesn't accept empty sets (NS, SS, BS). There's an argument to be made that aiodynamo should handle this automatically, converting F("some_set").set(set()) to F("some_set").remove(), possibly with a warning.

    opened by ojii 3
  • Shall we try goodfirstissue.dev ?

    Shall we try goodfirstissue.dev ?

    links: https://goodfirstissue.dev/language/python https://github.com/deepsourcelabs/good-first-issue#adding-a-new-project

    I think this project may qualify, and would get some exposure and helping hands

    opened by dimaqq 0
  • Support AWS Web Identity Token File authentication

    Support AWS Web Identity Token File authentication

    aiodynamo ~~doesn't support AWS Web Identity Token File authentication~~ doesn't support AWS Web Identity Token File authentication out of the box.

    It looks like a typical case. For my team, we would like to use that authentication in our services in k8s by using IAM Roles for Service Accounts(IRSA)

    Related issue: https://github.com/HENNGE/aiodynamo/issues/128 Related PR: https://github.com/HENNGE/aiodynamo/pull/127

    There was a similar issue, but I decided to create another one because I think we can describe our goal and issue more directly.

    So, as discovered in a related PR, we should implement the AssumeRoleWithWebIdentity API for this authentication.

    opened by mrkovalchuk 2
  • Auth Error With No Default Credentials

    Auth Error With No Default Credentials

    I have access to several AWS accounts via SSO and no default profile for AWS. When I sign in to AWS via SSO the .aws/credentials file is updated with a fresh Access Key, Secret Key, and Security Token.

    I specify which credentials I will use with the AWS_PROFILE env var, which is checked as part of the credentials chain lookup. If using boto3 or the aws-cli this method works. With this method I can sign in via SSO to dev and prod accounts. Then if I AWS_PROFILE=dev aws s3 ls(or similar call in boto3) it will run the command against the dev account. Then if I AWS_PROFILE=prod aws s3 ls it will run against prod.

    This doesn't seem to work with aiodynamo. In order to get it to work I need to create a default profile and then copy/paste the credentials from the desired profile into the default profile. Aiodynamo should respect the AWS_PROFILE env var.

    opened by BTripp1986 1
  • Surprising behaviour in `Client.table_exists`

    Surprising behaviour in `Client.table_exists`

    We use Table.exists as part of a health check for the containers in our application, which in turn calls Client.table_exists. Today we had a short outage when all health checks began failing. It turned out the Dynamo table had entered UPDATING status.

    The implementation does the following:

    async def table_exists(self, name: TableName) -> bool:
            try:
                description = await self.describe_table(name)
            except TableNotFound:
                return False
    
            return description.status is TableStatus.active
    

    This feels like a bug to us: UPDATING is a valid state while a variety of operations are being applied to the table. So the code should do return description.status in [TableStatus.active, TableStatus.updating].

    Thoughts?

    opened by darylweir 1
Releases(21.9)
  • 21.9(Sep 1, 2021)

  • 21.8(Aug 24, 2021)

  • 21.7(Aug 24, 2021)

  • 21.6(Aug 24, 2021)

  • 21.5(Aug 24, 2021)

  • 20.4.2(Apr 21, 2020)

    Release Date: April 15th, 2020

    • Fix comparison conditions (equals, not_equals, gt, gte, lt, lte on aiodynamo.expressions.F and aiodynamo.expressions.Size via aiodynamo.expressions.F.size() to support referencing other fields (using aiodynamo.expressions.F)
    • Fix timeout handling in aiohttp based client.
    Source code(tar.gz)
    Source code(zip)
  • 20.4.1(Apr 21, 2020)

    Release Date: April 13th, 2020

    • Fixed put_item and delete_item with a condition which does not carry any values.
    • Wrap underlying HTTP client errors, such as connection issues, so networking issues during requests are retried.
    Source code(tar.gz)
    Source code(zip)
  • 20.4(Apr 21, 2020)

    Release Date: April 3rd, 2020

    • Fixed scan with a projection but no filter_expression.
    • Fixed logs leaking session tokens (request sending) and keys (metadata fetch).
    Source code(tar.gz)
    Source code(zip)
  • 20.3(Apr 21, 2020)

    Release Date: March 31st, 2020

    • Added TTL support
    • Added support for pluggable HTTP clients. Built in support for httpx and aiohttp.
    • Added custom client implementation.
    • Added custom credentials loaders, with support for custom credential loaders.
    • Fixed a typo in delete_item
    • Improved item deserialization performance
    • Improved overall client performance, especially for query, scan and count, which are now up to twice as fast.
    • Changed condition, key condition and filter expression APIs to not rely on boto3.
    • Moved aiodynamo.models.F to aiodynamo.expressions.F.
    • Removed boto3 dependency
    • Removed botocore dependency
    • Removed aiobotocore dependency
    Source code(tar.gz)
    Source code(zip)
  • 19.9(Apr 21, 2020)

  • 19.3(Mar 6, 2019)

  • 19.1(Jan 22, 2019)

Python Wrapper For sqlite3 and aiosqlite

Python Wrapper For sqlite3 and aiosqlite

6 May 30, 2022
Python DBAPI simplified

Facata A Python library that provides a simplified alternative to DBAPI 2. It provides a facade in front of DBAPI 2 drivers. Table of Contents Install

Tony Locke 44 Nov 17, 2021
Database connection pooler for Python

Nimue Strange women lying in ponds distributing swords is no basis for a system of government! --Dennis, Peasant Nimue is a database connection pool f

1 Nov 09, 2021
DataStax Python Driver for Apache Cassandra

DataStax Driver for Apache Cassandra A modern, feature-rich and highly-tunable Python client library for Apache Cassandra (2.1+) and DataStax Enterpri

DataStax 1.3k Dec 25, 2022
a small, expressive orm -- supports postgresql, mysql and sqlite

peewee Peewee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use. a small, expressive ORM p

Charles Leifer 9.7k Dec 30, 2022
A Python DB-API and SQLAlchemy dialect to Google Spreasheets

Note: shillelagh is a drop-in replacement for gsheets-db-api, with many additional features. You should use it instead. If you're using SQLAlchemy all

Beto Dealmeida 185 Jan 01, 2023
A Telegram Bot to manage Redis Database.

A Telegram Bot to manage Redis database. Direct deploy on heroku Manual Deployment python3, git is required Clone repo git clone https://github.com/bu

Amit Sharma 4 Oct 21, 2022
Google Cloud Client Library for Python

Google Cloud Python Client Python idiomatic clients for Google Cloud Platform services. Stability levels The development status classifier on PyPI ind

Google APIs 4.1k Jan 01, 2023
MySQL database connector for Python (with Python 3 support)

mysqlclient This project is a fork of MySQLdb1. This project adds Python 3 support and fixed many bugs. PyPI: https://pypi.org/project/mysqlclient/ Gi

PyMySQL 2.2k Dec 25, 2022
ClickHouse Python Driver with native interface support

ClickHouse Python Driver ClickHouse Python Driver with native (TCP) interface support. Asynchronous wrapper is available here: https://github.com/myma

Marilyn System 957 Dec 30, 2022
Pystackql - Python wrapper for StackQL

pystackql - Python Library for StackQL Python wrapper for StackQL Usage from pys

StackQL Studios 6 Jul 01, 2022
Confluent's Kafka Python Client

Confluent's Python Client for Apache KafkaTM confluent-kafka-python provides a high-level Producer, Consumer and AdminClient compatible with all Apach

Confluent Inc. 3.1k Jan 05, 2023
Asynchronous, fast, pythonic DynamoDB Client

AsyncIO DynamoDB Asynchronous pythonic DynamoDB client; 2x faster than aiobotocore/boto3/botocore. Quick start With httpx Install this library pip ins

HENNGE 48 Dec 18, 2022
Query multiple mongoDB database collections easily

leakscoop Perform queries across multiple MongoDB databases and collections, where the field names and the field content structure in each database ma

bagel 5 Jun 24, 2021
A SQL linter and auto-formatter for Humans

The SQL Linter for Humans SQLFluff is a dialect-flexible and configurable SQL linter. Designed with ELT applications in mind, SQLFluff also works with

SQLFluff 5.5k Jan 08, 2023
A pandas-like deferred expression system, with first-class SQL support

Ibis: Python data analysis framework for Hadoop and SQL engines Service Status Documentation Conda packages PyPI Azure Coverage Ibis is a toolbox to b

Ibis Project 2.3k Jan 06, 2023
Py2neo is a comprehensive toolkit for working with Neo4j from within Python applications or from the command line.

Py2neo Py2neo is a client library and toolkit for working with Neo4j from within Python applications and from the command line. The library supports b

Nigel Small 1.2k Jan 02, 2023
Making it easy to query APIs via SQL

Shillelagh Shillelagh (ʃɪˈleɪlɪ) is an implementation of the Python DB API 2.0 based on SQLite (using the APSW library): from shillelagh.backends.apsw

Beto Dealmeida 207 Dec 30, 2022
Little wrapper around asyncpg for specific experience.

Little wrapper around asyncpg for specific experience.

Nikita Sivakov 3 Nov 15, 2021
Python PostgreSQL database performance insights. Locks, index usage, buffer cache hit ratios, vacuum stats and more.

Python PG Extras Python port of Heroku PG Extras with several additions and improvements. The goal of this project is to provide powerful insights int

Paweł Urbanek 35 Nov 01, 2022