Proposed changes

• Adds 2 variations on a proposal for graviton logo inspired by this diagram. Main readme contains colored gradient background (teal and green from algorand's website), and tutorial page is reversed black background and gradient on text itself.
• I noticed that markdownlint is used so i had to add one extra disable for a rule that would prevent having header image as a first content in md file. If you'd like to follow the convention though - feel free to to suggest moving text above the icon.
• Images are hosted on ipfs and here is a .psd version also on ipfs -> ipfs://QmVcS8ULiEUrzs3m6bFQncuRt4MUV8ydnJmyen2wWCt5uq, should be possible to download it and edit in photoshop further if needed.
• Lastly adds 2 tiny repo badges (visitors count and link to algorand website), could be used as a starter template in case you wanna add more badges (like coverage, ci status and etc) in future.

Enforces mypy throughout the repo making a best-effort attempt to remediate mypy errors.

The PR adds # type: ignore in several places as a workaround. In these cases, I was unable to remediate the mypy error in a way that preserved tests. I welcome feedback or alternative suggestions.

For certain use cases, it's necessary to set the balance of the app account to some nonzero value. One example is testing inner app calls. This PR allows the users to set balances for accounts and updates the existing dryrun cost field (which is now deprecated) and adds support for the new cost fields BudgetAdded and BudgetConsumed.

Testing: Unit tests.

First Real Unit Test

@algoidurovic pointed out the following confusing error that he encountered while using graviton for a test (cv. this pyteal commit):

require exactly 1 dry run transaction to create a singleton but had 0 instead

This PR will first check the existence of "error" in the dry-run response, and pass that through when encountered.

Additionally: Discarding the former sanity_test.py which was just a stub in favor or inspector_test.py

Illustrates running integration tests with Python 3.10 via asdf.

After working toward the PR, I discovered a better approach. In case the approach is a useful reference later on, I'm opening and closing the PR for illustrative purposes.

Problem

As you can see in the full printout below, when we call invariant.validates() on an invariant that was defined by a 2-variable predicate, we get an inscrutable expected description.

This invariant was defined via:

predicate = (lambda args, actual: "PASSE" == actual if args[0] else True)
name = 'DryRunProperty.status' 
invariant = Invariant(predicate, name=name)

and spat out the following msg:

<<<<<<<<<<<Invariant for 'DryRunProperty.status' failed for for args (2,): actual is [PASS] BUT expected [<function test_exercises.<locals>.<lambda> at 0x1082557e0>]>>>>>>>>>>>

Full error:

E           AssertionError: ===============
E               <<<<<<<<<<<Invariant for 'DryRunProperty.status' failed for for args (2,): actual is [PASS] BUT expected [<function test_exercises.<locals>.<lambda> at 0x1082557e0>]>>>>>>>>>>>
E               ===============
E               App Trace:
E                  step |   PC# |   L# | Teal              | Scratch   | Stack
E           --------+-------+------+-------------------+-----------+----------------------
E                 1 |     1 |    1 | #pragma version 6 |           | []
E                 2 |     2 |    2 | arg_0             |           | [0x0000000000000002]
E                 3 |     3 |    3 | btoi              |           | [2]
E                 4 |     7 |    6 | label1:           |           | [2]
E                 5 |     9 |    7 | store 0           | 0->2      | []
E                 6 |    11 |    8 | load 0            |           | [2]
E                 7 |    13 |    9 | pushint 2         |           | [2, 2]
E                 8 |    14 |   10 | exp               |           | [4]
E                 9 |     6 |    4 | callsub label1    |           | [4]
E                10 |    15 |   11 | retsub            |           | [4]
E               ===============
E               MODE: ExecutionMode.Signature
E               TOTAL COST: None
E               ===============
E               FINAL MESSAGE: PASS
E               ===============
E               Messages: ['PASS']
E               Logs: []
E               ===============
E               -----BlackBoxResult(steps_executed=10)-----
E               TOTAL STEPS: 10
E               FINAL STACK: [4]
E               FINAL STACK TOP: 4
E               MAX STACK HEIGHT: 2
E               FINAL SCRATCH: {0: 2}
E               SLOTS USED: [0]
E               FINAL AS ROW: {'steps': 10, ' top_of_stack': 4, 'max_stack_height': 2, '[email protected]': 2}
E               ===============
E               Global Delta:
E               []
E               ===============
E               Local Delta:
E               []
E               ===============
E               TXN AS ROW: {' Run': 1, ' cost': None, ' last_log': '`None', ' final_message': 'PASS', ' Status': 'PASS', 'steps': 10, ' top_of_stack': 4, 'max_stack_height': 2, '[email protected]': 2, 'Arg_00': 2}
E               ===============
E               <<<<<<<<<<<Invariant for 'DryRunProperty.status' failed for for args (2,): actual is [PASS] BUT expected [<function test_exercises.<locals>.<lambda> at 0x1082557e0>]>>>>>>>>>>>
E               ===============

Since the main consumer of this repo is pyteal and pyteal requires mypy and black, this repo should follow similar conventions and processes.

• [x] add black
• [x] add flake8
• [x] add mypy

TEAL Blackbox in Graviton

This PR takes over from the now defunct PySDK PR #298

TLDR; Dry Run a Sequence of Inputs on Apps/LogicSigs, View Stats, and make Assertions on Behavior

Please refer to the TEAL Blackbox HowTo / README for full details

A Place to Test Our Python Repo Best Practices

Since this is a brand new repo this is a good place to set some new conventions that our other Python repos should eventually follow also. This PR does not claim to have achieved such a state (especially in light of issues #4 and #7).

Some conventions are implicitly advanced in this PR:

• no requirements.txt:
  • instead, add an extras_require (e.g. extras_require={"test": "pytest==7.1.1"},) in setup.py
  • install the test-dependencies with pip install -e.[test]
• pin internal algorand dependencies using version tags
• (For projects that use github actions) Use act to simulate github actions locally
• Use Makefile to summarize basic setup/testing commands, and incorporate these commands in CI configs
• ~tests should contain all unit and integration tests (as opposed to pairing *_test.py files next to the file they are testing)~ No longer considered a "best practice"
• test files should not be installed by default (previous bullet point makes this goal easier)
• linter configurations should go into setup.cfg whenever possible
• lint with black --check and flake8

Testing

make integration-test

Simulating Github Actions Locally (on a Mac)

make mac-gh-simulate

TODO

CI / Best Practices

• [ ] #2
• [x] #4
• [x] #6
• [ ] #7

Features / Improvements

• [ ] #3
• [x] #5

Migrate Sandbox Code from PySDK and Run All Tests

Testing is cribbing off of PyTeal's "Run blackbox tests in CI" PR

• First successful docker build

  • TIME: 35 minutes
  • Compare with 35 minutes for PyTeal's first uncached build

• First cached docker build

  • TIME: 4 minutes
  • Compare with 5 minutes for a cached build on PyTeal

Cost is Deprecated in go-algorand's Dry Run

As of go-algorand PR #3957 the dry run response transaction top-level-field cost is being deprecated. There is one usage of this field in non-deprecated graviton code.

The good news is that new better fields BudgetConsumed and BudgetAdded are being introduced in the same PR and that the cost can be calculated as

net cost = BudgetConsumed - BudgetAdded

Action items

• [x] go-algorand # 3957 is released
• [x] have a PR that replaces cost with the net cost formula above and stops referencing the original cost field
• [x] have a PR that allows accessing BudgetConsumed and BudgetAdded in the same way that cost is
• [ ] PR's above have been merged and released

Summary of Changes

• .github/workflows/build.yml + Makefile:
  • allow testing and running jupyter notebooks
• graviton/blackbox.py:
  • pass mypy
  • allow specifying transaction params into dry-runs
• graviton/dryrun.py:
  • pass mypy
• graviton/deprecated_dry_run.py: new module that contains functionality mostly used by the now deprecated DryRunTestCaseMixin
• graviton/deprecated_dryrun_mixin.py: where the now deprecated DryRunTestCaseMixin is defined
• graviton/dryrun.py:
  • refactor to keep only the most useful portions of the original helper
  • make mypy pass
  • allow specifying transaction parameters
  • pprint() now returns a string, in addition to printing
• graviton/invariant.py: make mypy pass
• graviton/models.py: new module that contains lightweight models such as the App and LSig dataclasses
• mypy.ini: needed by mypy
• notebooks: new directory for Jupyter notebooks. It contains:
  • IMAGES.md: a static file that shows examples of 3D graphs of dry-runs
  • notebooks/notebooks_test.ipynb: explains how to test notebooks with pytest
  • notebooks/quadratic_factoring_game.ipynb: Blackbox demo
• setup.py: new Jupyter notebooks requirements, and declaration for receiving projects that this package is typed
• tests/integration/algod_test.py: separate out the test that algod is alive and well into a seperate integration test, for easier debugging
• tests/integration/algod_test.py: small refactoring, mostly for mypy
• tests/integration/dryrun_mixin_docs_test.py: refactoring mostly due to deprecations
• tests/integration/lsig_test.py: New logic sig only test. THIS NEEDS A CLEANUP.

adding pytest-xdist for faster integration testing.

Comparison on github:

• Before (58.19 secs for python 3.10)
• After (37.64 secs for python 3.10)

Local test comparison:

Runniing with a single thread:

❯ make integration-test NUM_PROCS=1

 ... 

================================================================== slowest 10 durations ===================================================================
16.47s call     tests/integration/blackbox_test.py::test_app_with_report[app_slow_fibonacci]
8.85s call     tests/integration/blackbox_test.py::test_logicsig_with_report[lsig_slow_fibonacci]
5.03s call     tests/integration/blackbox_test.py::test_logicsig_with_report[lsig_string_mult]
4.83s call     tests/integration/blackbox_test.py::test_app_with_report[app_string_mult]
4.70s call     tests/integration/lsig_test.py::test_factorizer_game_version_report[V4]
4.25s call     tests/integration/lsig_test.py::test_factorizer_game_version_report[V3]
3.90s call     tests/integration/lsig_test.py::test_factorizer_report_with_pymnt
3.56s call     tests/integration/lsig_test.py::test_factorizer_game_version_report[V2]
2.43s call     tests/integration/lsig_test.py::test_factorizer_game_version_report[V1]
1.49s call     tests/integration/dryrun_mixin_docs_test.py::ExampleTestCase::test_factorial
========= 1737 passed in 82.38s (0:01:22) ========= 

With

❯ make integration-test

...

================================================================== slowest 10 durations ===================================================================
38.58s call     tests/integration/blackbox_test.py::test_app_with_report[app_slow_fibonacci]
25.53s call     tests/integration/blackbox_test.py::test_logicsig_with_report[lsig_slow_fibonacci]
19.45s call     tests/integration/lsig_test.py::test_factorizer_game_version_report[V4]
18.16s call     tests/integration/lsig_test.py::test_factorizer_game_version_report[V3]
18.12s call     tests/integration/blackbox_test.py::test_app_with_report[app_string_mult]
16.77s call     tests/integration/blackbox_test.py::test_logicsig_with_report[lsig_string_mult]
6.77s call     tests/integration/dryrun_mixin_docs_test.py::ExampleTestCase::test_factorial
5.49s call     tests/integration/lsig_test.py::test_factorizer_report_with_pymnt
4.59s call     tests/integration/lsig_test.py::test_factorizer_game_version_report[V1]
3.27s call     tests/integration/lsig_test.py::test_factorizer_game_version_report[V2]
========= 1737 passed in 44.31s =========

The goal clerk dryrun command (in opposition to the goal clerk dryrun-remote) is in theory capable of running dry runs without an actual node running. If we use this functionality, we could potentially save on most of the startup times of the C.I. in this repo (and pyteal as well). In theory, this would work as follows:

1. add offline option to executions: e.g. DryRunExecutor.execute_one_dryrun(offline=True, ...)
2. this delegates to an os-level process goal clerk dryrun ...
3. the response should come back as a JSON
4. the response's transactions are then consumed as usual by blackbox.DryRunInspctor
5. inspections and assertions proceed as in the current approach

TODO: break up the PR so the following is no longer a problem

The diff in graviton_blackbox.py is difficult to read through. Instead of looking at the diff, I encourage looking at the actual code. Additionally, this is a testing library, and therefore (IMO) we can gain most of the confidence by looking at the resulting tests both here and in the companion pyteal PR.

Former PR

#44

Issues Addressed

• #38
• #40

Summary of changes

• graviton/blackbox.py - this one's hard to grok because I reordered the classes. In addition:
  • Introducing class DryRunTransactionParams for better grouping of some variables
  • Trimming stale comments
  • class DryRunExecutor:
    • Removed all the static dryrun_* methods and consolidated functionality as follows:
    • It is now expected that the user will instantiate a DryRunExecutor object and call run() on it. Convenience methods run_one() and run_sequence() are also provided, and these have better type guarantees.
• *_test.py and quadratic_factoring_game.ipynb - Migrating all the tests to use run_one() and run_sequence() methods (except for the usage of AbiContractExecutor.dry_run_on_sequence() which remains unaffected)

TODO's

• [ ] Address @michaeldiamant comments in #42 concerning an untested pathway and a suggested patch.

Problems Summary

Too many executing methods

See below for a list of DryRunExecutor methods that allow executing dryruns. It is cumbersome and confusing to have so many methods that basically do the same thing. Ideally, it would be nice to have a single method with the following API:

class DryRunExecutor:
    @singledispatch
    @classmethod
    def execute(
            cls: self,
            algod: AlgodClient,
            program: str,
            input: Any,
            *, 
            mode: ExecutionMode = ExecutionMode.Application,
            abi_method_signature: Optional[str] = None,
            ... several more params such as sender, sp, is_app_create, accounts ...
        ):
            raise NotImplementedError("Implement execute method")

This makes use of functools singledispatch decorator

Inconsistent abi-type handling

Currently, graviton's dry run execution are inconsistent in how abi-information is handled. Some allow abi_argument_types and abi_return_types to be provided while some also allow abi_method_signature to be provided. The last parameter includes all the information that abi_argument_types and abi_return_types provides and therefore it is confusing to provide all the parameters.

After an investigation into the usage of dry run execution in PyTeal, it became apparent that in all relevant situations, an ABIReturnSubroutine object is available for inspection, and it comes with a method method_signature() which could be used to populate an abi_method_signature parameter for dry run execution.

Action Items

Streamline the dry run execution API by refactoring the dry run executing methods as follows:

• [x] remove parameter abi_argument_types
• [x] remove parameter abi_return_type
• [x] add parameter abi_method_signature
• [x] ~make all the current dry run executing methods private~ these have been deleted
• [x] ~unify into a single execute() function using @singledispatch and @process.register to dispatch via input's type (Further investigation is required here. It may not be possible given the nesting of possible inputs (list[tuple] ... etc .... We may also prefer a multipledispatch approach taking into account the mode parameter's type. It may also be infeasible to dispatch directly and therefore we might need to match directly and delegate.)~ This didn't totally work out. However, we've unified everything into a run() method, with 2 companion convenience methods run_one() and run_sequence()
• [x] prepare a PyTeal PR which adapts to this new API (https://github.com/algorand/pyteal/pull/628)
• [ ] refactor supress_abi (not done)

List of dry run executing methods for refactoring

• [x] DryRunExecutor.execute_one_dryrun()
• [x] DryRunExecutor.dryrun_logicsig()
• [x] DryRunExecutor.dryrun_app()
• [x] DryRunExecutor.dryrun_logicsig_on_sequence()
• [x] DryRunExecutor.dryrun_multiapps_on_sequence()
• [x] DryRunExecutor.dryrun_app_pair_on_sequence()
• [x] ABIContractExecutor.dryrun_app_on_sequence()

Summary

Since #33 we have better visibility into program budget via budget_added and budget_consumed. However, these weren't added to the report() function. E.g., part of a recent report looked like had a TXN AS ROW missing this info:

    TXN AS ROW: {' Run': 0, ' cost': -2009, ' last_log': '`0000000000011950', ' final_message': 'PASS', ' Status': 'PASS', 'steps': 89, ' top_of_stack': 72016, 'max_stack_height': 3, '[email protected]': 72016, '[email protected]': 3}

Action Item

• [x] Bring budget_added and budget_consumed into csv_row()

Like https://github.com/algorand/pyteal/issues/610 and https://github.com/algorand/py-algorand-sdk/issues/408, graviton bundles extra artifacts into its source distribution (e.g. tests). The story's request is to limit the source distribution to graviton.

There are a bunch of violations of D.R.Y. in these methods. This deserves a re-evaluation with a bias towards removing the extract* methods if possible.