Python API Client for Close

(closeio)--- Projects/closeio-api ‹master» python scripts/bulk_update_leads_info.py ~/Downloads/test.csv --api-key <redacted> --confirmed
[2015-05-15 18:59:23,018] INFO Starting new HTTPS connection (1): app.close.io
[2015-05-15 18:59:23,539] INFO line 2 updated: lead_iyTYlkzfVWKn0rH0x1yGrPX2vMOzzCsOdBK205WrzLk Plumbing Medic
[2015-05-15 18:59:23,539] INFO summary: updated[1], new[0], skipped[0]
(closeio)--- Projects/closeio-api ‹master» cat ~/Downloads/test.csv
lead_id,contact_name,contact_phone,,,,,,
lead_iyTYlkzfVWKn0rH0x1yGrPX2vMOzzCsOdBK205WrzLk,Conference Call Line,+1818-452-3980,,,,,,

No contact added to https://app.close.io/lead/lead_iyTYlkzfVWKn0rH0x1yGrPX2vMOzzCsOdBK205WrzLk/

Right now the async client makes the code more complex and doesn't really add much value. I doubt it's used in the wild and if it is, then the usage is probably quite confusing and hacky. I think we might as well get rid of it.

Examples of current confusing behavor:

• api.get/post/put/delete is still synchronous
• api.map expects a list of grequests.AsyncRequest, but there's no easy way to construct them.

This PR introduces breaking changes and should be published along with a major version bump.

TODO:

• [x] Add mocked unit tests
• [x] Bump version to v1.0
• [x] Add a GitHub Release and publish on PyPI

Fixes #59

All the changes are supported by TDD with functional tests hitting the live API.

I'm not committing the tests but I can share them somewhere in case you want to have a look.

/cc @thomasst

As an example, today I ran this request:

resp = api.get('report/activity/ORGID', params=
{'user_id':'USERID', 'date_start':'2018-01-02', 'date_end':'2018-01-02'})

The response I got back was:

{
    "revenue_created_annual_created": 0, 
    "opportunities_created": 0, 
    "revenue_lost_annual_created": 0, 
    "sms_sent": 0, 
    "leads_created": 0, 
    "opportunities_lost": 0, 
    "revenue_won_monthly": 0, 
    "revenue_won_annual": 0, 
    "leads_contacted": 3, 
    "revenue_won_one_time": 0, 
    "revenue_lost_annual": 0, 
    "revenue_lost_one_time_created": 0, 
    "revenue_lost_one_time": 0, 
    "revenue_created_monthly": 0, 
    "revenue_won_annual_created": 0, 
    "opportunities_won": 0, 
    "opportunities_created_created": 0, 
    "emails_sent": 0, 
    "revenue_created_monthly_created": 0, 
    "emails_received": 0, 
    "calls_duration_total": 0, 
    "sms_received": 0, 
    "calls_duration_average": 0, 
    "revenue_won_one_time_created": 0, 
    "revenue_created_one_time": 0, 
    "revenue_won_monthly_created": 0, 
    "revenue_lost_monthly_created": 0, 
    "opportunities_won_created": 0, 
    "opportunities_lost_created": 0, 
    "revenue_lost_monthly": 0, 
    "_queries": {
        "leads_contacted": "call(duration > 0  date >= 2018-01-01 date <= 2018-01-01) or email(direction:sent  date >= 2018-01-01 date <= 2018-01-01) or sms(direction:sent  date >= 2018-01-01 date <= 2018-01-01)", 
        "emails_received": "email(direction:received  date >= 2018-01-01 date <= 2018-01-01)", 
        "calls": "call( date >= 2018-01-01 date <= 2018-01-01)", 
        "opportunities_created": "opportunity( created >= 2018-01-01 created <= 2018-01-01)", 
        "opportunities_won_created": "opportunity(status_type:won  closed >= 2018-01-01 closed <= 2018-01-01)", 
        "opportunities_lost_created": "opportunity(status_type:lost  lost >= 2018-01-01 lost <= 2018-01-01)", 
        "sms_sent": "sms(direction:sent  date >= 2018-01-01 date <= 2018-01-01)", 
        "leads_created": " created >= 2018-01-01 created <= 2018-01-01", 
        "opportunities_won": "opportunity(status_type:won  closed >= 2018-01-01 closed <= 2018-01-01)", 
        "opportunities_created_created": "opportunity( created >= 2018-01-01 created <= 2018-01-01)", 
        "emails_sent": "email(direction:sent  date >= 2018-01-01 date <= 2018-01-01)", 
        "sms_received": "smsdate >= 2018-01-01 date <= 2018-01-01)", 
        "opportunities_lost": "opportunity(status_type:lost" lost >= 2018-01-01 lost <= 2018-01-01)"
    }, 
    "revenue_created_one_time_created": 0, 
    "calls": 0, 
    "revenue_created_annual": 0
}

since the bounding dates are 2018-01-01 and 2018-01-01, it means that we aren't correctly factoring in timezone when passing through date params.

We need to add the tz.offset*-1 to the datetime in both directions for it to appear the same as it does in app.

The user_reassign script doesn't update all the objects. It paginates through objects that match certain criteria (i.e. the old user), reassigns them to the new user, and then increases the offset to fetch the next batch. The problem is that when we reassign old objects, the results don't contain them anymore, and we end up skipping objects. We instead need to keep the offset at 0 and loop until no more objects are left (except for in the dry run).

Should take a search query as input and 2 API keys (from different orgs)

Transfer all the leads matching the search query

As well as all the nested objects on the leads (Contacts, Tasks, Opportunities), and the Calls & Note Activities

Emails don't need to transfer since we'll sync them ourselves but all other lead data should transfer over.

If lead statuses or opportunity statuses don't exist it should clearly print out that you need to set up those statuses within the destination organization first.

(closeio-api)--- lib/closeio-api ‹master» ./scripts/user_reassign.py --to-user-id user_wWDo6ETmCC9mSL3YQt0wdierIIKfZnR0LShSiVxEXBF --from-user-id user_fLuLxKSVFjJTePCe660YI2f2b5JpwoBjWQaPqWUjm5V -k <redacted> --all-tasks --all-opportunities -c

Running the above command seems to modify the tasks and opportunities but if you run it again the see the same output / nothing actually changes.

bulk_update_address_countries.py

inputs:

• api key
• old country code
• new country code
• leads search query (defaults to * sort:created if none) -- can be coded in script since it's hard to pass complicated queries (e.g. quotes) to cli args

python cli script that takes these items as inputs. it should loop through each lead in the given search query, and if there are any addresses on that lead with the old country code then it should update the address with the new country code

Would also like a read-only preview mode of what will happen and then a --confirmed flag

@congocongo Can you work on this one after the other script?

Closes #80

This PR:

• Changes the user agent sent on requests from python-requests/X to python closeio vY python-requests/X, where Y is the version of the closeio-api wrapper and X is the python-requests version
• Adds a __version__ variable to __init__.py to keep track of the version.
• Adds a helper to setup.py to read the __version__ variable from __init__.py to keep things consistent when installing the package.
• Updates the version from 1.1 to 1.2

I used the same structure as the Cleancat repo to do this.

In response to https://github.com/closeio/closeio-api/issues/80.

This modifies the default User-Agent to include package details: python closeio v{VERSION} {DEFAULT}

Also breaks out the version constant into its' own version file. This helps to reduce the change of inconsistent versions, while also making it easy to add the seudo-standard __version__ attribute :)

Testing

Setup a simple server script to dump headers when making calls with the client. Then loaded the modified closeio_api and the output of the dump confirms this is working.

----- Request Start ----->

/lead/
Host: localhost:8000
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python closeio v0.5 python-requests/2.11.1
Content-Type: application/json
X-TZ-Offset: -7
Content-Length: 20
Authorization: Basic abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwx

{"name": "New Lead"}
<----- Request End -----

127.0.0.1 - - [13/Mar/2017 17:25:57] "POST /lead/ HTTP/1.1" 200 -

This is an alternative to #78 fixing some of the async client's flaws instead of removing it altogether.

Example usage:

In [1]: from closeio_api import Client

In [2]: api = Client('api_key', development=True, async=True)

In [3]: results = api.map([
   ...:   api.post('lead', {'name': 'New Lead 1'}),
   ...:   api.post('lead', {'name': 'New Lead 2'}),
   ...:   api.post('lead', {'name': 'New Lead 3'}),
   ...:   api.post('lead', {'name': 'New Lead 4'})
   ...: ])

In [4]: len(results)
Out[4]: 4

In [5]: results[0]['id']
Out[5]: u'lead_RpzCIwspNlTUowYUvIUaNE1tp3ncVFk8x3Uw4sv96FO'

I'm still conflicted whether such a deep integration of concurrency is good for this client library and its users or not. On the one hand, it makes concurrent requests easier to construct and send, and it handles API errors and retry logic properly (-ish). On the other hand, it makes the code more complex (and thus harder to read/understand), it forces a particular implementation of concurrency (green threads & gevent), and there are still some quirks with it (e.g. using the "debug" flag differs unintuitively between the sync and async client).

@closeio/engineering what do you think?

This PR introduces breaking changes and should be published along with a major version bump.

We want to update the behavior of our api and client to follow the draft RFC related to communicating rate limiting.

See https://github.com/closeio/closeio/issues/28735

So that it's obvious at a glance what kind of an error we experienced. Right now it's possible to get an enigmatic traceback:

In [3]: api.get('me')
Traceback (most recent call last)
<ipython-input-3-842652bd220e> in <module>()
----> 1 api.get('me')

/Users/wojcikstefan/Repos/closeio-api/closeio_api/__init__.py in get(self, endpoint, params, **kwargs)
    132         """
    133         kwargs.update({'params': params})
--> 134         return self._dispatch('get', endpoint+'/', **kwargs)
    135
    136     def post(self, endpoint, data, **kwargs):

/Users/wojcikstefan/Repos/closeio-api/closeio_api/__init__.py in _dispatch(self, method_name, endpoint, api_key, data, debug, **kwargs)
    108             raise ValidationError(response)
    109         else:
--> 110             raise APIError(response)
    111
    112     def _get_rate_limit_sleep_time(self, response):

APIError:

When a request fails because of an aborted connection or other unhandled exception the resulting APIError and the stack trace generated with it is not very informative. We should make the string representation of APIError include more detail.