A Python wrapper around the libmemcached interface from TangentOrg.

I was trying to enable zlib compression with memcached check-and-set and it appears that this is functionality that hasn't yet been built out: https://github.com/lericson/pylibmc/blob/78138d33c4156111294269a2a8f0cfcc66ac5c5c/src/_pylibmcmodule.c#L1041-L1043

I'm not an expert in C but would be happy to try to create a PR if it would be welcomed and useful!

in the documentation there is no mention of the behavior "tcp_keepalive". this behavior is important and necessary when running in a kubernetes or docker-compose scenario because the connection goes through something that monitors for dead connections and the thing that monitors for dead connections has a timeout of like 15 minutes or something.

so i set the memcached behavior "tcp_keepalive" to 300 to ensure that my connecitons do not get cut by the networking thing.

for more better description of the "networking thing" see https://github.com/zulip/zulip/pull/12711 which describes some about pika and also mentions the pylibmc.

my request: please add to the documentation some information regarding the "tcp_keepalive".

also i noticed there was missing a mention of "retry_timeout". please add that to the documentation too. (if you do not set this value to something like 5... does the client just error out on the first error?)

for the tcp keepalive to work you also need to set the "tcp_keepidle" value. unfortunately your python code does not expose this feature from the underlying c library. so for now i have a patch that i wrote.

diff --git a/src/_pylibmcmodule.h b/src/_pylibmcmodule.h
index cee9772..36a28bb 100644
--- a/src/_pylibmcmodule.h
+++ b/src/_pylibmcmodule.h
@@ -202,6 +202,7 @@ static PylibMC_Behavior PylibMC_behaviors[] = {
     { MEMCACHED_BEHAVIOR_TCP_NODELAY, "tcp_nodelay" },
 #if LIBMEMCACHED_VERSION_HEX >= 0x00044000
     { MEMCACHED_BEHAVIOR_TCP_KEEPALIVE, "tcp_keepalive" },
+    { MEMCACHED_BEHAVIOR_TCP_KEEPIDLE, "tcp_keepidle" },
 #endif
     { MEMCACHED_BEHAVIOR_HASH, "hash" },
     { MEMCACHED_BEHAVIOR_KETAMA_HASH, "ketama_hash" },

and i do a pip download --no-binary=pylibmc pylibmc. extract, cd, patch, build, install.

tcp_keepalive is boolean value. see https://man7.org/linux/man-pages/man7/socket.7.html section SO_KEEPALIVE.

tcp_keepidle is integer time in seconds. see https://man7.org/linux/man-pages/man7/tcp.7.html section on TCP_KEEPIDLE.

I believe this commit results in bools written with 1.5.2 to strings in 1.6.1, which causes the value to actually switch if you stored False. This makes upgrading from 1.5.2 to 1.6.1 (and getting the pickle_protocol behavior, thank you for adding that!) difficult. Unfortunately tracking down all the places we might be setting False is also difficult (and tedious). Any ideas about how we can work around this? The only thing I've come up with is forking 1.6 to have the write path set an extra flag for anything with FLAG_TEXT, and then checking for that extra flag on read, monitoring what fraction of our reads come back without that flag, and hoping that eventually it goes to 0.

(pylibmctest) ✔ 12:16 ~ $ python
Python 3.7.3 | packaged by conda-forge | (default, Dec  6 2019, 08:36:57)
[Clang 9.0.0 (tags/RELEASE_900/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pylibmc
>>> cl = pylibmc.Client(['localhost:11211'])
>>> cl.set('foo', False)
True
>>> cl.get('foo')
False
>>>
(pylibmctest) ✔ 12:44 ~ $ pip freeze | grep pylibmc
pylibmc==1.5.2
(pylibmctest) ✔ 12:45 ~ $ pip install pylibmc==1.6.1
Collecting pylibmc==1.6.1
  Using cached https://files.pythonhosted.org/packages/a7/0c/f7a3af34b05c167a69ed1fc330b06b658dac4ab25b8632c52d1022dd5337/pylibmc-1.6.1.tar.gz
Building wheels for collected packages: pylibmc
  Building wheel for pylibmc (setup.py) ... done
  Created wheel for pylibmc: filename=pylibmc-1.6.1-cp37-cp37m-macosx_10_9_x86_64.whl size=32878 sha256=5bde92da27513e7024d29f73dd0a9837ebb4c7024d2b553eac62db835d504fdf
  Stored in directory: /Users/aaronwebber/Library/Caches/pip/wheels/71/5e/41/6796b369874b1b0345bccf31449e162eadcda85b104922f2de
Successfully built pylibmc
Installing collected packages: pylibmc
  Found existing installation: pylibmc 1.5.2
    Uninstalling pylibmc-1.5.2:
      Successfully uninstalled pylibmc-1.5.2
Successfully installed pylibmc-1.6.1
(pylibmctest) ✔ 12:45 ~ $ python
Python 3.7.3 | packaged by conda-forge | (default, Dec  6 2019, 08:36:57)
[Clang 9.0.0 (tags/RELEASE_900/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pylibmc
>>> cl = pylibmc.Client(['localhost:11211'])
>>> cl.get('foo')
'0'
>>> bool(cl.get('foo'))    # This is the real problem  :(
True

I realize that I am very late with the python3 migration, so I understand if this ticket is closed right away. However others running into the same issue can hopefully benefit from this report.

According to the docs for pickle inPython 3:

Using encoding='latin1' is required for unpickling NumPy arrays and instances of datetime, date and time pickled by Python 2.

However the default encoding for unpickling is US-ASCII, thus one cannot deserialize datetime, date and time objects as well as NumPy arrays if they have been serialized in Python 2. Many migrate from Python 2 to Python 3 using a path were both versions run in parallel for some time.

Allowing to configure the encoding used for pickle.load()/_PylibMC_pickle_loads would resolve the issue. I also added a workaround below.

Environment

pylibmc==1.6.1 memcached 1.6.6 python 2.7.18 and 3.6.11

Steps to reproduce

In the Python 2 environment:

import sys
import pylibmc
from pickle import load as pickle_load
from io import BytesIO
from datetime import datetime
            

client = pylibmc.Client(["localhost:11211"], behaviors={"pickle_protocol": 2}, binary=True)

client.set("testText", "asdf")
client.set("testDict", {"a": 123})
client.set("testDatetime", datetime.now())

In the Python 3 environment:

import sys
import pylibmc
from pickle import load as pickle_load
from io import BytesIO
from datetime import datetime
            

client = pylibmc.Client(["localhost:11211"], behaviors={"pickle_protocol": 2}, binary=True)

# this works
client.get("testText")
client.get("testDict")

# this fails with: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)
client.get("testDatetime")

Expected behaviour

The datetime object (as well as the others) is correctly retrieved from the cache

Actual Behaviour

The datetime object cannot be deserialized

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)

Workaround

import sys
import pylibmc
from pickle import load as pickle_load
from io import BytesIO

class MemcachedClient(pylibmc.Client):

     if sys.version_info.major >= 3:        
        PYLIBMC_FLAG_PICKLE = 1
     
        def deserialize(self, value, flags):
            if flags & self.PYLIBMC_FLAG_PICKLE:
                return pickle_load(BytesIO(value), encoding=u"latin1")
            return super(MemcachedClient, self).deserialize(value, flags)
            

client = MemcachedClient(["localhost:11211"], behaviors={"pickle_protocol": 2}, binary=True)
client.get("testText")
client.get("testDict")
client.get("testDatetime")

An idea left as a comment in the code is to provide an iterable interface for multi getting, for example

mc = …

for key, val in mc.iter_multi_get(set_of_keys):
    use(key, val)

However, it is not clear if this would be a good idea as it may incur a very large overhead, switching between Python and C.