First of all this project looks great! I've taken an initial stab at #12 and also tried to add support querying data that has already been transformed. If you have data that you've already transformed (e.g. a UMAP embedding), you probably don't want to rerun encoder.transform again. In this case you want to index the transformed data and query it directly.

This is just a first crack so happy to incorporate any feedback you might have!

This is conceptual work in progress. The maintainer is actively researching this, please do not work on it.

Problem Statement

When you submit where is my phoone and you get similarities you may get things like:

• where is my phone
• where is my credit card

Depending on your task, either the "where is" part of the sentence is more important or the "phone" part is more important. The encoder, however, may be very brittle when it comes to spelling errors. So to put it more generally;

The similarity in an embedded space in our case is very much "general". I'm using "general" here, as opposed to "specific" to indicate that these similarities have been constructed without having a task in mind.

Similar Issue

Suppose that we are deduplicating and we have a zipcode, city, first-, and last-name. How would our encoding be able to understand that having the same city is not a strong signal while having the first name certainly is? Can we really expect a standard encoding to understand this? Without labels ... I think not.

As mentioned in https://github.com/koaning/simsity/pull/13:

I think the refit parameter should go in the Service() call. I think there should also be a parameter somewhere to avoid calling .transform() if the data has already been transformed. Do you think it is worth adding an additional parameter to Service() and keeping the indexed_from_transformed_data method?

It's a fair remark. I think preventing a transfrom() is fair, but the solution would be to have an Identity() transformer that just keeps the data as-is. This would also make a great default value for the encoder.

Made this issue to track progress and to discuss the approach.

Hi Vincent. Since I discovered you my barrier towards Python has eroded! Thank you. I'm a Data Scientist who wants to check if simsity can help with retrieving similar regions based on environmental variables.

Hi! Are there any plans to add support for updating the indexer, i.e. add new documents without retraining the entire pipeline? Would be a very useful feature .

from simsity.service import Service

service = Service(
    indexer=indexer,
    encoder=encoder
)

service.train_from_dataf(df, features=["text"])

....

service.update(new_docs, features=["text"])  # <- this

I think the original design was flawed and this project should stick to the scikit-learn API more.

from simsity.preprocessing import Grab
from simsity.service import Service
from simsity.indexer import (AnnoyIndexer, PynnDescentIndexed, NMSlibIndexer,
                             PineconeIndexer, QdrantIndexer, WeviateIndexer)


encoder = make_pipeline(
    make_union(
        make_pipeline(Grab("text"), SentenceEncoder()),
        make_pipeline(Grab("title"), SentenceEncoder())
    )
)

service = Service(encoder, indexer, batch_size=50)
service.index(X)
items, dists = service.query(X, n=10)

• [x] add typing + type checker
• [x] add tests for the minhash tools
• [ ] collect more useful datasets
• [x] automate the benchmarking
• [x] write getting started guides
• [ ] record a quick demo for colleagues
• [ ] add github actions stash

Adding the MinHash components. This is also an amazing opportunity to:

• [ ] add types and a type checker
• [ ] add some standard tests for indexers
• [ ] add a script to run some benchmarks on the clinc dataset