Hi, thanks for your very interesting work.

Could you share how you evaluate to get the results here. Did you use inverted indexing or use this code? I am trying the later approach, but it is very slow on MSMARCO. Thank you

I want to train a SPLADEv2 from scratch, but the model seems to converge at [email protected]=0.3034,[email protected][email protected]=0.9461. I am using the lambda_d=1e-4, lambda_q=3e-4. Do you have any suggestions? Thank you.

I recently read your SPLADE paper and I think it's quite interesting. I have a question concerning FLOPs calculation in the paper.

I think computing FLOPs for an inverted index involves the length of the activated posting lists(the overlapping terms in query and document). For example, a query a b c and a document c a e, since we must inspect the posting list of the overlapping terms a and c, the flops should at least be

posting_length(a) + posting_length(c)

because we perform summation for each entry in the posting list. However, in the paper you compute FLOPs by the probability that a, b, c are activated in the query and c, a, e are activated in the document. I think this may underestimate the flops of SPLADE because the less sparse the document, the longer posting lists in the inverted index.

Hi,

I'd like to build client code that depends on SPLADE. Please would you consider this PR, which moves all source code into a splade folder, rather than a src/ folder. This appears to work satisfactorily for my use case.

Craig

Hi-- thanks for the nice work.

I'm trying to index+retrieve using the naver/splade-cocondenser-ensembledistil model. Following the readme, I've done:

export SPLADE_CONFIG_FULLPATH="config_default.yaml"
python3 -m src.index \
  init_dict.model_type_or_dir=naver/splade-cocondenser-ensembledistil \ # <--- (from readme, using the new model)
  config.pretrained_no_yamlconfig=true \
  config.index_dir=experiments/pre-trained/index \
  index=msmarco  # <--- added

export SPLADE_CONFIG_FULLPATH="config_default.yaml"
python3 -m src.retrieve \
  init_dict.model_type_or_dir=naver/splade-cocondenser-ensembledistil \ # <--- (from readme, using the new model)
  config.pretrained_no_yamlconfig=true \
  config.index_dir=experiments/pre-trained/index \
  config.out_dir=experiments/pre-trained/out-dl19 \
  index=msmarco \  # <--- added
  retrieve_evaluate=msmarco # <--- added

Everything runs just fine, but I'm getting rather poor results in the end:

[email protected]: 0.18084248646927734
recall ==> {'recall_5': 0.2665353390639923, 'recall_10': 0.3298710601719197, 'recall_15': 0.3694364851957974, 'recall_20': 0.3951050620821394, 'recall_30': 0.4270654250238777, 'recall_100': 0.5166069723018146, 'recall_200': 0.5560768863419291, 'recall_500': 0.606984240687679, 'recall_1000': 0.6402578796561604}

I suspect it's a configuration problem on my end, but since the indexing process takes a bit of time, I thought I'd just ask before diving too far into the weeds: Is there a configuration file to use for the splade++ results, and how do I use it?

Thanks!

Hey, I recently read your SPLADEv2 paper. That's so insightful! But I still have a few questions about it.

1. Is the model trained with dot product similarity function included in the contrastive loss?
2. Evaluation on MS MARCO is performed via inverted index backed by anserine?
3. Evaluation on BEIR is implemented with sentencetransformer hence also via dot product?
4. How much can you gurantee the sparsity of learned representation since it's softly regularized by L1 and FLOPS loss? Did you use a tuned threshold to ''zerofy'' ~0 value?

In your paper, you said equation (1) is equivalent to the MLM prediction and E_j in equation (1) denotes the BERT input embedding for token j. If you use the default implementation of HuggingFace Transformers, E_j is not from the input layer but another embeddings matrix, which is called "decoder" in the "BertLMPredictionHead" (if you use BERT). Did you manually set the "decoder" weights to the input embedding weights?

My other question is concerning equation (4). It computes the summation of the weights of the document/query terms. In the "forward" function of the Splade class (models.py) however, you use "torch.max" function. Can you explain this issue?

I understand that the log-saturation function and regularization loss suppress the weights of the frequent terms. But when do you drop a term (setting the term weight to zero)? Is it when the logit is less or equal to zero, so that the log(1+ReLu(.)) function outputs zero?

I'm curious if you've run your model with a "second-stage" reranker, on the BEIR benchmarks. Would you expect much benefit from this?

Thank you, and excellent work!

Initial pull request to add networks from https://dl.acm.org/doi/10.1145/3477495.3531833

Networks are now available on huggingface as well:

V) https://huggingface.co/naver/efficient-splade-V-large-doc https://huggingface.co/naver/efficient-splade-V-large-query

VI) https://huggingface.co/naver/efficient-splade-VI-BT-large-doc https://huggingface.co/naver/efficient-splade-VI-BT-large-query

Still need to add the links in the naverlabs website for the small and medium networks

Firstly, thanks for your series of amazing papers and well-organized code implementations.

The two papers Wacky Weights in Learned Sparse Representations and the Revenge of Score-at-a-Time Query Evaluation and From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective show that using Pisa can make query retrieval much faster compared to using Anserini or code from the repo for Splade.

The folder efficient_splade_pisa/ in the repo contains the instructions on using Pisa for Splade but the instructions are only for processed queries and indexes. If I only have a well-trained Splade model, how can I process the outputs of the Splade model (sparse vectors or its quantized version for Anserini) to make them suitable for Pisa? Can you provide more specific instructions on this?

Best wishes

Hello!

I find that when I run flops, it always returns Nan.

I see your last commit fixed "force new", and changed line 25 in transformer_evaluator.py to force_new=True, but in inverted_index.py line 23, seems that the self.n will return 0 if force_new is True.

The flops no longer return nan after I remove the "force_new=True".

Am I doing sth wrong here? And how should I get the correct flops..

Thank you! Allen