Latent Semantic Analysis
Pipeline for training LSA models using Scikit-Learn.
Usage
Instead of writing custom code for latent semantic analysis, you just need:
- install pipeline:
pip install latent-semantic-analysis
- run pipeline:
- either in terminal:
lsa-train --path_to_config config.yaml
- or in python:
import latent_semantic_analysis
latent_semantic_analysis.train(path_to_config="config.yaml")
NOTE: more about config file here.
No data preparation is needed, only a csv file with raw text column (with arbitrary name).
Config
The user interface consists of only one files:
- config.yaml - general configuration with sklearn TF-IDF and SVD parameters
Change config.yaml to create the desired configuration and train LSA model with the following command:
- terminal:
lsa-train --path_to_config config.yaml
- python:
import latent_semantic_analysis
latent_semantic_analysis.train(path_to_config="config.yaml")
Default config.yaml:
seed: 42
path_to_save_folder: models
# data
data:
data_path: data/data.csv
sep: ','
text_column: text
# tf-idf
tf-idf:
lowercase: true
ngram_range: (1, 1)
max_df: 1.0
min_df: 1
# svd
svd:
n_components: 10
algorithm: arpack
NOTE: tf-idf and svd are sklearn TfidfVectorizer and TruncatedSVD parameters correspondingly, so you can parameterize instances of these classes however you want.
Output
After training the model, the pipeline will return the following files:
model.joblib- sklearn pipeline with LSA (TF-IDF and SVD steps)config.yaml- config that was used to train the modellogging.txt- logging filedoc2topic.json- document embeddingsterm2topic.json- term embeddings
Requirements
Python >= 3.6
Citation
If you use latent-semantic-analysis in a scientific publication, we would appreciate references to the following BibTex entry:
@misc{dayyass2021lsa,
author = {El-Ayyass, Dani},
title = {Pipeline for training LSA models},
howpublished = {\url{https://github.com/dayyass/latent-semantic-analysis}},
year = {2021}
}
9 Jul 17, 2021
42 Dec 20, 2022
190 Dec 21, 2022
7 Sep 20, 2022
57 Dec 7, 2022
74 Oct 7, 2022
6 Jun 27, 2022
654 Jan 2, 2023
14 Dec 9, 2022
70 Dec 26, 2022
28 Sep 13, 2022
2 Dec 21, 2022
6.5k Jan 08, 2023
105 Dec 18, 2022
138 Dec 30, 2022
77 Dec 07, 2022
2.3k Jan 05, 2023
135 Dec 29, 2022
40 Sep 19, 2022
195 Jan 01, 2023
29 Dec 01, 2022
9 Jan 11, 2022
6 Nov 20, 2022
55 Nov 17, 2022
5 Dec 20, 2021
137 Oct 26, 2022
88 Dec 31, 2022
395 Nov 21, 2022
2 Feb 17, 2022