Have not been able to get either the long form (from wiki) or short form (from github readme) files to work successfully.

The file @ https://github.com/DerwenAI/pytextrank/blob/master/example.py throws a KeyError: 1255 when run. Output for this is below.

I have been able to get the example from the github page working but only for very small strings. Anything larger than a few words throws a KeyError with varying number depending on the length of the string.

Can't figure out the issue even using all input (txt files) from the example on the wiki page and changing the spacy version to various releases from 2.0.0 to present.

KeyError Traceback (most recent call last) in () 31 text = f.read() 32 ---> 33 doc = nlp(text) 34 35 print("pipeline", nlp.pipe_names)

/home/pete/.local/lib/python3.5/site-packages/spacy/language.py in call(self, text, disable, component_cfg) 433 if not hasattr(proc, "call"): 434 raise ValueError(Errors.E003.format(component=type(proc), name=name)) --> 435 doc = proc(doc, **component_cfg.get(name, {})) 436 if doc is None: 437 raise ValueError(Errors.E005.format(name=name))

/usr/local/lib/python3.5/dist-packages/pytextrank/pytextrank.py in PipelineComponent(self, doc) 530 """ 531 self.doc = doc --> 532 Doc.set_extension("phrases", force=True, default=self.calc_textrank()) 533 Doc.set_extension("textrank", force=True, default=self) 534

/usr/local/lib/python3.5/dist-packages/pytextrank/pytextrank.py in calc_textrank(self) 389 390 for chunk in self.doc.noun_chunks: --> 391 self.collect_phrases(chunk) 392 393 for ent in self.doc.ents:

/usr/local/lib/python3.5/dist-packages/pytextrank/pytextrank.py in collect_phrases(self, chunk) 345 if key in self.seen_lemma: 346 node_id = list(self.seen_lemma.keys()).index(key) --> 347 rank = self.ranks[node_id] 348 phrase.sq_sum_rank += rank 349 compound_key.add(key)

KeyError: 1255

I think the current implementation returns keyphrases that are potential subsets of each other, that this is due to the use of noun_chunks and ents, and that this is not the desired output. Specifically, if a document has an entity that is a superset (as far as span start and end is concerned) of a noun chunk (or vice-versa), and both contain a key token, then both will be returned as keyphrases.

While also/possibly linked to the issue of entity linkage (which I'd love to know more about!), this can simply be a matter of defining "entity" boundaries and a "duplication" issue, as the example below with "Seouls Four Seasons hotel" and "Four Seasons", where I believe one keyphrase is enough and having both is confusing, demonstrates.

Am I missing something? Is this the desired logic?

Example:

from spacy.util import filter_spans
import pytextrank
import en_core_web_sm

nlp = en_core_web_sm.load()
nlp.add_pipe("textrank", last=True);

# from dat/lee.txt
text = """
After more than four hours of tight play and a rapid-fire endgame, Google's artificially intelligent Go-playing computer system has won a second contest against grandmaster Lee Sedol, taking a two-games-to-none lead in their historic best-of-five match in downtown Seoul.  The surprisingly skillful Google machine, known as AlphaGo, now needs only one more win to claim victory in the match. The Korean-born Lee Sedol will go down in defeat unless he takes each of the match's last three games. Though machines have beaten the best humans at chess, checkers, Othello, Scrabble, Jeopardy!, and so many other games considered tests of human intellect, they have never beaten the very best at Go. Game Three is set for Saturday afternoon inside Seoul's Four Seasons hotel.  The match is a way of judging the suddenly rapid progress of artificial intelligence. One of the machine-learning techniques at the heart of AlphaGo has already reinvented myriad online services inside Google and other big-name Internet companies, helping to identify images, recognize commands spoken into smartphones, improve search engine results, and more. Meanwhile, another AlphaGo technique is now driving experimental robotics at Google and places like the University of California at Berkeley. This week's match can show how far these technologies have come - and perhaps how far they will go.  Created in Asia over 2,500 year ago, Go is exponentially more complex than chess, and at least among humans, it requires an added degree of intuition. Lee Sedol is widely-regarded as the top Go player of the last decade, after winning more international titles than all but one other player. He is currently ranked number five in the world, and according to Demis Hassabis, who leads DeepMind, the Google AI lab that created AlphaGo, his team chose the Korean for this all-important match because they wanted an opponent who would be remembered as one of history's great players.  Although AlphaGo topped Lee Sedol in the match's first game on Wednesday afternoon, the outcome of Game Two was no easier to predict. In his 1996 match with IBM's Deep Blue supercomputer, world chess champion Gary Kasparov lost the first game but then came back to win the second game and, eventually, the match as a whole. It wasn't until the following year that Deep Blue topped Kasparov over the course of a six-game contest. The thing to realize is that, after playing AlphaGo for the first time on Wednesday, Lee Sedol could adjust his style of play - just as Kasparov did back in 1996. But AlphaGo could not. Because this Google creation relies so heavily on machine learning techniques, the DeepMind team needs a good four to six weeks to train a new incarnation of the system. And that means they can't really change things during this eight-day match.  "This is about teaching and learning," Hassabis told us just before Game Two. "One game is not enough data to learn from - for a machine - and training takes an awful lot of time."
"""

doc = nlp(text)

key_spans = []
for phrase in doc._.phrases:
    for chunk in phrase.chunks:
        key_spans.append(chunk)

print(len(key_spans))

full_set = set([p.text for p in doc._.phrases])

print(full_set)

print(len(filter_spans(key_spans)))

sub_set = set([pytextrank.util.default_scrubber(p) for p in filter_spans(key_spans)])

print(sub_set)

print(full_set - sub_set)

print(sub_set - full_set)

Possible solution?:

all_spans = list(self.doc.noun_chunks) + list(self.doc.ents)
filtered_spans = filter_spans(all_spans)
filtered_phrases = self._collect_phrases(filtered_spans, self.ranks) # replacing all_phrases

instead of

nc_phrases: typing.Dict[Span, float] = self._collect_phrases(self.doc.noun_chunks, self.ranks)
ent_phrases: typing.Dict[Span, float] = self._collect_phrases(self.doc.ents, self.ranks)
all_phrases: typing.Dict[Span, float] = { **nc_phrases, **ent_phrases }

see https://github.com/DerwenAI/pytextrank/blob/29339027b905844af0064ed9a0326e2578f21bf6/pytextrank/base.py#L362

Note:

• My understanding is that self._get_min_phrases is doing something else.
• spacy.util.filter_spans simply looks for the (first) longest span, which might not be the best solution.

Hi! I'm working on a project connected with NLP and was happy to find out that there is such a tool as PyTextRank. However, I've encountered an issue at the very beginning trying to just import package to run the example code given here. The error that I get is the following:

----> from pytextrank import json_iter, parse_doc, pretty_print
ImportError: cannot import name 'json_iter'
----> from pytextrank import parse_doc
ImportError: cannot import name 'parse_doc'

I've tried running it in iPython console and a Jupyter Notebook, both the same result. I've installed PyTextRank with pip, the python version that I have is 3.5.4., spacy 2.1.8., networkx 2.4, graphvis 0.13.2

It seems to me that nlp.add_pipe("textrank") must have "noun chunks" which will raise "NotImplementedError" for some language models where "noun chunks" have not been implemented. I've got "NotImplementedError" with "ru_core_news_lg" and "ru_core_news_sm" spacy models.

The proposal is to make the use of "noun chunks" optional to prevent such errors.

Hi there, I've been looking at your code and example for a long time and I still have no idea how to use this.

I have documents in string format, what JSON format should they have if I want to use the stages as in the examples?

I find there's a crucial piece of information missing in the documentation, which is how to use the functionality of this package with a simple document in string format (or list of strings, representing sentences). As I don't know beforehand what JSON format I have to convert my text to in order to use the stage pipeline.

Cheers

Hi there,

Does any know how to fix below errors when running example code?

Thanks.

Traceback (most recent call last): File "test.py", line 14, in nlp.add_pipe("textrank") File "/usr/local/lib64/python3.6/site-packages/spacy/language.py", line 773, in add_pipe validate=validate, File "/usr/local/lib64/python3.6/site-packages/spacy/language.py", line 639, in create_pipe raise ValueError(err) ValueError: [E002] Can't find factory for 'textrank' for language English (en). This usually happens when spaCy calls nlp.create_pipe with a custom component name that's not registered on the current language class. If you're using a Transformer, make sure to install 'spacy-transformers'. If you're using a custom component, make sure you've added the decorator @Language.component (for function components) or @Language.factory (for class components).

Available factories: attribute_ruler, tok2vec, merge_noun_chunks, merge_entities, merge_subtokens, token_splitter, parser, beam_parser, entity_linker, ner, beam_ner, entity_ruler, lemmatizer, tagger, morphologizer, senter, sentencizer, textcat, textcat_multilabel, en.lemmatizer

Are the changes between the two versions of pytextrank documented anywhere?

The queries seem to be giving different results, so I would like to understand if that is because of changes to spaCy or to the algorithm itself?

Thank you for your help.

Hi there, I'm working on a project extracting keywords from a german text. Is there a tutorial on how to extract keywords using pytextrank?

Best regards,

Fixed by changing the code on pytextrank (307) from: try: graph.edge[pair[0]][pair[1]]["weight"] += 1.0 except KeyError: graph.add_edge(pair[0], pair[1], weight=1.0) to:
if "edge" in dir(graph): graph.edge[pair[0]][pair[1]]["weight"] += 1.0 else: graph.add_edge(pair[0], pair[1], weight=1.0)

Hey @ceteri I have added basic version of biased textrank.

It takes into account "focus" as well "bias" to augment ranking in favour of focus. As per the paper, it should add bias to the graph based on similarity calculation between "focus" and nodes. But this version just assigns "bias" to focus terms while leaving other nodes unbiased.

Let me know of your ideas so that we can improve upon this version.

@louisguitton

Hi,

I'm getting the following error when trying to run pytextrank with my own data. Is there a way to fix this?

app_1 | Traceback (most recent call last): app_1 | File "index.py", line 26, in app_1 | for rl in pytextrank.normalize_key_phrases(path_stage1, ranks): app_1 | File "/usr/local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 581, in normalize_key_phrases app_1 | for rl in collect_entities(sent, ranks, stopwords, spacy_nlp): app_1 | File "/usr/local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 485, in collect_entities app_1 | w_ranks, w_ids = find_entity(sent, ranks, ent.text.split(" "), 0) app_1 | File "/usr/local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 454, in find_entity app_1 | return find_entity(sent, ranks, ent, i + 1) app_1 | File "/usr/local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 454, in find_entity app_1 | return find_entity(sent, ranks, ent, i + 1) app_1 | File "/usr/local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 454, in find_entity app_1 | return find_entity(sent, ranks, ent, i + 1) app_1 | [Previous line repeated 137 more times] app_1 | File "/usr/local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 451, in find_entity app_1 | w = sent[i + j] app_1 | IndexError: list index out of range

This PR was automatically created by Snyk using the credentials of a real user.

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
  • requirements-dev.txt

pymdown-extensions 8.0 requires Markdown, which is not installed.
mkdocs-material 8.0.1 requires mkdocs, which is not installed.
mkdocs-material 8.0.1 requires mkdocs-material-extensions, which is not installed.
mkdocs-material 8.0.1 requires markdown, which is not installed.
mkdocs-material 8.0.1 has requirement pymdown-extensions>=9.0, but you have pymdown-extensions 8.0.

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 441/1000
Why? Recently disclosed, Has a fix available, CVSS 3.1 | Regular Expression Denial of Service (ReDoS)
SNYK-PYTHON-SETUPTOOLS-3113904 | setuptools:
39.0.1 -> 65.5.1
| No | No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:

🦉 Regular Expression Denial of Service (ReDoS)

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
  • requirements-dev.txt

pymdown-extensions 8.0 requires Markdown, which is not installed.
mkdocs-material 8.0.1 requires mkdocs, which is not installed.
mkdocs-material 8.0.1 requires markdown, which is not installed.
mkdocs-material 8.0.1 requires mkdocs-material-extensions, which is not installed.
mkdocs-material 8.0.1 has requirement pymdown-extensions>=9.0, but you have pymdown-extensions 8.0.

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 441/1000
Why? Recently disclosed, Has a fix available, CVSS 3.1 | Regular Expression Denial of Service (ReDoS)
SNYK-PYTHON-SETUPTOOLS-3113904 | setuptools:
39.0.1 -> 65.5.1
| No | No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:

🦉 Regular Expression Denial of Service (ReDoS)

The current approach which specifies stopwords as lemma: [POS] presents two issues:

1. There are some terms which POS taggers will fail over. For example, Spacy labels "AI" (artificial intelligence) as PROPN
2. If I create software to be used by people without linguistic knowledge, I cannot expect them to know about POS.

As a work-around, it is necessary to specify all POS tags, which is rather inelegant.

I'm trying to use this package for the first time and followed the README:

!pip install pytextrank
!python -m spacy download en_core_web_sm
nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("textrank")

This throws an error at the last line:

ValueError: [E002] Can't find factory for 'textrank' for language English (en). This usually happens when spaCy calls 'nlp.create_pipe' with a custom component name that's not registered on the current language class. If you're using a Transformer, make sure to install 'spacy-transformers'. If you're using a custom component, make sure you've added the decorator '@Language.component' (for function components) or '@Language.factory' (for class components).

Available factories: attribute_ruler, tok2vec, merge_noun_chunks, merge_entities, merge_subtokens, token_splitter, doc_cleaner, parser, beam_parser, lemmatizer, trainable_lemmatizer, entity_linker, ner, beam_ner, entity_ruler, tagger, morphologizer, senter, sentencizer, textcat, spancat, future_entity_ruler, span_ruler, textcat_multilabel, en.lemmatizer`

Is this an incompatibility with SpaCy version 3.3.1 or have I overseen something crucial? Which SpaCy version do you recommend? (I restarted the kernel after installing pytextrank)

https://github.com/DerwenAI/pytextrank/blob/9ab64507a26f946191504598f86021f511245cd7/pytextrank/base.py#L305

self.focus_tokens is initialized to an empty set but I don't see where it is parameterized?

e.g.

nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("biasedtextrank")
focus = "my example focus"
doc = nlp(text)

At what point can I inform the model of the focus?

Hi,

I use this library together with spacy for the extraction of the most important words. However, when using the catalan model of spacy, the algorithm gives the following error:

`File "/code/app.py", line 20, in getNlpEntities

entities = runTextRankEntities(hl, contents['contents'], algorithm, num)

File "/code/nlp/textRankEntities.py", line 51, in runTextRankEntities

doc = nlp(joined_content)

File "/usr/local/lib/python3.9/site-packages/spacy/language.py", line 1022, in call

error_handler(name, proc, [doc], e)

File "/usr/local/lib/python3.9/site-packages/spacy/util.py", line 1617, in raise_error

raise e

File "/usr/local/lib/python3.9/site-packages/spacy/language.py", line 1017, in call

doc = proc(doc, **component_cfg.get(name, {}))  # type: ignore[call-arg]

File "/usr/local/lib/python3.9/site-packages/pytextrank/base.py", line 253, in call

doc._.phrases = doc._.textrank.calc_textrank()

File "/usr/local/lib/python3.9/site-packages/pytextrank/base.py", line 363, in calc_textrank

nc_phrases = self._collect_phrases(self.doc.noun_chunks, self.ranks)

File "/usr/local/lib/python3.9/site-packages/pytextrank/base.py", line 548, in _collect_phrases

return {

File "/usr/local/lib/python3.9/site-packages/pytextrank/base.py", line 549, in

span: self._calc_discounted_normalised_rank(span, sum_rank)

File "/usr/local/lib/python3.9/site-packages/pytextrank/base.py", line 592, in _calc_discounted_normalised_rank

phrase_rank = math.sqrt(sum_rank / (len(span) + non_lemma))

ZeroDivisionError: division by zero`