I am developing an Android application (I am a beginner). I want to use Stanford CoreNPL 3.8.0 library in my app to extract the part of speech, the lemma, the parser and so on from the user sentences.I have tried a simple java code in NetBeans by following this youtube tutorial https://www.youtube.com/watch?v=9IZsBmHpK3Y, and it is working perfectly.The jar files that I imported to the NetBeans project are: stanford-corenlp-3.8.0.jar and stanford-corenlp-3.8.0-models.jar.

And this is the java source code:

import edu.stanford.nlp.ling.CoreAnnotations;
import edu.stanford.nlp.ling.CoreLabel;
import edu.stanford.nlp.pipeline.Annotation;
import edu.stanford.nlp.pipeline.StanfordCoreNLP;
import edu.stanford.nlp.util.CoreMap;

import java.util.List;
import java.util.Properties;

public class CoreNlpExample {

    public static void main(String[] args) {

        // creates a StanfordCoreNLP object, with POS tagging, lemmatization, NER, parsing, and coreference resolution
        Properties props = new Properties();
        props.setProperty("annotators", "tokenize, ssplit, pos, lemma, ner, parse, dcoref");
        StanfordCoreNLP pipeline = new StanfordCoreNLP(props);

        // read some text in the text variable
        String text = "What is the Weather in Bangalore right now?";

        // create an empty Annotation just with the given text
        Annotation document = new Annotation(text);

        // run all Annotators on this text
        pipeline.annotate(document);

        List<CoreMap> sentences = document.get(CoreAnnotations.SentencesAnnotation.class);

        for (CoreMap sentence : sentences) {
            // traversing the words in the current sentence
            // a CoreLabel is a CoreMap with additional token-specific methods
            for (CoreLabel token : sentence.get(CoreAnnotations.TokensAnnotation.class)) {
                // this is the text of the token
                String word = token.get(CoreAnnotations.TextAnnotation.class);
                // this is the POS tag of the token
                String pos = token.get(CoreAnnotations.PartOfSpeechAnnotation.class);
                // this is the NER label of the token
                String ne = token.get(CoreAnnotations.NamedEntityTagAnnotation.class);

                System.out.println(String.format("Print: word: [%s] pos: [%s] ne: [%s]", word, pos, ne));
            }
        }
    }
}

I wanted to try the same code in Android Studio and display the result in a textview, but I am facing a problem with adding these external libraries in my Android Studio 3.0.1 project.

I have read on some websites that I need to reduce the size of the jar files, and I did that and made sure that the reduced jars are still working fine in the Netbeans project. But I am still facing problems in Android studio and this is the error that I am getting:

java.lang.VerifyError: Rejecting class edu.stanford.nlp.pipeline.StanfordCoreNLP that attempts to sub-type erroneous class edu.stanford.nlp.pipeline.AnnotationPipeline (declaration of 'edu.stanford.nlp.pipeline.StanfordCoreNLP' appears in /data/app/com.example.fatimah.nlpapplication-bhlUJOCUwLhSbkWE7NBERA==/split_lib_dependencies_apk.apk)

Any suggestions on how I can fix this and import Stanford library successfully?

These changes substantially cut down the processing time; by several hours when I process all of Wikipedia. Feel free to benchmark on your own data.

The first commit uses JaFaMa instead of java.lang.Math, which is 2-3x faster for exp, log: http://blog.element84.com/improving-java-math-perf-with-jafama.html In some places I switched back to log1p, because the runtime of log and log1p in JaFaMa are similar, and log1p offers better precision for small values of x than log(1+x).

The other patches optimize the crucial code around the Viterbi algorithm:

• HotSpot optimizes better if large functions with multiple loops are split into multiple methods (as they can be recompiled independently).
• It pays off to save repeated nested array lookups (e.g. array[i][j] in a loop over j; move array_i = array[i] outside of the loop and use array_i[j] inside).
• I also add a cache to avoid recomputing the open tags set in Ttags.

All of these may appear to be trivial changes, but once you benchmark you will see how much this improves the run time.

Processing the first 20000 articles with tokenize,ssplit,pos, doing some further processing such as my own lemmatization based on hunspell, and then loading them into a lucene index with the CoreNLP master branch took 08:51 minutes, and with my patches only 04:38 minutes (sloopy benchmark only). I consider this a substantial speedup, because Wikipedia is 5.3 million articles, and it still needed 19 hours to build the full text index, but it used to take almost two days...

I see a lot of logs printed to System.out or System.err. Would it be possible to use a library like log4j http://logging.apache.org/log4j/2.x/ and use log.error, log.warning, log.info, log.debug instead? That would make it easier for users of the StanfordCoreNLP to manage which logs should be printed by choosing the log level of the project.

Hello,

I had a situation with text that had this: ""=

It seems to throw an error when I try running the pipeline with quote annotation on this small fragment. Just wanted to verify that it was an issue.

Thank you.

Happens with the following sentence, under version 3.9.2, only when adding openIE annotator:

It was a long and stern face, but with eyes that twinkled in a kindly way.

stack trace:

java.lang.AssertionError at edu.stanford.nlp.naturalli.Util.cleanTree(Util.java:324) at edu.stanford.nlp.naturalli.OpenIE.annotateSentence(OpenIE.java:463) at edu.stanford.nlp.naturalli.OpenIE.lambda$annotate$2(OpenIE.java:547) at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) at edu.stanford.nlp.naturalli.OpenIE.annotate(OpenIE.java:547) at edu.stanford.nlp.pipeline.AnnotationPipeline.annotate(AnnotationPipeline.java:76) at edu.stanford.nlp.pipeline.StanfordCoreNLP.annotate(StanfordCoreNLP.java:637) at edu.stanford.nlp.pipeline.StanfordCoreNLP.annotate(StanfordCoreNLP.java:629)

to replicate:

        Properties props = new Properties();
        props.setProperty("annotators", "tokenize,ssplit,pos,lemma,ner,depparse,natlog,openie");
        StanfordCoreNLP pipeline = new StanfordCoreNLP(props);

        String text = "It was a long and stern face, but with eyes that twinkled in a kindly way.";

        CoreDocument document = new CoreDocument(text);
        pipeline.annotate(document);

works fine if openie is disabled, with other sentences, or when using https://corenlp.run/ so looks like it's fixed in later versions but I did not verify it locally as I can't upgrade at the moment anyway.

advice much appreciated

I have been trying to use the CoreNLP server using various python packages including Stanza. I am always running into the same problem that I do not hear back from the server.

I downloaded a copy of CoreNLP from the website. I then try to start a server from the terminal and go to my localhost as described here. Based on the documentation I should see something when I go to http://localhost:9000/, but nothing loads up.

Here are to commands I use:

cd stanford-corenlp-full-2018-10-05/
java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000

Here is the output of running the commands above:

Samarths-MacBook-Pro-2:stanford-corenlp-full-2018-10-05 samarthbhandari$ java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000
[main] INFO CoreNLP - --- StanfordCoreNLPServer#main() called ---
[main] INFO CoreNLP - setting default constituency parser
[main] INFO CoreNLP - warning: cannot find edu/stanford/nlp/models/srparser/englishSR.ser.gz
[main] INFO CoreNLP - using: edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz instead
[main] INFO CoreNLP - to use shift reduce parser download English models jar from:
[main] INFO CoreNLP - http://stanfordnlp.github.io/CoreNLP/download.html
[main] INFO CoreNLP -     Threads: 8
[main] INFO CoreNLP - Starting server...
[main] INFO CoreNLP - StanfordCoreNLPServer listening at /0:0:0:0:0:0:0:0:9000

I then go to http://localhost:9000/, nothing loads up. Like I mentioned above originally I have been trying to do the same thing using some of the python packages and observed similar behavior.

Here is a stack overflow post related to server not responding using Stanza.

OS: MacOS 10.15.4 Python: 3.7.7 Java: 1.8

double arrays are a large portion of the heap.

There are some places with 2d double arrays with dimensions like

345k x 16, 150k x 24, 80k x 46: CRFCLassifier.weights 100k x 1000: Classifier.saved in DependecyParser 60k x 50: Classifier.E, .eg2E 1000x2400: Classifier.W1, .wg2W1

Most are weights of some sort, making me wonder if they could be stored in less than 64bit each.

The obvious step would be to use float[], halving the memory use of this portion.

Another would be to encode weights in something else, for example a small integer and scale that into a float again when using the weight.

Machine Learning models often use fp16 or even fp8 to store weights, there are java implementations of float -> short -> float (with fp16 semantics stored in a 16bit short)

like https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/util/Half.java with https://android.googlesource.com/platform/libcore/+/master/luni/src/main/java/libcore/util/FP16.java

or https://stackoverflow.com/questions/6162651/half-precision-floating-point-in-java

The latter approached would need some performance testing as each time a weight is used it would have to be converted first.

I saw that some models serialize themselves using ObjectStreams, that would need an adapter to deserialize to double[] first and then array-cast it to float[].

Like in CRFClassifier.loadClassifier

Following function in TokenSequenceParser class ignores tail of patterns defined in rules for tokensregex

private String getStringFromTokens(Token head, Token tail, boolean includeSpecial) { StringBuilder sb = new StringBuilder(); for( Token p = head ; p != tail ; p = p.next ) { if (includeSpecial) { appendSpecialTokens( sb, p.specialToken ); } sb.append( p.image ); } return sb.toString(); }

Eg: ([{lemma:/([a-zA-Z]{2,}_)?[a-zA-Z]{2,}[0-9]{2,}/}]) gets converted to ([{lemma:/([a-zA-Z]{2,}_)?[a-zA-Z]{2,}[0-9]{2,}/}] while reading and don't provide intended matches

CoreNLP version 4.5.0 using pos lemma depparse. I run the pipeline within Spark (Scala). I lazy initialise the CoreNLP pipeline and I broadcast the pipeline to each executor using lazy instantiation wrapped in a case object. Also I force not to split the text fragment as it is intended to be a sentence already. The objective here is to do dependency analysis on the sentence and run some semgraph rules against it. We got a case where it throws an exception like this

Caused by: edu.stanford.nlp.semgraph.UnknownVertexException: Operation attempted on unknown vertex happens/VBZ'''' in graph -> observed/VBD (root)
  -> 24/CD (nsubj)
    -> response/NN (nmod:in)
      -> In/IN (case)
      -> CoV/NNP (nmod:to)
        -> to/IN (case)
        -> SARS/NNP (compound)
        -> ‐/SYM (dep)
        -> ‐/SYM (dep)
        -> peptides/NNS (dep)
          -> 2/CD (nummod)
  -> ,/, (punct)
  -> we/PRP (nsubj)
  -> unexpectedly/RB (advmod)
  -> associated/VBN (ccomp)
    -> that/IN (mark)
    -> sirolimus/NN (nsubj:pass)
    -> was/VBD (aux:pass)
    -> significantly/RB (advmod)
    -> release/NN (obl:with)
      -> with/IN (case)
      -> a/DT (det)
      -> proinflammatory/JJ (amod)
      -> cytokine/NN (compound)
      -> levels/NNS (nmod:including)
        -> including/VBG (case)
        -> higher/JJR (amod)
        -> α/NN (nmod:of)
          -> of/IN (case)
          -> TNF/NN (compound)
          -> ‐/SYM (dep)
          -> IL/NN (conj:and)
            -> and/CC (cc)
        -> IL/NN (nmod:of)
        -> 1β/NN (nmod)
          -> ‐/SYM (dep)
  -> ./. (punct)

	at edu.stanford.nlp.semgraph.SemanticGraph.parentPairs(SemanticGraph.java:730)
	at edu.stanford.nlp.semgraph.semgrex.GraphRelation$DEPENDENT$1.advance(GraphRelation.java:325)
	at edu.stanford.nlp.semgraph.semgrex.GraphRelation$SearchNodeIterator.initialize(GraphRelation.java:1103)
	at edu.stanford.nlp.semgraph.semgrex.GraphRelation$SearchNodeIterator.<init>(GraphRelation.java:1084)
	at edu.stanford.nlp.semgraph.semgrex.GraphRelation$DEPENDENT$1.<init>(GraphRelation.java:310)
	at edu.stanford.nlp.semgraph.semgrex.GraphRelation$DEPENDENT.searchNodeIterator(GraphRelation.java:310)
	at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.resetChildIter(NodePattern.java:339)
	at edu.stanford.nlp.semgraph.semgrex.SemgrexMatcher.resetChildIter(SemgrexMatcher.java:80)
	at edu.stanford.nlp.semgraph.semgrex.CoordinationPattern$CoordinationMatcher.resetChildIter(CoordinationPattern.java:168)
	at edu.stanford.nlp.semgraph.semgrex.CoordinationPattern$CoordinationMatcher.resetChildIter(CoordinationPattern.java:168)
	at edu.stanford.nlp.semgraph.semgrex.CoordinationPattern$CoordinationMatcher.resetChildIter(CoordinationPattern.java:168)
	at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.resetChild(NodePattern.java:363)
	at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.goToNextNodeMatch(NodePattern.java:457)
	at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.matches(NodePattern.java:574)
	at edu.stanford.nlp.semgraph.semgrex.SemgrexMatcher.find(SemgrexMatcher.java:193)
	at az.bikg.nlp.etl.common.nlp.Pattern.go$3(Pattern.scala:200)
	at az.bikg.nlp.etl.common.nlp.Pattern.$anonfun$findCauseEffectMatches$6(Pattern.scala:268)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at az.bikg.nlp.etl.common.nlp.Pattern.findCauseEffectMatches(Pattern.scala:266)
	at az.bikg.nlp.etl.steps.ERs$.findRelations(ERs.scala:107)
	at az.bikg.nlp.etl.steps.ERs$.findRelationsSpark(ERs.scala:229)
	at az.bikg.nlp.etl.steps.ERs$.$anonfun$extractERs$1(ERs.scala:242)
	... 28 more

Am I doing anything wrong because of this exception? It didn't happen with version 4.4.0.

I tested the latest Chinese CoreNLP 3.92 version, and found the results are quite horrible. Here are few examples:

我的朋友：always tags "我的" as one NN token. 我的狗吃苹果： ‘我的狗’ tagged as one NN token. 他的狗吃苹果：'狗吃' tagged as one NN token. 高质量就业成时代: '就业' tagged as VV

When I compared them with the results from http://nlp.stanford.edu:8080/parser/index.jsp, surprisingly, the results of those examples are all right. Why is that? Are the models different? Is there a bug in the new 3.92 version model?

I use Stanford Core NLP in Java as a Maven dependency. I want to use the MaxentTagger with a model supplied in the stanford-corenlp-3.5.2-models package. The problem is that I cannot access this model through classpath.

My code is

tagger = new MaxentTagger("/edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger");

The file "/edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger" exists in the jar and should be loaded through classpath, but the following exception is thrown

Exception in thread "main" edu.stanford.nlp.io.RuntimeIOException: Unrecoverable error while loading a tagger model
    at edu.stanford.nlp.tagger.maxent.MaxentTagger.readModelAndInit(MaxentTagger.java:770)
    at edu.stanford.nlp.tagger.maxent.MaxentTagger.<init>(MaxentTagger.java:298)
    at edu.stanford.nlp.tagger.maxent.MaxentTagger.<init>(MaxentTagger.java:263)
    at cz.zcu.kiv.nlp.semeval.cwi.features.POSFeature.<init>(POSFeature.java:24)
    at cz.zcu.kiv.nlp.semeval.cwi.CWIModel.train(CWIModel.java:60)
    at cz.zcu.kiv.nlp.semeval.cwi.TrainingCrossValidation.main(TrainingCrossValidation.java:51)
Caused by: java.io.IOException: Unable to resolve "/edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger" as either class path, filename or URL
    at  edu.stanford.nlp.io.IOUtils.getInputStreamFromURLOrClasspathOrFileSystem(IOUtils.java:481)
    at edu.stanford.nlp.tagger.maxent.MaxentTagger.readModelAndInit(MaxentTagger.java:765)
    ... 5 more

If I copy the model out of the jar and use e.g.

tagger = new MaxentTagger("./english-left3words-distsim.tagger");

then everything works perfectly.

The problem is probably in the class IOUtils, method findStreamInClasspathOrFileSystem(String name).

In the line

InputStream is = IOUtils.class.getClassLoader().getResourceAsStream(name);

the returned classloader is probably the JarClassLoader which loaded the library (stanford-corenlp-3.5.2.jar) and it does not have access to other libraries.

This theory is supported by the following code

InputStream stream = POSFeature.class.getResourceAsStream("/edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger");
System.out.println("Stream == null: " + (stream == null));

tagger = new MaxentTagger("/edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger");

which outputs

Stream == null: false
Exception in thread "main" edu.stanford.nlp.io.RuntimeIOException: Unrecoverable error while loading a tagger model
...
Caused by: java.io.IOException: Unable to resolve "/edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger" as either class path, filename or URL

On README, the way of how to install models jars is written but a method using build tools(e.g. Gradle) is not written. However, I try this way( https://stackoverflow.com/a/68859054/3809427 ) and succeeded. Why don't you write this useful method?

I ran into an issue where if an empty string is passed in then getting the entityMentions returns null instead of an empty list like I figure out be standard practice.

Example code:

StanfordCoreNLP processor = new StanfordCoreNLP(props);
CoreDocument nlpDocument = new CoreDocument("");

nlpProcessor.annotate(nlpDocument);
List<CoreEntityMention>  entities = nlpDocument.entityMentions(); <== returns null

Just wanted to know if there is any light that can be shed on this. If this is expected behavior then I will try my best to document it in the documentation

In the following sentence (from Twitter), 'email' is being tokenized as 'em, ail, and '. This is obviously incorrect. What can be done to stop this split?

• It's official (according to the AP) it's 'email' not 'e-mail' and 'website' not 'web-site'!

I have the following parameters set: tokenize.language: English tokenize.whitespace: false (because we want tokens like it's to separate into it and 's) tokenize.keepeol: false tokenize.verbose: false tokenize.options: invertible=true,splitAssimilations=false,splitHyphenated=false,splitForwardSlash=true,untokenizable=allKeep,strictTreebank3=true,normalizeSpace=false,ellipses=original

curl 'http://localhost:9000/?properties={%22annotators%22%3A%22lemma%22%2C%22outputFormat%22%3A%22json%22}' -d '`'

Gives me:

{
  "sentences": [
    {
      "index": 0,
      "tokens": [
        {
          "index": 1,
          "word": "`",
          "originalText": "`",
          "lemma": "`",
          "characterOffsetBegin": 0,
          "characterOffsetEnd": 1,
          "pos": "``",
          "before": "",
          "after": ""
        }
      ]
    }
  ]
}

With the standard English model. Is this expected? I'm particularly surprised at the POS.

A unit test that runs in a loop calling pipeline.annotate(document) appears to be taking about 50% longer. Our configuration properties didn't change during the upgrade, but maybe some new properties have been added in 4.5.1? Below is what we have. Is there a way to determine which annotator is using more time now?

customAnnotatorClass.tokensregex=edu.stanford.nlp.pipeline.TokensRegexAnnotator sutime.binders=0 tokensregex.rules= .... (omitted) ssplit.eolonly=false customAnnotatorClass.tokenOverride_en= .... (omitted) annotators=tokenize, ssplit, tokenOverride_en, pos, lemmaOverride_en, ner, tokensregex, entitymentions, parse language=en tokenize.whitespace=false customAnnotatorClass.lemmaOverride_en=.... (omitted) tokenize.options=untokenizable=allKeep,americanize=false ssplit.isOneSentence=true nermention.acronyms=true