Differences between revisions 26 and 27

Benchmarks

This page documents performance of most popular contemporary NLP systems for Polish.

Morphological analysis

System name and URL	Approach	Main publication	License	P	R	F
Morfeusz		Woliński, M. Morfeusz — a practical tool for the morphological analysis of Polish. In M.A. Kłopotek, S.T. Wierzchoń, K. Trojanowski (eds.) Proceedings of the International IIS:IIPWM 2006 Conference, pp. 511–520, Wisła, Poland, 2006.	2-clause BSD	%	%	%
Morfologik		Miłkowski M. Developing an open-source, rule-based proofreading tool. Software: Practice and Experience, 40(7):543–566, 2010.		%	%	%

POS tagging

The comparisons are performed using plain text as input and reporting the accuracy lower bound (Acc_lower) metric proposed by Radziszewski and Acedański (Taggers gonna tag: an argument against evaluating disambiguation capacities of morphosyntactic taggers. In Proceedings of TSD 2012, LNCS, Springer Verlag). The metric penalizes all segmentation changes in regard to the gold standard and treats such tokens as misclassified. Furthermore, we report separate metric values for both known and unknown words to assess the performance of guesser modules built into the taggers. These are indicated as Acc^K_lower for known and Acc^U_lower for unknown words.

The experiments have been performed on the manually annotated part of the National Corpus of Polish v. 1.1 (1M tokens). The ten-fold cross-validation procedure has been followed, by re-evaluating the methods ten times, each time selecting one of ten parts of the corpus for testing and the remaining parts for training the taggers. The provided results are averages calculated over ten training and testing sequences. Each of the taggers and each tagger ensemble has been trained and tested on the same set of cross-validation folds, so the results are directly comparable. Each of the training folds has been reanalyzed, according to the procedure described in (Radziszewski A. A Tiered CRF Tagger for Polish. In R. Bembenik, Ł. Skonieczny, H. Rybiński, M. Kryszkiewicz, M. Niezgódka (eds.) Intelligent Tools for Building a Scientific Information Platform, Springer Verlag, 2013.), using the Maca toolkit (Radziszewski A., Śniatowski T. Maca – a configurable tool to integrate Polish morphological data. In Proceedings of the 2nd International Workshop on Free/Open-Source Rule-Based Machine Translation, 2011). The idea of a morphological reanalysis of the gold-standard data is to allow the trained tagger to see similar input that is expected in the tagging phase. The training data is firstly turned into plain text and analyzed using the same mechanism that will be used by the tagger during actual tagging process. The output of the analyzer is then synchronized with the original gold-standard data, by using the original tokenization. Tokens with changed segmentation are taken from the gold-standard intact. In the case of tokens for which the segmentation did not change in the process of morphological analysis, the produced interpretations are compared with the original. A token is marked as an unknown word, if the correct interpretation has not been produced by the analyzer. In our experiments, Maca has been run with the morfeusz-nkjp-official configuration, which uses Morfeusz SGJP analyzer (Woliński, M. Morfeusz — a practical tool for the morphological analysis of Polish. In: Kłopotek, M.A., Wierzchoń, S.T., Trojanowski, K. (eds.) Proceedings of the International IIS: IIPWM’06 Conference, pp. 511–520, Wisła, Poland, 2006) and no guesser module.

System name and URL	Approach	Main publication	License	Acc_lower	Acc^K_lower	Acc^U_lower
Pantera	rule-based adapted Brill tagger	Acedański S. A Morphosyntactic Brill Tagger for Inflectional Languages. In H. Loftsson, E. Rögnvaldsson, S. Helgadóttir (eds.) Advances in Natural Language Processing, LNCS 6233, pp. 3–14, Springer, 2010.	GPL 3	88.95%	91.22%	15.19%
TaKIPI	hybrid (multiclassifier) rule-based	Piasecki M. Polish Tagger TaKIPI: Rule Based Construction and Optimisation. Task Quarterly 11, pp. 151-167, 2007.		%	%	%
WMBT	memory-based	Radziszewski A., Śniatowski T. A memory-based tagger for Polish. In Proceedings of LTC 2011.		90.33%	91.26%	60.25%
WCRFT	tiered, CRF-based	Radziszewski A. A Tiered CRF Tagger for Polish. In R. Bembenik, Ł. Skonieczny, H. Rybiński, M. Kryszkiewicz, M. Niezgódka (eds.) Intelligent Tools for Building a Scientific Information Platform, Springer Verlag, 2013.	LGPL 3.0	90.76%	91.92%	53.18%
Concraft	mutually dependent CRF layers	Waszczuk J. Harnessing the CRF complexity with domain-specific constraints. The case of morphosyntactic tagging of a highly inflected language. In Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012), pp. 2789–2804, Mumbai, India, 2012.	2-clause BSD	91.07%	92.06%	58.81%
PoliTa	voting ensemble	Kobyliński Ł. PoliTa: A multitagger for Polish. In N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, S. Piperidis (eds.) Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014), pp. 2949–2954, Reykjavík, Iceland, ELRA, 2014.	GPL	92.01%	92.91%	62.81%

Constituency parsing

System name and URL

Approach

Main publication

License

|| Spejd || rule-based || Buczyński A., Przepiórkowski A. Spejd: A shallow processing and morphological disambiguation tool. In Z. Vetulani, H. Uszkoreit (eds.) Human Language Technology: Challenges of the Information Society, LNCS 5603, pp. 131–141. Springer-Verlag, Berlin, 2009. || GPL 3 || % || % || % ||

Dependency parsing

System name and URL	Approach	Main publication	License	LAS	UAS
Polish Dependency parser (MaltParser)	trained on the extended version of the Polish dependency treebank with MaltParser	Wróblewska A. Polish Dependency Parser Trained on an Automatically Induced Dependency Bank. PhD dissertation, Institute of Computer Science, Polish Academy of Sciences, Warsaw, 2014.		84%	89%
Polish Dependency parser (MateParser)	trained on the same data with MateParser			89%	93%
Google parser				%	%

Deep parsing

System name and URL	Approach	Main publication	License	P	R	F
Świgra				%	%	%
POLFIE			GPL 3 (grammar)	%	%	%
ENIAM				%	%	%

Word sense disambiguation

System name and URL	Approach	Main publication	License	P	R	F
WSDDE	Machine-learning	Kopeć M., Młodzki R., Przepiórkowski A. Automatyczne znakowanie sensami słów. In A. Przepiórkowski, M. Bańko, R.L. Górski, B. Lewandowska-Tomaszczyk (eds.) Narodowy Korpus Języka Polskiego, pp. 209–224. Wydawnictwo Naukowe PWN, Warsaw, 2012.	GPL 3	%	%	%

Named entity recognition

System name and URL	Approach	Main publication	License	P	R	F
NERF linear-chain CRF	Savary A., Waszczuk J., Przepiórkowski A. Towards the annotation of named entities in the National Corpus of Polish. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010), pp. 3622–3629, Valletta, Malta, ELRA, 2010.	2-clause BSD		%	%	%
Liner2 with TIMEX model		Kocoń J., Marcińczuk M. Liner2.5 model Timex, CLARIN-PL digital repository, 2016.		%	%	%

Sentiment analysis

System name and URL	Approach	Main publication	License	P	R	F
Sentipejd				%	%	%

Mention detection

Precision, recall and F-measure are calculated on Polish Coreference Corpus data with two alternative mention detection scores:

EXACT: score of exact boundary matches (an automatic and a manual mention match if they have exactly the same boundaries; in other words, they consist of the same tokens)
HEAD: score of head matches (we reduce the automatic and the manual mentions to their single head tokens and compare them).

System name and URL	Approach	Main publication	License	EXACT			HEAD
System name and URL	Approach	Main publication	License	P	R	F	P	R	F
Mention Detector	Collects mention candidates from available sources – morphosyntactical, shallow parsing, named entity and/or zero anaphora detection tools	Ogrodniczuk M., Głowińska K., Kopeć M., Savary A., Zawisławska M. Coreference in Polish: Annotation, Resolution and Evaluation, chapter 10.6. Walter De Gruyter, 2015.	CC BY 3	66.79%	67.21%	67.00%	88.29%	89.41%	88.85%

Coreference resolution

As there is still no consensus about the single best coreference resolution metrics, CoNLL measure is used (average of MUC, B3 and CEAFE F-measure values). For end-to-end systems CoNLL-2011 shared task-based approach is used, so two result calculation strategies are presented:

INTERSECT: consider only correct system mentions (i.e. the intersection between gold and system mentions)
TRANSFORM: unify system and gold mention sets using the following procedure for twinless mentions (without a corresponding mention in the second set):
1. insert twinless gold mentions into system mention set as singletons
2. remove twinless singleton system mentions
3. insert twinless non-singletion system mentions into gold set as singletons.

The results are produced on Polish Coreference Corpus data.

System name and URL	Approach	Main publication	License	GOLD	EXACT INTERSECT	EXACT TRANSFORM	HEAD INTERSECT	HEAD TRANSFORM
Ruler	Rule-based	Ogrodniczuk M., Kopeć M. End-to-end coreference resolution baseline system for Polish. In Z. Vetulani (ed.), Proceedings of the 5th Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, pp. 167–171, Poznań, Poland, 2011.	CC BY 3	73.40%	78.54%	66.55%	76.27%	70.11%
Bartek3	Statistical	Kopeć M., Ogrodniczuk M. Creating a Coreference Resolution System for Polish. In Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012, pp. 192–195, ELRA.	CC BY 3	78.41%	80.86%	68.96%	78.58%	72.15%

Extractive summarization

System name and URL	Approach	Main publication	License	P	R	F
PolSum		Ciura M., Grund D., Kulików S., Suszczańska N. A System to Adapt Techniques of Text Summarizing to Polish. In Okatan A. (ed.) International Conference on Computational Intelligence, pp. 117–120, Istanbul, Turkey. International Computational Intelligence Society, 2004.		%	%	%
Lakon		Dudczak A. Zastosowanie wybranych metod eksploracji danych do tworzenia streszczeń tekstów prasowych dla języka polskiego. MSc thesis, Poznań Technical University, 2007.		%	%	%
Summarizer	machine-learning	Świetlicka J. Metody maszynowego uczenia w automatycznym streszczaniu tekstów. MSc thesis, Warsaw University 2010.		%	%	%
Emily				%	%	%
Nicolas				%	%	%

-  ⇤ ← Revision 26 as of 2016-10-28 00:29:10 → 
  Size: 13119
  Editor: MaciejOgrodniczuk
  Comment:
+   ← Revision 27 as of 2016-10-28 00:50:40 → ⇥
  Size: 14847
  Editor: MaciejOgrodniczuk
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 21:
-|| [[http://zil.ipipan.waw.pl/PANTERA|Pantera]]   || rule-based adapted Brill tagger      || Acedański S. ''A Morphosyntactic Brill Tagger for Inflectional Languages''. In H. Loftsson, E. Rögnvaldsson, S. Helgadóttir (eds.) Advances in Natural Language Processing, LNCS 6233, pp. 3–14, Springer, 2010. || GPL 3 || 88.95%  || 91.22% || 15.19% ||
|| [[http://nlp.pwr.wroc.pl/takipi/|TaKIPI]]     || hybrid (multiclassifier) rule-based  || Piasecki M. ''Polish Tagger TaKIPI: Rule Based Construction and Optimisation''. Task Quarterly 11, pp. 151-167, 2007. ||  || %  || % || % ||
|| [[http://nlp.pwr.wroc.pl/redmine/projects/wmbt/wiki|WMBT]]         || memory-based      || Radziszewski A., Śniatowski T. ''A memory-based tagger for Polish''. In Proceedings of LTC 2011. ||  || 90.33% || 91.26% || 60.25% ||
|| [[http://nlp.pwr.wroc.pl/redmine/projects/wcrft/wiki|WCRFT]]       || tiered, CRF-based || Radziszewski A. ''A Tiered CRF Tagger for Polish.'' In R. Bembenik, Ł. Skonieczny, H. Rybiński, M. Kryszkiewicz, M. Niezgódka (eds.) Intelligent Tools for Building a Scientific Information Platform, Springer Verlag, 2013. || LGPL 3.0 || 90.76%  || 91.92% || 53.18% ||
|| [[http://zil.ipipan.waw.pl/Concraft|Concraft]] || mutually dependent CRF layers    || Waszczuk J. ''Harnessing the CRF complexity with domain-specific constraints. The case of morphosyntactic tagging of a highly inflected language''. In Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012), pp. 2789–2804, Mumbai, India, 2012. || 2-clause BSD || 91.07%  || 92.06% || 58.81% ||
|| [[http://zil.ipipan.waw.pl/PoliTa|PoliTa]]     || voting ensemble    || Kobyliński Ł. ''PoliTa: A multitagger for Polish''. In N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, S. Piperidis (eds.) Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014), pp. 2949–2954, Reykjavík, Iceland, ELRA, 2014. || GPL || 92.01%  || 92.91% || 62.81% ||
+|| [[http://zil.ipipan.waw.pl/PANTERA|Pantera]]   || rule-based adapted Brill tagger      || Acedański S. [[http://ripper.dasie.mimuw.edu.pl/~accek/homepage/wp-content/papercite-data/pdf/ace10.pdf|A Morphosyntactic Brill Tagger for Inflectional Languages]]. In H. Loftsson, E. Rögnvaldsson, S. Helgadóttir (eds.) Advances in Natural Language Processing, LNCS 6233, pp. 3–14, Springer, 2010. || GPL 3 || 88.95%  || 91.22% || 15.19% ||
|| [[http://nlp.pwr.wroc.pl/takipi/|TaKIPI]]     || hybrid (multiclassifier) rule-based  || Piasecki M. [[http://www.task.gda.pl/files/quart/TQ2007/01-02/tq111t-g.pdf|Polish Tagger TaKIPI: Rule Based Construction and Optimisation]]. Task Quarterly 11, pp. 151-167, 2007. ||  || %  || % || % ||
|| [[http://nlp.pwr.wroc.pl/redmine/projects/wmbt/wiki|WMBT]]         || memory-based      || Radziszewski A., Śniatowski T. [[http://nlp.pwr.wroc.pl/redmine/attachments/download/420/wmbt.pdf|A memory-based tagger for Polish]]. In Proceedings of LTC 2011. ||  || 90.33% || 91.26% || 60.25% ||
|| [[http://nlp.pwr.wroc.pl/redmine/projects/wcrft/wiki|WCRFT]]       || tiered, CRF-based || Radziszewski A. [[http://nlp.pwr.wroc.pl/ltg/files/publications/wcrft.pdf|A Tiered CRF Tagger for Polish]]. In R. Bembenik, Ł. Skonieczny, H. Rybiński, M. Kryszkiewicz, M. Niezgódka (eds.) Intelligent Tools for Building a Scientific Information Platform, Springer Verlag, 2013. || LGPL 3.0 || 90.76%  || 91.92% || 53.18% ||
|| [[http://zil.ipipan.waw.pl/Concraft|Concraft]] || mutually dependent CRF layers    || Waszczuk J. [[http://www.aclweb.org/anthology/C12-1170|Harnessing the CRF complexity with domain-specific constraints. The case of morphosyntactic tagging of a highly inflected language]]. In Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012), pp. 2789–2804, Mumbai, India, 2012. || 2-clause BSD || 91.07%  || 92.06% || 58.81% ||
|| [[http://zil.ipipan.waw.pl/PoliTa|PoliTa]]     || voting ensemble    || Kobyliński Ł. [[http://www.lrec-conf.org/proceedings/lrec2014/pdf/1018_Paper.pdf|PoliTa: A multitagger for Polish]]. In N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, S. Piperidis (eds.) Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014), pp. 2949–2954, Reykjavík, Iceland, ELRA, 2014. || GPL || 92.01%  || 92.91% || 62.81% ||
 Line 32:
-|| [[http://zil.ipipan.waw.pl/Spejd|Spejd]]   || rule-based  || Buczyński A., Przepiórkowski A. ''Spejd: A shallow processing and morphological disambiguation tool''. In Z. Vetulani, H. Uszkoreit (eds.) Human Language Technology: Challenges of the
+|| [[http://zil.ipipan.waw.pl/Spejd|Spejd]]   || rule-based  || Buczyński A., Przepiórkowski A. [[http://nlp.ipipan.waw.pl/~adamp/Papers/2007-ltc-spade/Spade.pdf|Spejd: A shallow processing and morphological disambiguation tool]]. In Z. Vetulani, H. Uszkoreit (eds.) Human Language Technology: Challenges of the
 Line 44:
-|| '''System name and URL'''            || '''Approach''' || '''Main publication''' || '''License''' || '''P''' || '''R''' || '''F''' ||
|| [[http://zil.ipipan.waw.pl/|Świgra]] ||     || ||  || %  || % || % ||
|| [[http://zil.ipipan.waw.pl/|POLFIE]] ||     || ||  || %  || % || % ||
|| [[http://zil.ipipan.waw.pl/|ENIAM]]  ||     || ||  || %  || % || % ||
+|| '''System name and URL'''                  || '''Approach''' || '''Main publication''' || '''License''' || '''P''' || '''R''' || '''F''' ||
|| [[http://zil.ipipan.waw.pl/Świgra|Świgra]] ||     || ||  || %  || % || % ||
|| [[http://zil.ipipan.waw.pl/LFG|POLFIE]]    ||     || || GPL 3 (grammar) || %  || % || % ||
|| [[http://zil.ipipan.waw.pl/ENIAM|ENIAM]]   ||     || ||  || %  || % || % ||
 Line 52:
-|| [[http://zil.ipipan.waw.pl/WSDDE|WSDDE]] || Machine-learning || Kopeć M., Młodzki R., Przepiórkowski A. ''Automatyczne znakowanie sensami słów''. In A. Przepiórkowski, M. Bańko, R.L. Górski, B. Lewandowska-Tomaszczyk (eds.) Narodowy Korpus Języka Polskiego, pp. 209–224. Wydawnictwo Naukowe PWN, Warsaw, 2012.  || GPL 3 || %  || % || % ||
+|| [[http://zil.ipipan.waw.pl/WSDDE|WSDDE]] || Machine-learning || Kopeć M., Młodzki R., Przepiórkowski A. [[http://nkjp.pl/settings/papers/NKJP_ksiazka.pdf|Automatyczne znakowanie sensami słów]]. In A. Przepiórkowski, M. Bańko, R.L. Górski, B. Lewandowska-Tomaszczyk (eds.) Narodowy Korpus Języka Polskiego, pp. 209–224. Wydawnictwo Naukowe PWN, Warsaw, 2012.  || GPL 3 || %  || % || % ||
 Line 76:
-|| [[http://zil.ipipan.waw.pl/MentionDetector|Mention Detector]]   || Collects mention candidates from available sources – morphosyntactical, shallow parsing, named entity and/or zero anaphora detection tools || Ogrodniczuk M., Głowińska K., Kopeć M., Savary A., Zawisławska M. ''Coreference in Polish: Annotation, Resolution and Evaluation'', chapter 10.6. Walter De Gruyter, 2015.  ||  CC BY 3  || 66.79% || 67.21% || 67.00% || 88.29% || 89.41% || 88.85% ||
+|| [[http://zil.ipipan.waw.pl/MentionDetector|Mention Detector]]   || Collects mention candidates from available sources – morphosyntactical, shallow parsing, named entity and/or zero anaphora detection tools || Ogrodniczuk M., Głowińska K., Kopeć M., Savary A., Zawisławska M. [[http://www.degruyter.com/view/product/428667|Coreference in Polish: Annotation, Resolution and Evaluation]], chapter 10.6. Walter De Gruyter, 2015.  ||  CC BY 3  || 66.79% || 67.21% || 67.00% || 88.29% || 89.41% || 88.85% ||
 Line 92:
-|| [[http://zil.ipipan.waw.pl/Ruler|Ruler]]   || Rule-based              || Ogrodniczuk M., Kopeć M. ''End-to-end coreference resolution baseline system for Polish''. In Z. Vetulani (ed.), Proceedings of the 5th Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, pp. 167–171, Poznań, Poland, 2011.  || CC BY 3 || 73.40%  || 78.54% || 66.55% || 76.27% || 70.11% ||
|| [[http://zil.ipipan.waw.pl/Bartek|Bartek3]] || Statistical             || Kopeć M., Ogrodniczuk M. ''Creating a Coreference Resolution System for Polish''. In Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012, pp. 192–195, ELRA. || CC BY 3 || 78.41% ||  80.86% || 68.96% || 78.58% || 72.15% ||
+|| [[http://zil.ipipan.waw.pl/Ruler|Ruler]]   || Rule-based              || Ogrodniczuk M., Kopeć M. [[http://nlp.ipipan.waw.pl/Bib/ogro:kop:11:ltc.pdf|End-to-end coreference resolution baseline system for Polish]]. In Z. Vetulani (ed.), Proceedings of the 5th Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, pp. 167–171, Poznań, Poland, 2011.  || CC BY 3 || 73.40%  || 78.54% || 66.55% || 76.27% || 70.11% ||
|| [[http://zil.ipipan.waw.pl/Bartek|Bartek3]] || Statistical             || Kopeć M., Ogrodniczuk M. [[http://www.lrec-conf.org/proceedings/lrec2012/pdf/1064_Paper.pdf|Creating a Coreference Resolution System for Polish]]. In Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012, pp. 192–195, ELRA. || CC BY 3 || 78.41% ||  80.86% || 68.96% || 78.58% || 72.15% ||
 Line 96:
-== Summarization ==
+== Extractive summarization ==
 Line 99:
-|| [[http://zil.ipipan.waw.pl/|]]   ||     || ||  || %  || % || % ||
|| [[http://zil.ipipan.waw.pl/|]]   ||     || ||  || %  || % || % ||
+|| [[http://las.aei.polsl.pl/PolSum/#/Home|PolSum]]   ||     || Ciura M., Grund D., Kulików S., Suszczańska N. [[http://sun.aei.polsl.pl/~mciura/publikacje/summarizing.pdf|A System to Adapt Techniques of Text Summarizing to Polish]]. In Okatan A. (ed.) International Conference on Computational Intelligence, pp. 117–120, Istanbul, Turkey. International Computational Intelligence Society, 2004.  ||  || %  || % || % ||
|| [[http://www.cs.put.poznan.pl/dweiss/research/lakon/|Lakon]] ||     || Dudczak A. [[http://www.cs.put.poznan.pl/dweiss/research/lakon/publications/thesis.pdf|Zastosowanie wybranych metod eksploracji danych do tworzenia streszczeń tekstów prasowych dla języka polskiego]]. MSc thesis, Poznań Technical University, 2007. ||  || %  || % || % ||
|| [[http://clip.ipipan.waw.pl/Summarizer|Summarizer]]   || machine-learning    || Świetlicka J. [[http://nlp.ipipan.waw.pl/~adamp/msc/swietlicka.joanna/TekstPracy.pdf.gz|Metody maszynowego uczenia w automatycznym streszczaniu tekstów]]. MSc thesis, Warsaw University 2010. ||  || %  || % || % ||
|| Emily     ||     || ||  || %  || % || % ||
|| Nicolas   ||     || ||  || %  || % || % ||

Diff for "benchmarks"

Menu

Wiki