|
Size: 19273
Comment:
|
Size: 19393
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 76: | Line 76: |
| || [[http://zil.ipipan.waw.pl/Nerf|NERF]] || linear-chain CRF || Savary A., Waszczuk J., Przepiórkowski A. [[http://nlp.ipipan.waw.pl/~adamp/Papers/2010-lrec-as/sav-wasz-przep-lrec-2010.pdf|Towards the annotation of named entities in the National Corpus of Polish]]. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010), pp. 3622–3629, Valletta, Malta, ELRA, 2010. || 2-clause BSD || 83% || 78% || 79% || | || [[http://zil.ipipan.waw.pl/Nerf|NERF]] || linear-chain CRF || Savary A., Waszczuk J., Przepiórkowski A. [[http://nlp.ipipan.waw.pl/~adamp/Papers/2010-lrec-as/sav-wasz-przep-lrec-2010.pdf|Towards the annotation of named entities in the National Corpus of Polish]]. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010), pp. 3622–3629, Valletta, Malta, ELRA, 2010.<<BR>> <<BR>>See also Chapters 9 and 13 in the [[http://nkjp.pl/settings/papers/NKJP_ksiazka.pdf|NKJP book]] (in Polish). || 2-clause BSD || 83% || 78% || 79% || |
Benchmarks
This page documents performance of most popular contemporary NLP systems for Polish.
Single-word lemmatization and morphological analysis
System name and URL |
Approach |
Main publication |
License |
P |
R |
F |
|
Woliński, M. Morfeusz — a practical tool for the morphological analysis of Polish. In M.A. Kłopotek, S.T. Wierzchoń, K. Trojanowski (eds.) Proceedings of the International IIS:IIPWM 2006 Conference, pp. 511–520, Wisła, Poland, 2006. |
2-clause BSD |
% |
% |
% |
|
|
Miłkowski M. Developing an open-source, rule-based proofreading tool. Software: Practice and Experience, 40(7):543–566, 2010. |
|
% |
% |
% |
|
dictionary-based rules and heuristics |
Kobyliński Ł. (unpublished) |
GPL |
% |
% |
% |
Multi-word lemmatization
System name and URL |
Approach |
Main publication |
License |
Accuracy |
– |
rule-based |
Degórski, Ł. Towards the lemmatisation of Polish nominal syntactic groups using a shallow grammar. In P. Bouvry, M.A. Kłopotek, F. Leprevost, M. Marciniak, A. Mykowiecka, H. Rybiński (eds.) Security and Intelligent Information Systems, Lecture Notes in Computer Science vol. 7053, pp. 370–378, Springer-Verlag Berlin Heidelberg, 2012. |
? |
82.90% |
– |
automatic generation of lemmatization rules using CRF |
Radziszewski A. Learning to lemmatise Polish noun phrases. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), Volume 1: Long Papers. ACL, pp. 701–709. |
GPL |
80.70% |
– |
automatic generation of lemmatization rules from a corpus |
Abramowicz W., Filipowska A., Małyszko J., Wagner T. Lemmatization of Multi-Word Entity Names for Polish Language Using Rules Automatically Generated Based on the Corpus Analysis. In Z. Vetulani, J. Mariani (eds.) Human Language Technologies as a Challenge for Computer Science and Linguistics, Fundacja Uniwersytetu im. A. Mickiewicza, pp. 540–544, Poznań 2015. |
? |
82.10% |
dictionary-based rules and heuristics |
Marcińczuk M. (in print) |
GPL |
97.99% |
Disambiguated POS tagging
The comparisons are performed using plain text as input and reporting the accuracy lower bound (Acclower) metric proposed by Radziszewski and Acedański (2012). The metric penalizes all segmentation changes in regard to the gold standard and treats such tokens as misclassified. Furthermore, we report separate metric values for both known and unknown words to assess the performance of guesser modules built into the taggers. These are indicated as AccKlower for known and AccUlower for unknown words.
The experiments have been performed on the manually annotated part of the National Corpus of Polish v. 1.1 (1M tokens). The ten-fold cross-validation procedure has been followed, by re-evaluating the methods ten times, each time selecting one of ten parts of the corpus for testing and the remaining parts for training the taggers. The provided results are averages calculated over ten training and testing sequences. Each of the taggers and each tagger ensemble has been trained and tested on the same set of cross-validation folds, so the results are directly comparable. Each of the training folds has been reanalyzed, according to the procedure described in (A Tiered CRF Tagger for Polish, using the Maca toolkit (Radziszewski and Śniatowski 2011). The idea of a morphological reanalysis of the gold-standard data is to allow the trained tagger to see similar input that is expected in the tagging phase. The training data is firstly turned into plain text and analyzed using the same mechanism that will be used by the tagger during actual tagging process. The output of the analyzer is then synchronized with the original gold-standard data, by using the original tokenization. Tokens with changed segmentation are taken from the gold-standard intact. In the case of tokens for which the segmentation did not change in the process of morphological analysis, the produced interpretations are compared with the original. A token is marked as an unknown word if the correct interpretation has not been produced by the analyzer. Maca has been run with the morfeusz-nkjp-official configuration, which uses Morfeusz SGJP analyzer (Woliński 2006) and no guesser module.
Tagger efficiency was compared by measuring training and tagging times of each of the methods on the same machine. 1.1M token set was used both for training and tagging stages. The total processing time included model loading/saving time and other I/O operations (e.g. reading/writing the tokens).
System name and URL |
Approach |
Main publication |
License |
Acclower |
AccKlower |
AccUlower |
Training time |
Tagging time |
MaxEnt model |
Kobyliński Ł., Kieraś W. Part of Speech Tagging for Polish: State of the Art and Future Perspectives. In Proceedings of the 17th International Conference on Intelligent Text Processing and Computational Linguistics (CICLing 2016), Konya, Turkey, 2016. |
GPL |
87.24% |
88.02% |
62.05% |
11095 s |
362 s |
|
rule-based adapted Brill tagger |
Acedański S. A Morphosyntactic Brill Tagger for Inflectional Languages. In H. Loftsson, E. Rögnvaldsson, S. Helgadóttir (eds.) Advances in Natural Language Processing, LNCS 6233, pp. 3–14, Springer, 2010. |
GPL 3 |
88.95% |
91.22% |
15.19% |
2624 s |
186 s |
|
memory-based |
Radziszewski A., Śniatowski T. A memory-based tagger for Polish. In: Z. Vetulani (ed.) Proceedings of the 5th Language and Technology Conference (LTC 2011), pp. 556–560, Poznań, Poland. |
|
90.33% |
91.26% |
60.25% |
548 s |
4338 s |
|
tiered, CRF-based |
Radziszewski A. A Tiered CRF Tagger for Polish. In R. Bembenik, Ł. Skonieczny, H. Rybiński, M. Kryszkiewicz, M. Niezgódka (eds.) Intelligent Tools for Building a Scientific Information Platform, pp. 215–230, Springer Berlin Heidelberg, 2013. |
LGPL 3.0 |
90.76% |
91.92% |
53.18% |
27242 s |
420 s |
|
mutually dependent CRF layers |
Waszczuk J. Harnessing the CRF complexity with domain-specific constraints. The case of morphosyntactic tagging of a highly inflected language. In Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012), pp. 2789–2804, Mumbai, India, 2012. |
2-clause BSD |
91.07% |
92.06% |
58.81% |
26675 s |
403 s |
|
voting ensemble |
Kobyliński Ł. PoliTa: A multitagger for Polish. In N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, S. Piperidis (eds.) Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014), pp. 2949–2954, Reykjavík, Iceland, ELRA, 2014. |
GPL |
92.01% |
92.91% |
62.81% |
N/A |
N/A |
Dependency parsing
10-fold cross-validation was used to test the quality of MaltParser and MateParser on Polish data (average LAS and UAS scores of 10 tests are provided).
System name and URL |
Approach |
Main publication |
License |
LAS |
UAS |
trained on the extended version of the Polish dependency treebank with MaltParser |
Wróblewska A. Polish Dependency Parser Trained on an Automatically Induced Dependency Bank. PhD dissertation, Institute of Computer Science, Polish Academy of Sciences, Warsaw, 2014. |
GPL 3 |
84% |
89% |
|
trained on the same data with MateParser |
GPL 3 |
89% |
93% |
||
|
|
|
% |
% |
Shallow parsing
System name and URL |
Approach |
Main publication |
License |
P |
R |
F |
rule-based |
Buczyński A., Przepiórkowski A. Spejd: A shallow processing and morphological disambiguation tool. In Z. Vetulani, H. Uszkoreit (eds.) Human Language Technology: Challenges of the Information Society, LNCS 5603, pp. 131–141. Springer-Verlag, Berlin, 2009. |
GPL 3 |
% |
% |
% |
Deep parsing
System name and URL |
Approach |
Main publication |
License |
P |
R |
F |
|
|
|
% |
% |
% |
|
|
|
GPL 3 (grammar) |
% |
% |
% |
|
|
|
GPL 3 |
% |
% |
% |
Word sense disambiguation
Manually annotated subcorpus of the National Corpus of Polish with 3889 texts and 1217822 segments was used for training and test (cross-validation); it contained 34186 occurrences of multi-sense words (1–7 senses). Simple heuristics selecting the most frequent sense resulted in 78.3% accuracy. The table presents results of leave-one-out evaluation with individual selection of each ambiguous words (as described in the publication).
System name and URL |
Approach |
Main publication |
License |
Accuracy |
Machine-learning |
Kopeć M., Młodzki R., Przepiórkowski A. Automatyczne znakowanie sensami słów. In A. Przepiórkowski, M. Bańko, R.L. Górski, B. Lewandowska-Tomaszczyk (eds.) Narodowy Korpus Języka Polskiego, pp. 209–224. Wydawnictwo Naukowe PWN, Warsaw, 2012. |
GPL 3 |
91.46% |
|
BabelNet-powered |
? |
? |
? |
Named entity recognition
System name and URL |
Approach |
Main publication |
License |
P |
R |
F |
linear-chain CRF |
Savary A., Waszczuk J., Przepiórkowski A. Towards the annotation of named entities in the National Corpus of Polish. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010), pp. 3622–3629, Valletta, Malta, ELRA, 2010. |
2-clause BSD |
83% |
78% |
79% |
|
Liner2 with TIMEX model |
|
Kocoń J., Marcińczuk M. Liner2.5 model Timex, CLARIN-PL digital repository, 2016. |
|
% |
% |
% |
Sentiment analysis
System name and URL |
Approach |
Main publication |
License |
P |
R |
F |
|
|
|
% |
% |
% |
Mention detection
Precision, recall and F-measure are calculated on Polish Coreference Corpus data with two alternative mention detection scores:
- EXACT: score of exact boundary matches (an automatic and a manual mention match if they have exactly the same boundaries; in other words, they consist of the same tokens)
- HEAD: score of head matches (we reduce the automatic and the manual mentions to their single head tokens and compare them).
System name and URL |
Approach |
Main publication |
License |
EXACT |
HEAD |
||||
P |
R |
F |
P |
R |
F |
||||
Collects mention candidates from available sources – morphosyntactical, shallow parsing, named entity and/or zero anaphora detection tools |
Ogrodniczuk M., Głowińska K., Kopeć M., Savary A., Zawisławska M. Coreference in Polish: Annotation, Resolution and Evaluation, chapter 10.6. Walter De Gruyter, 2015. |
CC BY 3 |
66.79% |
67.21% |
67.00% |
88.29% |
89.41% |
88.85% |
|
Coreference resolution
As there is still no consensus about the single best coreference resolution metrics, CoNLL measure is used (average of MUC, B3 and CEAFE F-measure values). For end-to-end systems CoNLL-2011 shared task-based approach is used, so two result calculation strategies are presented:
- INTERSECT: consider only correct system mentions (i.e. the intersection between gold and system mentions)
- TRANSFORM: unify system and gold mention sets using the following procedure for twinless mentions (without a corresponding mention in the second set):
- insert twinless gold mentions into system mention set as singletons
- remove twinless singleton system mentions
- insert twinless non-singletion system mentions into gold set as singletons.
The results are produced on Polish Coreference Corpus data.
System name and URL |
Approach |
Main publication |
License |
GOLD |
EXACT INTERSECT |
EXACT TRANSFORM |
HEAD INTERSECT |
HEAD TRANSFORM |
Rule-based |
Ogrodniczuk M., Kopeć M. End-to-end coreference resolution baseline system for Polish. In Z. Vetulani (ed.), Proceedings of the 5th Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, pp. 167–171, Poznań, Poland, 2011. |
CC BY 3 |
73.40% |
78.54% |
66.55% |
76.27% |
70.11% |
|
Statistical |
Kopeć M., Ogrodniczuk M. Creating a Coreference Resolution System for Polish. In Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012, pp. 192–195, ELRA. |
CC BY 3 |
78.41% |
80.86% |
68.96% |
78.58% |
72.15% |
|
Sieve-based |
Nitoń B., Ogrodniczuk M. Multi-Pass Sieve Coreference Resolution System for Polish. In J. Gracia, F. Bond, J. P. McCrae, P. Buitelaar, Ch. Chiarcos, S. Hellmann (eds.) Proceedings of the 1st Conference on Language, Data and Knowledge (LDK 2017), Galway, Ireland, 19–20 June 2017. |
CC BY 3 |
80.47% |
? |
? |
? |
? |
Extractive summarization
The table presents results of evaluation on the Polish Summaries Corpus (abstractive summaries, 20% length of the original document). ROUGE covers several metrics, with the following variants used here:
- ROUGE n which counts n-gram co-occurrences between reference (gold) summaries and system summary
- ROUGE Mn with ROUGE score calculated for each text using a single manual summary which gives the highest score as a reference.
System name and URL |
Approach |
Main publication |
License |
ROUGE 1 |
ROUGE 2 |
ROUGE 3 |
ROUGE M1 |
ROUGE M2 |
ROUGE M3 |
? |
Ciura M., Grund D., Kulików S., Suszczańska N. A System to Adapt Techniques of Text Summarizing to Polish. In Okatan A. (ed.) International Conference on Computational Intelligence, pp. 117–120, Istanbul, Turkey. International Computational Intelligence Society, 2004. |
? |
? % |
? % |
? % |
? % |
? % |
? % |
|
sentence extraction relying on one of: positional heuristics, word frequency features, lexical chains information |
Dudczak A. Zastosowanie wybranych metod eksploracji danych do tworzenia streszczeń tekstów prasowych dla języka polskiego. MSc thesis, Poznań Technical University, 2007. |
3-clause BSD |
55.4% |
20.8% |
14.4% |
62.9% |
33.3% |
27.4% |
|
sentence extraction with machine learning |
Świetlicka J. Metody maszynowego uczenia w automatycznym streszczaniu tekstów. MSc thesis, Warsaw University 2010. |
GPL 3 |
58.0% |
22.6% |
16.1% |
65.4% |
35.8% |
29.8% |
|
word frequency based sentence extraction |
Rotem N. Open Text Summarizer. 2003. |
GPL? |
51.3% |
13.6% |
9.0% |
58.5% |
22.5% |
17.9% |
|
Emily-C |
? |
? |
? |
52.9% |
15.1% |
10.0% |
58.8% |
24.2% |
19.2% |
Emily-S |
? |
? |
? |
53.0% |
15.4% |
10.4% |
59.4% |
24.7% |
20.1% |
? |
? |
CC BY 3 |
58.1% |
23.8% |
17.1% |
66.4% |
38.4% |
32.7% |