|
Size: 2537
Comment:
|
Size: 3492
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 6: | Line 6: |
: Morfeusz, Concraft/WCRFT, Spejd, Dependency Parser, TIMEX/Nerf, |
|
| Line 39: | Line 44: |
| [[http://clip.ipipan.waw.pl/PCC|Polish Coreference Corpus]] | As there is still no consensus about the single best coreference resolution metrics, CoNLL measure is used (average of MUC, B3 and CEAFE F-measure values). For end-to-end systems CoNLL-2011 shared task-based approach is used, so two result calculation strategies are presented: * INTERSECT: consider only correct system mentions (i.e. the intersection between gold and system mentions) * TRANSFORM: unify system and gold mention sets using the following procedure for twinless mentions (without a corresponding mention in the second set): 1. insert twinless gold mentions into system mention set as singletons 1. remove twinless singleton system mentions 1. insert twinless non-singletion system mentions into gold set as singletons. |
| Line 41: | Line 51: |
| || '''System name''' || '''Short description''' || '''Main publication''' || '''License''' || '''P''' || '''R''' || '''F''' || || [[http://zil.ipipan.waw.pl/Ruler|Ruler]] || Rule-based || Ogrodniczuk M., Kopeć M. 'End-to-end coreference resolution baseline system for Polish'. In Z. Vetulani (ed.), Proceedings of the 5th Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, pp. 167–171, Poznań, Poland, 2011. || CC BY 3 || || || || || [[http://zil.ipipan.waw.pl/Bartek|Bartek]] || Statistical || Kopeć M., Ogrodniczuk M. 'Creating a Coreference Resolution System for Polish'. In Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012, pp. 192–195, ELRA. || CC BY 3 || || || || |
The results are produced on [[http://clip.ipipan.waw.pl/PCC|Polish Coreference Corpus]] data. || '''System name''' || '''Short description''' || '''Main publication''' || '''License''' || '''Gold''' || '''EXACT INTERSECT''' || '''EXACT TRANSFORM''' || '''HEAD INTERSECT''' || '''HEAD TRANSFORM''' || || [[http://zil.ipipan.waw.pl/Ruler|Ruler]] || Rule-based || Ogrodniczuk M., Kopeć M. 'End-to-end coreference resolution baseline system for Polish'. In Z. Vetulani (ed.), Proceedings of the 5th Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, pp. 167–171, Poznań, Poland, 2011. || CC BY 3 || || || || || || || [[http://zil.ipipan.waw.pl/Bartek|Bartek]] || Statistical || Kopeć M., Ogrodniczuk M. 'Creating a Coreference Resolution System for Polish'. In Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012, pp. 192–195, ELRA. || CC BY 3 || || || || || || |
Benchmarks
This page documents performance of various NLP systems for Polish.
Morphological analysis
- : Morfeusz, Concraft/WCRFT, Spejd, Dependency Parser, TIMEX/Nerf,
POS tagging
Shallow parsing
Dependency parsing
Deep parsing
Word sense disambiguation
Named entity recognition
Sentiment analysis
Mention detection
Precision, recall and F-measure are calculated on Polish Coreference Corpus data with two alternative mention detection scores:
- EXACT: score of exact boundary matches (an automatic and a manual mention match if they have exactly the same boundaries; in other words, they consist of the same tokens)
- HEAD: score of head matches (we reduce the automatic and the manual mentions to their single head tokens and compare them).
System name |
Short description |
Main publication |
License |
EXACT |
HEAD |
||||
P |
R |
F |
P |
R |
F |
||||
Collects mention candidates from available sources – morphosyntactical, shallow parsing, named entity and/or zero anaphora detection tools |
Ogrodniczuk M., Głowińska K., Kopeć M., Savary A., Zawisławska M. Coreference in Polish: Annotation, Resolution and Evaluation, chapter 10.6. Walter De Gruyter, 2015. |
CC BY 3 |
66.79% |
67.21% |
67.00% |
88.29% |
89.41% |
88.85% |
|
Coreference resolution
As there is still no consensus about the single best coreference resolution metrics, CoNLL measure is used (average of MUC, B3 and CEAFE F-measure values). For end-to-end systems CoNLL-2011 shared task-based approach is used, so two result calculation strategies are presented:
- INTERSECT: consider only correct system mentions (i.e. the intersection between gold and system mentions)
- TRANSFORM: unify system and gold mention sets using the following procedure for twinless mentions (without a corresponding mention in the second set):
- insert twinless gold mentions into system mention set as singletons
- remove twinless singleton system mentions
- insert twinless non-singletion system mentions into gold set as singletons.
The results are produced on Polish Coreference Corpus data.
System name |
Short description |
Main publication |
License |
Gold |
EXACT INTERSECT |
EXACT TRANSFORM |
HEAD INTERSECT |
HEAD TRANSFORM |
Rule-based |
Ogrodniczuk M., Kopeć M. 'End-to-end coreference resolution baseline system for Polish'. In Z. Vetulani (ed.), Proceedings of the 5th Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, pp. 167–171, Poznań, Poland, 2011. |
CC BY 3 |
|
|
|
|
|
|
Statistical |
Kopeć M., Ogrodniczuk M. 'Creating a Coreference Resolution System for Polish'. In Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012, pp. 192–195, ELRA. |
CC BY 3 |
|
|
|
|
|