Japanese Wikification Corpora

Last Update: June 14, 2017.

(UNDER CONSTRUCTION)

Joint work with Assoc. Prof. Shinsuke Mori from the Academic Center for Computing and Media Studies, Kyoto University

Annotation for wikification. — An example of wikification. (a) Mappings from mentions to entities. (b) Wikitext as in Wikipedia. (c) Our annotation.

Features

Any topics, not just named entities.
Exhaustive annotation, not just "important" concepts.
No NIL detection (all entities have corresponding Wikipedia articles).

Download

Annocated Corpora

COMING SOON.

A subset of BCCWJ (OW and OY) will be able to be downloaded from NINJAL.

A tweet corpus will be available upon request.

Wikipedia Snapshot

jawiki-20150512-pages-meta-current.xml.bz2 (our mirror).

We chose an XML dump of Japanese Wikipedia dated 12 May 2015. For the sake of reproducibility, we decided to stick with this snapshot. Since this snapshot is no longer available in Wikimedia, we provide our copy here.

Reference(s)

Yugo Murawaki and Shinsuke Mori.
Wikification for Scriptio Continua.
In Proceedings of the 10th Edition of its Language Resources and Evaluation Conference (LREC 2016), pp. 1346-1351, Portorož, Slovenia, May 2016. (to appear).
[paper] [errata] [poster]