🚀 Submit
Participants are required to prepare their submissions as *.zip files, following the same structure as shown in the Sample submission.
All systems are required to produce valid output in the CoNLL-U format. Each submitted prediction file should adhere to the CoNLL-u format and have the corresponding .conllu extension. Additionally, each prediction file should inherit its name from the corresponding text file. For example, if the input text file is named test.txt, then the prediction file should be named test.conllu.
Please, name the <dataset_name> directories analogously to how they appear in the dataset.zip.
The directory structure should be:
/<TAGSET_NAME>/<DATASET_NAME>/test.conllu
in this case, for predictions for ud tagset, it would look like this
ud:
pdb:
test.conllu
nkjp1m_by_type:
test.conllu
pdb3:
test.conllu
and for nkjp tagset it should look like this:
nkjp:
nkjp1m_by_type:
test.conllu
For each tagset, you need to provide prediction files for all datasets, otherwise, the submission will be rejected.
During the evaluation of NKJP-based predictions, the metrics using dependency parsing features (HEAD + DEPREL) will not be calculated due to the absence of these syntactic features in gold standard data. However, it is necessary to include HEAD + DEPREL information in the NKJP-based prediction files to maintain the correct CoNLL-u format. If a dependency parsing module predicting these features is unavailable, tokens should be assigned consecutive natural numbers as the HEAD value, starting from 0. The first token should be assigned root as the DEPREL value, and other tokens should be assigned an underscore (or dep).
You can download a sample submission file for UD tagset below:
When submitting, please provide:
- The name and URL of the benchmarked model.
- Information about the embeddings used for initialisation during training. If no embeddings were used, please denote this with a hyphen (-).
After uploading the prediction, you will see the results of your model. From there, you can choose to publish them on the leaderboard or withdraw your entry.
Please note that for the final evaluation, results are calculated across several datasets, which may take a while for the page to refresh.