Evaluating machine translation in a low-resource language combina- tion: Spanish-Galician María do Campo Bayón Pilar Sánchez-Gijón Grup Tradumàtica Grup Tradumàtica Universitat Autònoma de Barcelona Dept. de Traducció i d’Interpretació i maria.docampo@e-campus.uab.cat d’Estudis de l’Àsia Oriental Universitat Autònoma de Barcelona pilar.sanchez.gijon@uab.cat more adequate in the context of a minoritized Abstract language such as Galician in an MT+Post-editing (PE) workflow. For that purpose, the quality of all This paper reports the results of a study three raw outputs was established with the designed to assess the perception of ade- following metrics: quacy of three different types of machine translation systems within the context of a • Evaluating which type of MT system ob- minoritized language combination (Span- tains better results applying the BLEU ish-Galician). To perform this evaluation, metric. a mixed design with three different met- • Evaluating which type of MT system ob- rics (BLEU, survey and error analysis) is tains better results in a human evaluation used to extract quantitative and qualitative (quality perception survey conducted data about two marketing letters from the among professional post-editors). energy industry translated with a rule- • Evaluating which type of MT system ob- based system (RBMT), a phrase-based tains better results following an error system (PBMT) and a neural system analysis framework (MQM). (NMT). Results show that in the case of low-resource languages rule-based and phrase-based machine translations sys- 3 Background tems still play an important role. 3.1 NMT Evaluation 1 Introduction With the outbreak of NMT, many studies have tried to shed some light on the real and the per- In the last couple of years, Neural Machine ceived quality of this kind of MT systems. Shter- Translation is gaining more attention in the ionov et al. (2018) show that a few translators see translation industry and becoming more popular NMT as a booster of their productivity. Some thanks to the considerably good results obtained translators even see (N)MT as a handicap for their in certain language combinations. Nevertheless, productivity while others perceive it the other way low-resource languages and minoritized around (Sánchez-Gijón et al., 2019). In terms of languages represent some challenges for machine NMT quality perception, Castilho et al. (2017) translation (MT) usage and training. This paper conclude that raw NMT segments may not be pre- describes the process followed to test and evaluate ferred by translators. In the same paper, they con- three different MT systems in a closely related cluded that, compared to PBMT, NMT represents language combination such as Spanish-Galician. a step forward but it implies also some limitations. The same idea of strengths and weaknesses on 2 Aim of this study NMT with respect to PBMT can be found in Po- povic, 2017. Most of these studies describe tests This study aims to determine which type of MT involving language combinations of high-re- system (RBMT, PBMT or NMT) is perceived as source languages. This paper approaches this ________________________ © 2019 The authors. This article is licensed under a Crea- tive Commons 4.0 licence, no derivative works, attribution, CCBY-ND. Proceedings of MT Summit XVII, volume 2 Dublin, Aug. 19-23, 2019 | p. 30 topic from the perspective of a low-resource lan- Once all three MT systems in the language guage: Galician. combination Spanish-Galician were available, a set of 32 Spanish segments was translated with 3.2 MT in Galician each of them. The quality of the raw MT segments As a minoritized language, Galician represents a obtained was measured in the next phase of the serious challenge to develop MT systems due to investigation following different approaches. the lack of technological and data resources. In re- The second phase of the investigation involved cent years, there has been an enormous effort, the evaluation of the quantitative data results ob- mainly from the academic community, to develop tained applying the automatic metric Bilingual Natural Language Processing (NLP) resources Evaluation Understudy, abbreviated as BLEU and compile corpora such as GalNet, the Galician (Papineri et al., 2001). Then, a survey was de- WordNet (Gomez Guinovart & Solla Portela, signed to compile qualitative information about 2017), SemCor (Solla Portela & Gomez Guino- the quality perception of Spanish-Galician post- vart, 2017), several terminology projects (Solla editors. For that purpose, a sample of 14 segments Portela & Gomez Guinovart, 2015), big corpus from the whole set was used. 69 professional annotation (Gomez Guinovart & Lopez Fernán- translators with experience in Spanish-Galician dez, 2009), Freeling (Padro & Stanilovsky, 2012) post-editing were selected from the CPSL Lan- and Linguakit (Gamallo & Garcia, 2017). guage Solutions resource database and Proz por- There are also some MT systems specifically tal. Finally, 15 people participated in that survey. created for Galician: the RBTA MT system of the To complete the qualitative results, an error anal- Centro Ramón Piñeiro para a Investigación en ysis was performed following the MQM frame- Humanidades (Diz Gamallo, 2001), OpenTrad work. Once all the individual results were ana- Apertium (Armentano-Oller & Forcada, 2006) lysed, a global evaluation was performed to trian- and Carvalho PBMT system (Pichel Campos et gulate the resulting data. al., 2009). Nevertheless, the need to keep investigating in 5 Results NLP and Deep Learning (DL) in Galician is very 5.1 Automatic evaluation clear in order to develop the corpora and the strat- egies needed to train phrase-based and neural sys- The BLEU score on the whole set of segments is tems and obtain better results (Agerri et al., 2018: as follows: 2322). 4 Methodology The investigation is divided into three different phases. The first one consisted of choosing the source document to be processed by the three different MT systems. Two marketing letters of approximately 500 words in total with specific terminology from the energy industry were chosen. After that, RBMT (OpenTrad Apertium) and PBMT (ModernMT v. 2.5) systems were created and trained. In the case of Apertium, the stable version of the pair Spanish-Galician was downloaded into an Ubuntu environment and trained with specific terminology of the source field. Similarly, a new engine was created in MMT v. 2.5 and trained with a thematic translation memory (TM) of 4315 translation units and a Figure 1. BLEU Score parallel corpus of 6 million words from the legal RBMT and PBMT segments show higher and administrative field. Lastly, regarding NMT scores than NMT. There is not significant differ- system, due to the lack of enough high-quality ence between RBMT and PBMT scores, but dif- training data, we selected Google Neural Spanish- ferences are significant between these two sys- Galician engine to perform the texts. tems and NMT: Proceedings of MT Summit XVII, volume 2 Dublin, Aug. 19-23, 2019 | p. 31 5.2.1 Global evaluation Human evaluators were asked to answer 2 questions. In each question, they had to rank the three different raw machine translations as 1st, 2nd and 3rd place. Then, they had to specify if they Figure 2. p-values per pairs would use or not the machine translation to post- edit (binary response). In relation with BLEU Finally, 14 of the source segments contains scores, the results of usable/not usable segments more than 30 words. These segments were identi- show that RBMT and PBMT would be used to be fied as long segments and analysed separately. post-edited. This is the BLUE score obtained in the subset of 14 long segments analysed by post-editors: Usable vs. Not Usable segments NMT PBMT RBMT 0 50 100 150 200 250 Usable Not usable Figure 5. Segments usable vs. not usable to post- edit To analyse this data, the non-parametric statis- tical test Cochran ‘s Q test is applied. Figure 3. BLEU Score of long segments RBMT segments show higher and more homo- geneous scores than NMT and PBMT. Differences between NMT and both PBMT and RBMT are significant. Differences between PBMT and RBMT are not significant: Figure 4. p-values per pairs in long segments Figure 6. Cochran’s Q test results 5.2 Human evaluation Differences are significant (p-value < 0.0001). Proportions among the three groups are statisti- The human evaluation was designed to gather two cally significant (Marascuilo procedure): different pieces of information segment by segment: ranking of MT system and which MT system is considered good enough to be post- edited. 14 translated segments, one by each MT system, were selected as sample. Equal translation Figure 7. Marascuilo procedure results results from different MT systems or too bad translations were excluded from the survey in And the proportions show that the three groups order not to distort the survey results. are different: Proceedings of MT Summit XVII, volume 2 Dublin, Aug. 19-23, 2019 | p. 32 Figure 8. Proportions of MT systems Regarding the ranking, these are the results from each segment. As Figure 9 shows, RBMT and PBMT are better positioned that NMT. Post- editors also agreed that all segments selected as 1st place would be used to post-edit. Figure 11. Cochran’s Q test results in long seg- ments Differences are significant (p-value = 0.001). Proportions are statistically significant, but not among all three groups (Marascuilo procedure): And the proportions show that there are differ- ences between NMT and the other two MT sys- tems: Figure 9. Ranking of MT systems Figure 12. Marascuilo procedure results in long To establish whether these differences among segments MT systems are significant, the Kruskal-Wallis test was applied. Comparing results per pairs, in all cases p-value was under 0.05, meaning that dif- ferences are significant. 5.2.2 Long segments Figure 13. Proportions in long segments Four of the source segments contains more than 30 words. These segments were identified as long Usable scenario in long segments differs from segments and analysed separately. the whole document. Figure 14 shows which seg- ment from each MT system would be chosen to be post-edited in 1st, 2nd and 3rd place. Figure 10. Usable vs. not usable in long segments To analyse this data, the non-parametric statis- tical test Cochran ‘s Q test is applied. Figure 14. Ranking of MT systems in long seg- ments Proceedings of MT Summit XVII, volume 2 Dublin, Aug. 19-23, 2019 | p. 33 To establish whether these differences among MT systems are significant, the Kruskal-Wallis Error Types test was applied. Comparing results per pairs, the p-value was under 0.05 only between NMT and 12 the other to MT systems: 10 8 RBMT PBMT NMT 6 RBMT No Yes 4 PBMT No Yes 2 NMT Yes Yes 0 Incinsistent with… Mistranslation Typography Awkward Untranslated Addition Omission Spelling Agreement Tense-mood-aspect Function-words Register Unidiomatic Word-order Part-of-speech Figure 15. Statistical differences in long segments 5.3 Error analysis A Multidimensional Quality Metrics (MQM) customized framework was used to identify the errors made by each MT system. Only relevant types of errors from accuracy, fluency, style and RBMT PBMT NMT terminology were selected. Figure 16 shows the total number of errors obtained per segment in Figure 17. Error types each MT system: Finally, a repetitive error in NMT is the change Error Analysis of register. In this phrase, not only the verbal form is incorrect but also changes from the second per- 300 son plural to the singular even if it is the same in Error percentage / the source text: segment (%) 200 Spanish: Instala ahora el gas y disfruta de todas 100 sus ventajas […] en todos los rincones de tu casa Galician: Agora instálalle [incorrect verbal 0 Whole document Long segments construction] o gas e goce [second person plural] de todas as súas [second person plural] […] en to- RBMT PBMT NMT dos os recunchos da túa [second person singular] casa. Figure 16. Error percentage per segment 6 Conclusions Regarding the type of errors, there’s a clear The main conclusion is that although NMT seems prevalence in all MT systems of mistranslations, promising in frequent language combinations, gender and number agreement errors, function- especially if English is involved, it is not obtaining words errors, word-order errors and unidiomatic the desired results in low-resource languages such expressions. Also, PBMT and NMT made more as the pair Spanish-Galician. NMT has not yet addition, omission, orthography, typography and unseated RBMT and PBMT, performing, in fact, part-of-speech errors, and domain terminology in- worse than these systems. consistencies. RBMT and NMT registered verb This small study reveals that more tests should concordance errors and awkward constructions. be done to replicate results and evaluate special Finally, the only system with register errors was needs to have a competitive NMT. Subsequent in- NMT. vestigations must consolidate error patterns of The clearest example of error in RBMT is the each system to address some of the most promi- wrong identification of the preposition para (for, nent issues. Thus, there’s a clear need to work in in English) and the undefined feminine form of the access to the resources and parallel data the article: una (a in English). RBMT interprets needed to train MT systems, especially in PBMT these words as verbs so they are translated as and NMT. parar (stop) and unir (join). PBMT sometimes Consequently, any future lines of investigation makes errors in verbal constructions such as the in MT and minoritized languages should be fo- wrong translation of hemos venido by comezamos cused on searching and optimizing NLP and text viñesen (we started coming instead of we have resources. come). Proceedings of MT Summit XVII, volume 2 Dublin, Aug. 19-23, 2019 | p. 34 Acknowledgement transferencia. IV Jornadas en Tecnología del Habla, November 8-10 of 2006, Zaragoza. pp. 111-116. This work was supported by the ProjecTA-U project “Machine Translation Users”, grant Padró, L. & Stanilovsky, E. 2012. Freeling 3.0: Towards wider multilinguality. In Nicoletta number FFI2016-78612-R [MINECO / FEDER, Calzolari et al. (Eds.). Proceedings of the Eighth UE]. International Conference on Language Resources and Evaluation (LREC-2012), pp. 2473–2479, References Istanbul, Turkey. Agerri, R., Gómez Guinovart, X., Rigau, G. & Solla Papineni, K., Roukos, S., Ward, T. & Zhu, W. 2001. Portela, M. A. 2018. Developing New Linguistic BLEU: A Method for Automatic Evaluation of Resources and Tools for the Galician Language. Machine Translation, IBM Research Report Proceedings of the 11th Language Resources and RC22176 (W0109−022). Evaluation Conference (LREC'18): 2322-2325. Popović, M. 2017. Comparing Language Related Armentano-Oller, C. & Forcada, M. L. 2006. Open- Issues for NMT and PBMT between German and source machine translation between small English. The Prague Bulletin of Mathematical languages: Catalan and Aranese Occitan. Strategies Linguistics, 108(1): 209-220. for developing machine translation for minority languages (5th SALTMIL workshop on Minority Pichel Campos, J. R., Malvar Fernández, P., Senra Languages). May 22-28, p. 51-54. Gómez, O., Gamallo Otero, P. & García González, A. 2009. Carvalho: English-Galician SMT system Armentano-Oller, C., Carrasco, R. C., Corbí-Bellot, A. from EuroParl English-Portuguese parallel corpus. M., Forcada, M. L., Ginestí-Rosell, M., Ortiz-Rojas, Procesamiento del Lenguaje Natural, 23: 379-381. S. et al. 2006. Open-source Portuguese-Spanish machine translation. In Computational Processing Sánchez-Gijón, P., Moorkens, J., & Way, A. (2019). of the Portuguese Language: 7th Workshop on Post-editing neural machine translation versus Computational Processing of Written and Spoken translation memory segments. Machine Translation, Portuguese, PROPOR. Lecture Notes in Artificial 31-59. . Burchardt, A. & Lommel, A. 2014. Practical Shterionov, D., Superbo, R., Nagle, P., Casanellas, L., Guidelines for the Use of MQM in Scientific O’Dowd, T. & Way, A. (2018). Human versus Research on Translation Quality. Available at automatic quality evaluation of NMT and PBSMT. . . Castilho, S., Moorkens, J., Gaspari, F., Calixto, I., Solla Portela, M. A. & Gómez Guinovart, X. 2015. Tinsley, J., & Way, A. 2017. Is Neural Machine Termonet: Construcción de terminologías a partir de Translation the New State of the Art? In The Prague WordNet y corpus especializados. Procesamiento Bulletin of Mathematical Linguistics, 108:1, 109– del Lenguaje Natural, 55:165–168. 120. . Solla Portela, M. A. & Gómez Guinovart, X. 2016. Diz Gamallo, I. 2001. The importance of MT for the Dbpedia del gallego: recursos y aplicaciones en survival of minority languages: Spanish-Galician procesamiento del lenguaje. Procesamiento del MT system. Proceedings of MT Summit VIII, Lenguaje Natural, 57:139–142. November 2001, Spain. Solla Portela, M. A. & Gómez Guinovart, X. 2017. Gamallo, P. & Garcia, M. 2017. Linguakit: a Diseño y elaboración del corpus SemCor del gallego multilingual tool for linguistic analysis and anotado semánticamente con wordnet 3.0. information extraction. Linguamática, 9(1): 19–28. Procesamiento del Lenguaje Natural, 59:137–140. Gómez Guinovart, X. & López Fernández, S. 2009. Anotación morfosintáctica do Corpus Técnico do Galego. Linguamática, 1(1): 61–71. Gómez Guinovart, X. & Solla Portela, M. A. 2017. Building the galician wordnet: methods and applications. Language Resources and Evaluation, 52 (1): 317–339. Iglesias, G., Rodríguez Liñares, L., Rodríguez Banga, E., Campillo Díaz, F. L. & Méndez Pazó, F. 2010. Perspectivas de la traducción automática castellano- gallego mediante técnicas estadísticas y por Proceedings of MT Summit XVII, volume 2 Dublin, Aug. 19-23, 2019 | p. 35