Automatic classification of human translation and machine translation: a study from the perspective of lexical diversity

Yingxue Fu; Mark Jan Nederhof

Automatic classification of human translation and machine translation: a study from the perspective of lexical diversity

Yingxue Fu, Mark Jan Nederhof

School of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

14 Downloads (Pure)

Abstract

By using a trigram model and fine-tuning a pretrained BERT model for sequence classification, we show that machine translation and human translation can be classified with an accuracy above chance level, which suggests that machine translation and human translation are different in a systematic way. The classification accuracy of machine translation is much higher than of human translation. We show that this may be explained by the difference in lexical diversity between machine translation and human translation. If machine translation has independent patterns from human translation, automatic metrics which measure the deviation of machine translation from human translation may conflate difference with quality. Our experiment with two different types of automatic metrics shows correlation with the result of the classification task. Therefore, we suggest the difference in lexical diversity between machine translation and human translation be given more attention in machine translation evaluation.

Original language	English
Title of host publication	Proceedings for the First Workshop on Modelling Translation
Subtitle of host publication	Translatology in the Digital Age
Editors	Yuri Bizzoni, Elke Teich, Cristina España-Bonet, Josef van Genabith
Publisher	Linkoping University Electronic Press
Pages	91–99
Publication status	Published - 31 May 2021
Event	Workshop on Modelling Translation: Translatology in the Digital Age - Online City, Iceland Duration: 31 May 2021 → 2 Jun 2021 Conference number: 1 https://easychair.org/cfp/MoTra21

Publication series

Name	NEALT Proceedings Series
Publisher	Linköping University Electronic Press
ISSN (Print)	1650-3686
ISSN (Electronic)	1650-3740

Workshop

Workshop	Workshop on Modelling Translation
Abbreviated title	MoTra21
Country/Territory	Iceland
City	Online City
Period	31/05/21 → 2/06/21
Internet address	https://easychair.org/cfp/MoTra21

Access to Document

Fu-2021-Automatic-classification-of-human-MOTRA-91-CCBY
Copyright © 2021 by the authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/.
Final published version, 136 KBLicence: CC BY

https://aclanthology.org/previews/ingest-nodalida/2021.motra-1.10/Licence: CC BY

Cite this

Fu, Y., & Nederhof, M. J. (2021). Automatic classification of human translation and machine translation: a study from the perspective of lexical diversity. In Y. Bizzoni, E. Teich, C. España-Bonet, & J. van Genabith (Eds.), Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age (pp. 91–99). (NEALT Proceedings Series). Linkoping University Electronic Press. https://aclanthology.org/previews/ingest-nodalida/2021.motra-1.10/

Fu, Yingxue ; Nederhof, Mark Jan. / Automatic classification of human translation and machine translation : a study from the perspective of lexical diversity. Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age. editor / Yuri Bizzoni ; Elke Teich ; Cristina España-Bonet ; Josef van Genabith. Linkoping University Electronic Press, 2021. pp. 91–99 (NEALT Proceedings Series).

@inproceedings{777591c5314d4ecbbe422d2ae1d834bc,

title = "Automatic classification of human translation and machine translation: a study from the perspective of lexical diversity",

abstract = "By using a trigram model and fine-tuning a pretrained BERT model for sequence classification, we show that machine translation and human translation can be classified with an accuracy above chance level, which suggests that machine translation and human translation are different in a systematic way. The classification accuracy of machine translation is much higher than of human translation. We show that this may be explained by the difference in lexical diversity between machine translation and human translation. If machine translation has independent patterns from human translation, automatic metrics which measure the deviation of machine translation from human translation may conflate difference with quality. Our experiment with two different types of automatic metrics shows correlation with the result of the classification task. Therefore, we suggest the difference in lexical diversity between machine translation and human translation be given more attention in machine translation evaluation.",

author = "Yingxue Fu and Nederhof, {Mark Jan}",

year = "2021",

month = may,

day = "31",

language = "English",

series = "NEALT Proceedings Series",

publisher = "Linkoping University Electronic Press",

pages = "91–99",

editor = "Yuri Bizzoni and Elke Teich and Cristina Espa{\~n}a-Bonet and {van Genabith}, Josef",

booktitle = "Proceedings for the First Workshop on Modelling Translation",

address = "Sweden",

note = "Workshop on Modelling Translation : Translatology in the Digital Age, MoTra21 ; Conference date: 31-05-2021 Through 02-06-2021",

url = "https://easychair.org/cfp/MoTra21",

}

Fu, Y & Nederhof, MJ 2021, Automatic classification of human translation and machine translation: a study from the perspective of lexical diversity. in Y Bizzoni, E Teich, C España-Bonet & J van Genabith (eds), Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age. NEALT Proceedings Series, Linkoping University Electronic Press, pp. 91–99, Workshop on Modelling Translation, Online City, Iceland, 31/05/21. <https://aclanthology.org/previews/ingest-nodalida/2021.motra-1.10/>

Automatic classification of human translation and machine translation: a study from the perspective of lexical diversity. / Fu, Yingxue; Nederhof, Mark Jan.
Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age. ed. / Yuri Bizzoni; Elke Teich; Cristina España-Bonet; Josef van Genabith. Linkoping University Electronic Press, 2021. p. 91–99 (NEALT Proceedings Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Automatic classification of human translation and machine translation

T2 - Workshop on Modelling Translation

AU - Fu, Yingxue

AU - Nederhof, Mark Jan

N1 - Conference code: 1

PY - 2021/5/31

Y1 - 2021/5/31

N2 - By using a trigram model and fine-tuning a pretrained BERT model for sequence classification, we show that machine translation and human translation can be classified with an accuracy above chance level, which suggests that machine translation and human translation are different in a systematic way. The classification accuracy of machine translation is much higher than of human translation. We show that this may be explained by the difference in lexical diversity between machine translation and human translation. If machine translation has independent patterns from human translation, automatic metrics which measure the deviation of machine translation from human translation may conflate difference with quality. Our experiment with two different types of automatic metrics shows correlation with the result of the classification task. Therefore, we suggest the difference in lexical diversity between machine translation and human translation be given more attention in machine translation evaluation.

AB - By using a trigram model and fine-tuning a pretrained BERT model for sequence classification, we show that machine translation and human translation can be classified with an accuracy above chance level, which suggests that machine translation and human translation are different in a systematic way. The classification accuracy of machine translation is much higher than of human translation. We show that this may be explained by the difference in lexical diversity between machine translation and human translation. If machine translation has independent patterns from human translation, automatic metrics which measure the deviation of machine translation from human translation may conflate difference with quality. Our experiment with two different types of automatic metrics shows correlation with the result of the classification task. Therefore, we suggest the difference in lexical diversity between machine translation and human translation be given more attention in machine translation evaluation.

M3 - Conference contribution

T3 - NEALT Proceedings Series

SP - 91

EP - 99

BT - Proceedings for the First Workshop on Modelling Translation

A2 - Bizzoni, Yuri

A2 - Teich, Elke

A2 - España-Bonet, Cristina

A2 - van Genabith, Josef

PB - Linkoping University Electronic Press

Y2 - 31 May 2021 through 2 June 2021

ER -

Fu Y, Nederhof MJ. Automatic classification of human translation and machine translation: a study from the perspective of lexical diversity. In Bizzoni Y, Teich E, España-Bonet C, van Genabith J, editors, Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age. Linkoping University Electronic Press. 2021. p. 91–99. (NEALT Proceedings Series).

Automatic classification of human translation and machine translation: a study from the perspective of lexical diversity

Abstract

Publication series

Workshop

Access to Document

Fingerprint

Cite this