Examining the Differences in Assessing Quality of Translations and Acceptability of Texts

The main focus of modern Translation Studies seems to be the ever changing challenges of successful cross–cultural communication. With globalisation of society, limitations in mutual understanding are surfacing, which are usually followed by the communicator’s failure to meet the other party’s expectations. Such expectations are not only linguistically, but culturally embedded and might prove difficult to grasp for those who are not closely connected to the particular culture. Mastering linguistic abilities is an inevitable requirement and represents an important aspect of cross–cultural communication, but achieving a high level of acceptability of (translated) texts proves to be just as important, while not always directly related to the traditional linguistic aspects of the text. It requires meeting the expectations of the target audience which exceed the scope of grammar and vocabulary. Prior cultural and discoursal experience with relevant texts proves to play an important role in producing translations which meet the target audience’s expectations in terms of linguistic, but more importantly in terms of discoursal characteristics.


examining the Differences in Assessing Quality of translations and Acceptability of texts 1. Introduction
Experience with translation texts has shown that there is an important difference in assessing translation quality as opposed to text acceptability. In order to delineate between the two terms -quality of translation and acceptability of text -we need to look into various levels and aspects of text processing and emphasise the important differences between them. While the two terms might overlap at certain levels or in certain aspects, they should not be confused or treated as 'interchangeable'.
The following paper is based on a study conducted at the Department of Translation (cf. Burazer 2011). It focuses on identifying the differences between assessing translation quality and assessing text acceptability by examining the different aspects of text processing. It starts by delineating between two sets of assessment criteria, one for assessing quality of translation and the other for assessing acceptability of texts, then it focuses on the differences in the prospective target audience, and it concludes by discussing the target audience expectations.

The differences in assessing quality of translation and acceptability of texts
When assessing quality of texts in general, the assessors have to focus on specific textual features or standards which have been determined in advance, prior to the task of assessing. The assessors are usually experts within the particular linguistic community; these could be either linguists or language teachers or other language professionals. The textual aspect we traditionally focus on in assessing quality of texts are usually along the lines of grammatical accuracy, appropriateness of vocabulary choices, phraseology, tense use, and similar micro-level linguistic aspects.
When assessing the quality of translation, the assessors are usually translation experts or translation studies teachers who are measuring the quality of the target text (TT) against the quality of the source text (ST). In order to perform such a task well, the assessors need to have mastered both languages -the ST and the TT. They need to be familiar with the languages' specific characteristics at various levels, their standards and the expectations outlined by the specific language communities pertaining to the text type in question. Quality of translation would then be assessed by close comparison of the TT with the ST, on examining the translator's accuracy of translation choices at word and phrase level, examining the tense structure in both texts, and so forth.
Assessing acceptability of texts, on the other hand, is an entirely different process. It requires the involvement of the target audience, who need not be linguistics or any other language experts, nor do they need to be translation studies experts. They need to belong to the particular discourse community of the target text. It is the actual readers and users of the target texts that will determine, according to their own criteria, whether the task of translation or TT production has been carried out successfully, whether the writer/translator of the text has done a good job in meeting the target audience's expectations.
This would require examining the standards proposed by the expert community against those employed by the target audience. It would require a direct involvement of the target audience in TT assessment by asking them about their opinions on acceptability of the TT at various levels and finding out their preferences. In order to do that, an experimental scientific project has been designed and carried out (cf. Burazer 2011), the results of which have pointed us in a somewhat unexpected direction.

The assessment criteria
The first important difference in assessing quality of translation and acceptability of texts is the assessment criteria. Assessing a text simply as a text and not as a translation requires an approach where assessors should be able to disregard the original text and not try to imagine or guess what the original structures would have been, but simply read the text as the target audience would read it -not as a translation but simply as a text. This differs considerably from assessing a text as a translation where close comparison with the ST is inevitable and required. It is therefore imperative that the selected assessors grade the (translated) texts on the basis of the proposed criteria of text clarity, stylistic acceptability, acceptability of vocabulary, acceptability of grammar (cf. Burazer 2011), with a strong emphasis on acceptability, not accuracy. The assessment criteria should be designed in such a way as to clearly focus on grading a text as a whole, on text level, and not on the level of specific structures or vocabulary.

The choice of assessors
Another level of approaching the task of assessing texts deals with the choice of assessors. In assessing texts in general, the assessors should be people who would in reality read these texts. It is their opinion that really matters, it is their expectations that really have to be met. On the other hand, assessing a translation as a target text would require a group of experts in the field of Translation Studies, for instance, to apply the previously agreed upon criteria and/or standards to the translated texts and see whether the expectations of the expert community have been met, whether the pre-set and pre-agreed upon standards have been observed. These two processes are entirely different, they require completely different skills to be applied and they also aim at achieving quite different goals.
We therefore first chose a text for translation from the field of law. There are several reasons for the choice of a legal text, such as: legal texts can be broken down into smaller manageable pieces, which aided in the practical aspects of the experiment; students are generally not too familiar with legal texts, they usually find them difficult; legal texts have certain distinctive features which are easily noted; legal texts are part of people's reality regardless of their professional environment.
Then we chose a group of assessors that consisted of Translation Studies experts such as teachers/ professors from the Department of Translation (Faculty of Arts), linguistics professors at university level, professional translators specializing in legal texts, and legal experts working as lawyers, prosecutors and judges (12 assessors altogether). They were all asked to assess the texts independently of the ST. They were not given the ST, even though some of them were extremely frustrated by having to assess a translated text as a text, and not as a translation. The TS experts and professional translators were expected to provide feedback on which translation strategies were observed to produce a text that does not 'sound like a translation', but rather one that functions as an independent text. The linguistics experts were supposed to primarily focus on micro-level textual characteristics and assess the use of grammar and vocabulary, while the legal experts were anticipated to be the actual target readers of such legal texts and were thus expected to provide feedback in terms of the texts' purpose, clarity and effectiveness.

The target audience expectations
Expectations in general have been extensively discussed by authors from various fields of study, ranging from linguistics (Yngve 1996) and discourse analysis (DA) (Brown and Yule 1983;Cumming and Ono 1997;de Beaugrande 1996) to translation studies (TS) (Gutt 1991) and cognitive science (Minsky 1974;Wilks 1973). Linguistic processing is heavily dependent on previous experience through which expectations are formed. Subsequent experience with texts either consolidates the expectations or not. If consolidated, they might turn into norms, otherwise new expectations are formed.
The expectations of the linguists and TS experts might differ considerably from those of the actual target audience, regardless of the grammar rules, semantic restrictions, translation conventions, and the like, based on the differences in their respective contexts, backgrounds, focus in text processing and goals.

Linguistics
Yngve's Human Linguistics theory introduces the so-called expectation procedures, which are mechanisms functioning within an individual as a system. They are formed within the system on the basis of experience with various communicating events and stored in the individual's memory. They are then triggered at the time of communicating and produce the sense of familiarity with the communicating event based on previous experience with similar communicating events. The communicating individual can thus make sense of the communicated substance, provided that the relevant expectation procedures have been triggered and that their expectations have been met.

Discourse analysis
Within the framework of DA, the emphasis is placed on texts and processes related to previous experience with texts, which is the common denominator of several theories within DA. Here are a few examples of DA theories dealing with target audience expectations. Sperber and Wilson (1997) in their relevance theory presuppose the factor of relevance to the receiver of the communication. The target audience selects the relevant message out of a myriad of possible interpretations. We are talking about the so-called "mini-max" or "cost-benefit" strategy, where the communication principle is: the less energy needed to process a communication with the most benefit the better. This shifts the attention from the author of the text to its audience and their expectations. Cumming and Ono's (1997) theory discusses the importance of the so-called text frequency and its role in forming expectations about text characteristics. This coincides with the definition of the notion of text acceptability, since the expectations about what texts of a particular genre should be like are formed on the basis of existing texts, not on the basis of linguistic theories authored by linguistics or any other theoreticians. The first approach is pragmatic and based on the discourse community's preferences -the latter would be prescriptive and based on theoretic concepts and preferences of those who might not be directly involved in the particular discourse community at all. Brown and Yule (1983) introduce the term local text interpretation, relating to the analogy principle which presupposes text processing in compliance with previous experience with the genre. This is the so-called prototypisation of the world where our perception needs to comply with some sort of stereotype in order to be interpreted. In other words, the prototypes and the stereotypes are representations of the expectations of the audience. Van Dijk's (1977) assumed normality of the world proposes that in the process of communication, people seek regularities and patterns we as a community share with the other members. Therefore we form expectations within the communicative process and expect them to be met as well as seek to meet other participants' expectations in the communicative process.
De Beaugrande (1996) touches upon expectations within a discourse community by discussing the so-called process of self-organization of context, where in the process of communication all possible meanings are triggered, but the context automatically deactivates the irrelevant ones on the basis of close connections between language and general knowledge of the world which trigger expectations in reference to a particular context.

translation studies (tS)
Within the framework of TS, the emphasis has been placed on the translator's specific roles of reader and writer. When performing the role of the reader, the emphasis is on the importance of text processing; on the basis of previous experiences with texts, specifically texts of the particular genre, the translator as reader has certain expectations about the text at hand which in the process of reading the text are either met or not.
Further on, in the role of the writer of the target text, the translator's emphasis is placed on target audience expectations as mechanisms influencing the choice of translation strategies based on experience with texts. At this stage of the translation process, the translator is trying to estimate the expectations of the actual readers of the TT and mould the translation in a way that would be most likely to meet the target audience's expectations. Gutt (1991), for instance, focuses on how people share thoughts with one another. He emphasises the importance of target readers and their expectations, because getting the message across, which is the ultimate goal of any communication, involves knowing your audience, their expectations, having experience with the audience and being willing to meet their expectations.

Cognitive science
Expectations based on previous experience with similar situations or events found their way into cognitive science a few decades ago. Back in the 1970s, Minsky (1974) and Wilks (1973) introduced the terms frames and terminals, respectively. They argued that what makes a written communication successful is the reader's expectation of structure within a story, as well as their expectation of coherence. People have an inherent urge to make sense of the text that is being communicated to them, which is why they manage to make sense of most texts.
Roughly a decade later, Kintsch (1988) discusses priming in memory as a way of accounting for successful communication. In the process of communication, there is only one language variant that is primed, not the whole linguistic system; therefore on the basis of expectations within the particualr language variant, the possibilities of achieving success in communicating process are raised.

testing the hypotheses of the role of the preceding experience in achieving a higher level of acceptability in texts
In accordance with the theoretical background presented in the previous chapter, there are several questions that might arise in reference to the extent of the previous experience with texts of a particular genre and its influence on the quality of (translated) texts. Therefore several hypotheses were formed which were tested in a study conducted at the University of Ljubljana (cf. Burazer 2011).
The first hypothesis was formed in reference to how the reader processes the text and which textual features the reader might be more inclined to remember after having read the text, depending on his/her immediately preceding activities. The supposition was that upon reading a text, the reader remembers certain micro level text characteristics (e.g., lexico-grammatical features, collocations, prepositional phrases) and macro level (e.g., informativity, register, terminology, impersonal/ passive structures), depending on the immediately preceding context. If the reader of the ST was exposed to predominantly micro-level context, s/he would focus more on the micro-level features of the ST. S/He would therefore have the familiar micro-level features primed in memory and would consequently be more likely to apply these in the production of the TT.
The second hypothesis was formed in reference to who would then actually produce a better text in terms of its acceptability. We anticipated that the translators who were previously exposed to the content relating to macro-level text processing would produce translation texts graded better on the level of acceptability by the target audience.

Drawing a demarcation line between micro and macro level textual features
An important task that had to be carried out in the project design stages was also drawing a demarcation line between micro and macro features of the text which the chosen assessors would be able to measure the selected texts against. Of course, we only selected a set of criteria; the assessors would give their own opinions of the acceptability in reference to the pre-chosen criteria based on their own knowledge of linguistics and their own experience with texts.
Drawing a clear demarcation line between micro and macro textual features proved to be a difficult task. Linguistic features are not really independent language pieces functioning either on their own or on the text level. Most of the linguistic features function only in reference to other parts of text. Grammatically they might represent a separate unit that can be studied -to an extent -on its own, but when digging deeper into that feature, one will discover a tight relation to other parts of the text and co-dependent relationships between different linguistic features within the same text as well as on the level of inter-textuality.
We therefore decided to compromise and nevertheless draw a line between micro and macro textual features, bearing in mind that there are no clear cut categories. The group of micro textual features contained cases of the use of specific grammatical structures, prepositional collocations and other mostly grammatical characteristics. The group of macro textual features was defined as a group dealing with linguistic features functioning above sentence level, on the level of the whole text, such as specific (in this case 'legal') terminology, grammatical voice (which is a specific feature of legal texts), sentence length (long in this case), repetition of vocabulary items or structures and similar.
In the process of determining the two groups, we realized that one of the main difficulties of the task was involving subjectivity of the assessors which cannot be switched off at any point. Assessors are only human and linguistic or translation accuracy are elusive categories which cannot be clearly determined since people's language experience differs a great deal and each individual develops their own linguistic patterns and preferences. Which also raises the question of objectivity of assessments, regardless of the fact that we have predetermined the assessment criteria, the categories, as well as provided rough descriptions of the expected evaluations.
So, it is not only the assessors that are a subjective factor in this process; it is also the criteria, the categories, and most of all the very notions of 'acceptability' and 'quality'.

The experiment
In order to test the first hypothesis, we designed an experiment where we tested 4 groups of students (39 altogether) in a task of translation of a short legal text. Two of them were control groups; one consisted of students from the Department of Translation, mostly 4th year students with at least 3 years of translation experience/practise within the Department, and at least some experience with translation of legal texts; the other consisted of law students from all 4 years of study, with no or extremely limited previous experience with translation. The other two groups were experimental. Prior to the translation task, one of the experimental groups was subjected to a treatment with micro-level textual content (the 'micro group'), the other was subjected to treatment with macro-level textual features (the 'macro group'). The 'micro' treatment consisted of several exercises designed specifically for the purpose of focusing on micro-level textual features, such as searching for the correct preposition in prepositional phrases, or checking accuracy of collocations (legal vocabulary). The macro-group treatment exercises were designed to focus specifically on macro level textual features, such as determining and verifying specific discoursal features of legal texts, comparing legal excerpts on discourse level (cf. also Burazer 2011).
The translation task which followed involved translation of a short legal test (3 short paragraphs, approximately 120 words). The time limit was 60 minutes. The task was carried out in a supervised formal academic environment, and the students were allowed to use any resources except for the internet or computer (such as word processors, grammar and spell checkers, and the like).
The anticipation was that the 'micro group' would remember the micro, while the 'macro group' would remember the macro features and that the students would apply the features that were primed in their memories during the production of the TT. The students' focus on the micro or macro features would lead to achieving a better text on the level of linguistic or discoursal features, respectively.
In reference to the two control groups, we anticipated that the 'legal control group' would remember the macro, while the 'translation control group' would remember the micro features, since the law students are supposed to deal with legal texts on the level of their functionality, efficiency and meaning, while the translation students were supposed to deal with texts not only but mostly on lexico-grammatical linguistic level. To prove this point, we asked the tested students to write down the text related features that they remembered right after having finished the translation task.
In order to test the second hypothesis, we had to carefully select a group of assessors who would grade the translations in terms of their linguistic as well as text level acceptability. For the task we chose a number of professionals from the fields of linguistics, TS and the law. We asked them to grade the translations as texts, not as translations and decide which of them was the best and which was the worst in terms of the proposed assessment criteria of text clarity, stylistic acceptability, acceptability of vocabulary, acceptability of grammar. The texts were graded on a grading scale from 1 to 10, where 1 was the least acceptable and 10 was the most acceptable.
We anticipated that the assessors would select a text from either the macro or the legal group as the most acceptable, since the two groups were supposed to process texts predominantly on textual or discourse level.

Results of the experiment
Generally speaking, research results largely confirmed the theses. In reference to the first thesis, the students from the micro group and the translation control group mostly described micro textual features in the questionnaire, such as individual words, prepositional phrases, collocations, while the students from the macro and the legal groups mostly described macro textual features, such as listing overall terminology or even summarizing the whole text.
At the same time, all groups very closely followed the structures employed in the source text, which would fall into the category of macro textual features. Slight deviations were found in the control groups in which the students used less repetition (a macro textual feature) and focused more on informativity than on accuracy. The latter observation supports our thesis that the readers who are exposed to processing texts on discourse rather than grammatical level will indeed focus more on these levels in text production or translation and apply the discourse textual features, which would result in higher level of acceptability of the translated text.
Based on the questionnaire for the selected assessors, the most acceptable translation/text was deemed a translation from the 'macro group'. It was best rated in 3 out of 4 categories: stylistic acceptability, acceptability of vocabulary and acceptability of grammar. Overall it was also rated best.
Rated worst was a translation from the 'legal group', which was a surprise and not in line with our anticipations. Possible explanations for this surprising outcome can be found in various aspects of the experiment. One such explanation might be the fact that the tested subjects were chosen randomly, which means that we could have selected one extremely weak student. Another explanation might be that the student did poorly only in one of the aspects of the text, such as impersonality of structure or terminology, which consequently triggered the negative response of the target readers.
Nevertheless, closely following the 'worst' legal translation was a translation from the 'micro group', which supports our thesis that the readers whose attention had been drawn to the micro textual features prior to the task of text production would have the micro features primed in memory during the task, which would result in their focusing largely on the micro textual features in the process of text production. Focusing mainly on the micro textual features and neglecting the macro discoursal features in most cases generates the overall impression that on some level the text is lacking in acceptability to the target audience.

Discussion
In tying the results of the experiment to the theoretical background of the research, we can say that there is indeed evidence that expectations of the target audience are a relevant factor in the process of making translation choices and can be anticipated by the translator based on the relevant experience with texts, which results in a higher level of acceptability of translations.
The selected theories within the field of DA focusing on target reader expectations propose emphasis on processes related to previous experience with texts of a particular genre. By focusing on the genre characteristics in text production and following genre conventions, we make the text relevant to the target audience (cf. Sperber and Wilson 1997). The so-called "cost-benefit" strategy is applied, which serves the audience in terms of investing as little energy as possible in text processing. The macro and the legal control groups were supposed to focus more on these text characteristics, which they did. The translations from these two groups showed more freedom in translation choices; they showed more variety in translation solutions. The students seemed not to be limited by the linguistic or grammatical restrictions, or even limitations of dictionary solutions. They would provide solutions which conveyed the relevant message at the cost of micro-textual level accuracy. This rendered the text more acceptable, but possibly lowered the level of the quality of translation.
The notion of the importance of 'text frequency' (cf. Cumming and Ono 1997) is also related to the formation of expectations of the target audience. People who read a number of texts of a particular genre intuitively perceive the relevant textual conventions of the genre and thus play an important role in determining pragmatically which texts are acceptable on discourse level. Within a discourse community, 'people share thoughts with one another' (cf. Gutt 1991), which makes it easier to recognize and meet each other's expectations. The legal expert assessors and the law students tested had an advantage in this sense, because, as members of the discourse community, they were expected to be familiar with the discourse features and conventions of the legal genre. The TS students were supposed to be slightly handicapped in this aspect, but on the other hand they were expected to be better equipped in terms of translation strategies and linguistic skills in acquiring the genre conventions. However, the results of the experiment show that the legal students did slightly worse than the translation students. The translation from the macro-group (by a translation student) was rated best even by the legal expert assessors.
The pre-treatment in the micro and the macro groups proved to have had an important impact on text processing of the tested students. The micro-linguistic features seem to have been primed in memory (cf. Kintsch 1988) in the case of the students from the micro-group, while the focus on the macro features was more evident in the macro-group. This is shown in the students' questionnaire where the micro-group mostly listed micro-textual features, while the macro and the control groups either listed terminology or produced whole summaries, which proves that their text processing was taking place on discourse level, not on micro-linguistic level.
In order to be able to produce any kind of translation, the translator as reader of the ST needs to first be able to make sense of it. The reader's expectations of text structure and coherence need to be met (cf. frames in Minsky 1974 andterminals in Wilks 1973). The law students and the translation students control groups both showed evidence of processing the text on discourse level since they displayed a certain level of flexibility in terms of making daring choices. They felt comfortable making changes to the ST, they were not reluctant to change certain vocabulary items for their synonyms or even hypernyms. Because they were able to extract the message of the text, they felt they should make it as easy as possible for the target reader to process (cf. the relevance theory, Sperber and Wilson 1997), even if it meant making a few unjustified changes to the ST. This proves that their focus was indeed on the macro level, on the text effectiveness, not on individual vocabulary items or grammatical structures, which are micro level features.

Conclusion
Assessing acceptability of texts is an important task especially in the process of educating translator trainees. It is imperative that we do not confuse acceptability of texts with the quality of translation, since the latter focuses on the aspects of accuracy and faithfulness between the ST and the TT, while the former emphasizes the link between the textual features displayed in similar texts of the same genre and the ones displayed in the translation text, which is no longer viewed by the target audience as translation, but simply as text.
The results of the research have shown that register and genre appropriateness are the main textual features which trigger the response of text appropriateness, regardless of the grammatical accuracy, accuracy of translation choices, collocational structures, and other aspects. These findings point in the direction of the importance of studying and recognizing discoursal text characteristics, rather than mostly focusing on lexico-grammatical micro-level textual features if we wish to achieve a higher level of acceptability in the response of the target audience to translated texts.
Such goals can be reached by placing more emphasis on corpus work and less on glossary or dictionary approaches in teaching. Another proposed solution is raising awareness of the importance of each translation choice, because the research results have shown that even if the translation is generally good, one unfortunate choice of words or of structure that violates the genre conventions could result in a general impression of low level of acceptability. Working at the global level and emphasising discoursal text features as opposed to solving lexico-grammatical tasks will help develop skills that will ensure better acceptability of (translated) texts.
We have also established that the notion of expectations is not a new formulation within a linguistic or Translation Studies discourse, nor is it restricted to purely psychological, cognitive or sociological notions of human behaviour. Expectations are becoming an inevitable area of investigation within the framework of studying human linguistic behaviour, part of which is represented by the process of translation.
As target text writers, translators carry the great burden of being responsible for revealing communication clues to target audiences and other societal groups. They should therefore be able to anticipate their target audience's expectations and do their best to meet them by making the right decisions in terms of selecting and properly employing appropriate and acceptable textual features that which make the text appropriate and acceptable on the textual level as a whole -that is, that do not merely correspond to the ST vocabulary and structures on the micro, sentential level of text processing. This way they help in the process of development and interconnectedness at the level of cross-cultural communication, as well as at the level of making the discourse knowledge available and acceptable to its target audience.