THE ATTITUDE OF DICTIONARY USERS TOWARDS AUTOMATICALLY EXTRACTED COLLOCATION DATA: A USER STUDY

The paper is based on a survey conducted within the framework of the basic research project Collocations as a Basis for Language Description: Semantic and Temporal Perspectives (KOLOS; J6-8255). It presents a qualitative analysis of a user evaluation of the interface of the Collocations Dictionary of Modern Slovene (CDS). It discusses an alternative perspective—the user's point of view—on problematic aspects of individual dictionary features, which require further lexicographic analysis and discussion. The collocations user study presents a model of the process of user evaluation; its findings are significant primarily for determining problems encountered by users. They also serve as a useful basis for methodology improvements in future, comparable lexicographic user studies and analyses.


I N T R O D U C T I O N
In the digital world, a dictionary is increasingly becoming a network of dynamic shifts between different language information and resources, as well as a testing ground for various contemporary conceptual lexicographic approaches. The concept of a "responsive dictionary"-a dictionary characterised by its capacity to respond to the dynamics of language development and include the interested language community in the development of language resources in a methodologically transparent manner (Arhar In addition to coming up with an alternative dictionary form, modern lexicography has increasingly recognised the undeniable value of dictionary users. Despite the growing interest of international lexicographers in user studies, in Slovenia the field remains understudied and overlooked. This is why the present study examines the role of user reception and contribution to the upgrades and improvements of dictionaries. The idea of a responsive dictionary recognises the user as an active co-creator of (digital) language resources, as well as a critical evaluator of the features offered. The results of an open discussion between linguists and users represent a useful starting point for further analysis of the design of dictionaries, and, in the present case, of the general role of the collocations dictionary as a responsive dictionary within the field of lexicography.
The present study focuses on the users' attitudes towards automatically extracted collocation data, especially in relation to specific features introduced into lexicography by responsive dictionaries. In their initial phase, responsive dictionaries are automatically compiled and relatively quickly published for public use; alongside linguists, the language community then gradually helps improve and clean the data. The Collocations Dictionary of Modern Slovene was also immediately made available to the public, i.e. in the initial, unprocessed stage containing noise or errors. The design of the dictionary interface, however, featured options to eliminate these shortcomings (data evaluation and cleaning), information about the linguistic completeness of the entry, and other similar features (Kosem et al., 2018c). The present study was interested in specific groups of users and their attitudes towards the present state of the dictionary, their opinion on its responsiveness (which includes automatic compilation, gradual upgrades, and user involvement), and their response to particular types of existing errors in the data. The user evaluation is intended to serve as a basis for identifying problematic areas, as well as less problematic areas in need of improvement, and will play a key role in the improvement of the collocations dictionary interface.
The paper begins by presenting the method of user evaluation of the Collocations Dictionary of Modern Slovene 1.0. This is followed by an analysis of the three thematic segments of the user evaluation, i.e. the three-part design of the evaluation interview. A representative case (proper nouns) demonstrates user perspective on (non-)problematic features of data and the dictionary interface. The conclusion summarizes the key findings of the study and examines the suitability of the applied method as a model for user evaluation in similar lexicographic user studies.
Despite growing opportunities for user involvement, Slovene lexicography has been relatively slow in developing an interest in user studies. This is why, as mentioned in previous research (Rozman, 2004;Stabej, 2009;Logar, 2009;Gorjanc, 2017), Slovene lexicography has a glaring lack of data in relation to user habits, needs, capacities, and preferences. Over the past few years, important steps have been taken, such as the development of a user typology (Arhar Holdt et al., 2016), the research of user needs in relation to selected language problems (Čibej et al., 2016;Arhar Holdt et. al, 2017), the participation in an international study on user attitudes to general monolingual dictionaries (Kosem et al., 2018a(Kosem et al., , 2018b, and the development of methodologies for user inclusion and tracking within the framework of a responsive dictionary (Arhar Holdt et al., 2018).
The present study contributes to the available array of tried and tested methodologies (a comprehensive overview of existing methodologies is provided in Welker, 2013aWelker, , 2013b with the addition of user evaluation based on the guided think-aloud method. Think-aloud protocols have been described by Tarp (2009, p. 287)  We used the basic idea of the method, but adapted it to serve the purposes of a straightforward evaluative approach: the participants were presented with the dictionary; while they were using it, an interviewer was actively involved, suggesting queries and guiding the "thinking" with a set of prepared questions.
Both the audio and the participants' interaction with the screen were recorded. However, only the audio was transcribed and analyzed (as the "protocol" itself was guided and thus comparable).

Research Goals and Sample Structure
The primary aim of the study was to determine the participants' opinion on the advantages and disadvantages of the Collocations Dictionary of Modern Slovene and responsive dictionaries in general, and to find ways of improving its user-friendliness. It was our intention to examine whether adult speakers of Slovene -particularly those with linguistic background or keen linguistic sensibility -know how to use, read and interpret the Collocations Dictionary of Modern Slovene, despite the fact that the dictionary featured raw, automatically extracted data. Our focus was on determining the participants' attitudes towards: • automatic data compilation and errors; • continuous dictionary upgrades and updates; • possibility of user inclusion or contribution; • innovative interface functions.
Following the typology of potential dictionary users (Arhar Holdt et al., 2016), the study included four distinct target groups of participants: translators and proof-readers; teachers of Slovene as a first language; teachers of Slovene as a second or foreign language; and lexicographers. The selected sample covers different scenarios of potential use, which allows the joined feedback on the dictionary to be perceived as more representative. Teachers were included to evaluate the didactic value of the dictionary, primarily its usefulness for teaching vocabulary to students . Translators can benefit significantly from  knowing what collocations and colligations are typical for a given word, while   proofreaders need straightforward normative information to support their decisions. Finally, the group of lexicographers was included to identify whether and how their views differ from the opinions of actual dictionary users, e.g. whether as the creators of the dictionary, they perceive its pros and cons similarly to other groups, and whether they propose similar steps for further development than other groups. 3 The study included 40 participants. As seen in Table 1, the participants were primarily between 30-50 years of age, with 10-30 years of work experience; they originated from different Slovene regions or-in the case of teachers of Slovene as a second or foreign language-from abroad. The call for participation was circulated widely through various means of communication 3 Students of Slovene as an L1 and learners of Slovene as an L2 did not participate in this step of the study. We chose to focus on adult professional users to make the best of the time and resources available within the project. Compared to the selected user groups, students are more easily accessible and after the project, the study can be continued to include both them as well as other potentially relevant user groups.
(such as mailing lists). The participants responded voluntarily, which needs to be taken into account in the interpretation of the results: the sample consists of participants who are relatively familiar with innovative, digital, and responsive language and dictionary resources, as they use them in their everyday work.

Evaluation Interview: Design
The evaluation interview was carefully planned and pre-tested on a group of researchers, i.e. linguists and research colleagues assuming the roles of interviewees. Our method was selected in order to enable identification of relevant data communicated in various ways by the interviewee, with minimal interviewer influence; its aim was to detect problems encountered by the interviewee while attempting to complete a specific task-working with a dictionary, on particular dictionary entries. To facilitate internal processing and analysis of acquired data, the participants were guaranteed full anonymity and asked for prior written consent for the recording of their screen and voice.
The approximately 30-minute long evaluation interview was based on a prepared three-part questionnaire (Appendix 1). During the first part of the session, the participants were asked-while thinking aloud-to click randomly in the dictionary and to query entries of their own choice. In this way, they could familiarize themselves with the Collocations Dictionary and form a first impression. At the same time, they were encouraged to spontaneously express their thoughts, feelings, and emotions and report whether they encountered, sensed or noticed any problems. Attention was primarily focused on the par- The second segment of the interview involved working with specific headwords; the participants were guided and tested to determine whether they recognized the (non-)problematic nature of particular entries. We were interested in their ability to interpret raw data, the amount of problems or errors detected, the nature of these errors, and the levels of distraction posed by the errors. The evaluation included three types of dictionary entries; prior to conducting interviews, we created a list of existing data errors for each entry and thus anticipated the participants' potential observations. a) An example of a non-problematic and lexicographically fully examined entry, albeit highly polysemous and thus collocationally diverse: down', which appears in inadequate structures due to the absence of the reflexive pronoun se.

Errors on the level of syntactic structures
The collocate was tagged with an incorrect part-of-speech.
pivo pite instead of pivo piti 'beer of pie' instead of 'to drink beer' The verb collocate should appear in the negative form.
piti piva instead of ne piti piva 'to drink beer' instead of 'to not drink beer' [missing negative particle]

Errors on the level of collocations
The collocation is nonsensical as it makes no sense if taken out of context or without additional elements.
pivo k ustom instead of dvigniti kozarec piva k ustom 'beer to the mouth' instead of '[to raise a glass of] beer to the mouth' The headword appears next to a syntactic structure in the genitive plural or is a plural noun; the collocation makes no sense without an additional, quantitative element.
pivo po tolarja instead of pivo po 300 tolarjev 'beer for tolar' instead of 'beer for 300 tolars' The third and final segment of the interview examined the participant's opinion on the general usefulness of the dictionary, its digital form (continuous upgrades) and their assessment of its look.

Transcription and Annotation
The annotation of interviews with the participants was done on the transcriptions of audio recordings, which were completed by four students of linguistics.
The transcription followed a set of clear guidelines; one of the key guidelines was that the transcription should not be reduced to summarizing, but should instead record the conversations as faithfully as possible, with linguistic adaptation and standardization only permissible on the morphological level.
The annotation process followed the general thematic structure of the questionnaire (Appendix 1). A set of annotation guidelines was prepared, con- relevant information with their own tags, which can be easily grouped in the end to achieve the final annotation scheme). There are two main advantages of this approach to qualitative data analysis: a) tagging the transcriptions can provide a quantifiable overview of the data (e.g. the frequency of the tags reveals the most frequently discussed topics, issues, and recurring patterns in the analyzed texts); and b) Taguette is designed in a way that allows segments related to a specific feature to be exported to a separate file, essentially combining all related segments from different transcriptions into a single document. This allows for a more thorough analysis of a specific issue across all participants or participant groups.
Because the interviews in our research were semi-structured and focused on specific features of the Collocations Dictionary of Modern Slovene, we elected to follow a top-down approach and prepared a limited tagset for the annotators to use. The higher the frequency of the annotation, the more prevalent or topical the discussed argument in the user group. On the other hand, less frequently annotated topics might indicate that the user either has not noticed a feature or found it less important compared to others.

Annotation Results
The annotation typology (shown in Table 3, along with the total frequency of each tag) consists of 4 main categories 4 with multiple subcategories. The table also presents the general attitude towards a specific feature indicating whether the participating evaluators expressed more arguments pro or contra. These labels are discussed in more detail in Section 3.

D A T A A N A L Y S I S O V E R V I E W
The initial overview and analysis of categorized opinions included all the structural and thematic segments covered by the evaluation interview (Appendix 1): examining the intuitiveness of the dictionary interface, the participants' attitudes towards errors and selected general features of the dictionary. All the assessed categories mentioned above were divided into groups according to predominant opinion on their adequacy (the category is marked by PRO) or inadequacy (the category is marked by CONTRA) (Table 3). 5 We were interested in determining the areas in which the participants agreed or disagreed.
This data is relevant for identifying problematic and less problematic categories, and for further improvements of the dictionary interface.
An example of an opinion 6 marked by PRO: [ An example of an opinion marked by CONTRA: [2] "I'm put off by mistakes, because I find this slows down my work considerably." [translator, on automatic noise in dictionary data]

Evaluating Features of the User Interface
The first part of the interview involved the participant exploring the dictionary features in a free and unstructured manner. The aim was to evaluate the intuitiveness of the user interface, e.g. the entry phase indicator (pyramid icon), the presence or absence of sense indicators (sense menus), the three-dot icon for accessing specific syntactic structures, etc.
As shown in Table 3, the participants from all groups described all the selected features as positive (PRO): they rated them as excellent, highly useful, functional and intuitively designed dictionary elements. None of the participants expressed arguments against any of the features.
However, we have identified a common suggestion (across all participant groups) for improvement relating to the visual upgrade of the pyramid icon, i.e. the icon should be more noticeable and its function clarified. Divergent opinions (PRO/CONTRA) were noted with regards to the possibility of user involvement. All the participants see the option of up-or downvoting the collocations as a useful and welcome feature; proof-readers and translators, however, pointed out that they often lack time for doing so, whereas the teachers expressed concern about the feature being used by non-competent users:

Evaluating Data Error Distraction
The second part of the interview, which focused on examining the participants' attitudes to various types of errors, demonstrated that the participants-judging by their response to test entries and their self-reports on previous, oftentimes daily dictionary use-mostly do not seem to notice them. In fact, they seemed to first become aware of the errors only during their participation in the user study, after being guided in their work on specific entries (belina, pivo, klop and usesti (se), i.e. after being systematically queried whether they noticed any errors and asked about the extent of their disruption. 7 Prompted by the interviewer, the participants evaluated specific types of errors, such as the absence of the reflexive pronoun se in the verb headword, errors due to homonymy, the inclusion of proper nouns in the dictionary, etc.
As seen in Table 3, the most distracting type of error occurs due to homonymy and was mostly independently detected by the participants. In the headword klop, homonymy results in most of the collocates being wrong (greti klôpa 'to keep a tick warm' -instead of greti klóp 'to keep a bench warm', guliti klôpa 'to wear out a tick'-instead of guliti klóp 'to wear out a bench', sesti v klôpu 'to sit on a tick' -instead of sesti v klopí 'to sit on a bench'). 8 The participants also had mixed opinions (PRO/CONTRA) on the inclusion of proper nouns in the dictionary. Due to the diversity of opinions on this issue and some very interesting results, we examine the issue in more detail in Section 4. The participants marked all the other shortcomings (i.e. types of errors) with CONTRA, and mostly did not notice them independently during their work with dictionary entries, as mentioned above: 7 It should be noted that the above was not true for the group of lexicographers-unlike the other participants, who encountered such errors for the first time, the lexicographers were well acquainted with the dictionary. Namely, the group of lexicographers included many of the original authors involved in the diverse stages of the building of the collocations dictionary (data processing, user interface design, and other processes of development).
8 Homonymy-related problems can occur because of incorrect morphosyntactic tagging and/or problems in post-processing. One particular issue of corpus data is that lemmas are form-based, so differently-pronounced headwords with the same form will be combined under the same lemma. The problems become particularly noticeable when such a word (as a headword or a collocate) features in the grammatical structure in a case that is not nominative.

Evaluating General Features of the Dictionary
In the final part of the interview, the participants evaluated the general features of the collocations dictionary, such as its automatic compilation, digital-only form, and look/design. [translator, on the advantages of a digital-only dictionary form]

Participants' Improvement Suggestions
While evaluating specific interface features, the participants also suggested several improvements on their own initiative. The suggested improvements included adding information on the collocate or collocation frequency, the option to export data, the addition of accents and pronunciation to headwords (especially homonymous headwords). The bulk of suggestions was primarily concerned with the option to click on the headword in order to return to the initial page, the visual upgrade of specific interface elements, such as upgrading the frequency filter with a color scheme or a color code, making the pyramid icon more graphically pronounced by enlarging it, using intense colors or stripes, including a short headline, description, etc.
In this section, we describe a qualitative analysis of the participants' attitude towards the inclusion of proper nouns. The complexity of this issue and its possible solutions were reflected in the results of the participants' evaluation. Most participants supported the inclusion of proper nouns in the dictionary (see Table 3). However, all the participant groups identified reasons both for and against the inclusion. This was especially pronounced in the group of lexicographers, where all the participants listed reasons both for and against the inclusion. Table 4 gives an overview of the above discussed opinions within individual groups. Teachers of Slovene as L2 9 1 0 Translators, proof-readers 6 3 1 Lexicographers 0 0 10

Attitude of Teachers of Slovene as a First Language
The majority of teachers of Slovene as a first language (Table 4) had a positive attitude towards the inclusion of proper nouns, especially for the following reasons: • the students find them more illustrative and concrete; • they pique the interest of students and promote intellectual and cognitive processes; • their specificity is attractive and intuitive, which is reflected in increased study motivation of the student and, consequently, in a more flexible understanding and adequate language use.
While giving a positive evaluation of the inclusion of proper nouns because of their ability to illustrate and convey a more specific example of language use, one of the teachers expressed doubts regarding the benefits of including trademarks (e.g. Laško pivo, a Slovene beer brand) and questioned their contribution towards understanding word use.

Attitude of Teachers of Slovene as a Second/Foreign Language
Almost all teachers of Slovene as a second language ( Azerbaijan]'. There was a suggestion to exclude specific types of proper nouns, such as personal names and surnames.
As seen in Table 4, only one of the teachers was of opposed to proper nouns.
The teacher pointed out several proper nouns incorrectly spelled with a lower-case initial letter ( context may make it difficult to determine that this is a football club.
On the other hand, a smaller number of proof-readers and translators-3 out of 10-argued against the inclusion, especially in relation to trademarks (e.g. Illy kava 'Illy coffee', Laško pivo 'Laško beer'), since they find this degree of specificity meaningless and unnecessary. Furthermore, one of the participants had a mixed opinion, since they believe that the decision regarding the inclusion of proper nouns in the dictionary depends primarily on the type of proper noun and the relevance of the information conveyed by the proper noun.

Attitude of Lexicographers
As already mentioned above, all the participating lexicographers expressed arguments both for and against the inclusion (Table 4), which is to be expected considering the fact that they see the dictionary not only from the perspective of the user, but also as content developers and originators of lexicographic concepts.
The arguments for the inclusion were related to semantically relevant proper nouns; the participants stressed that not all proper nouns are equally semantically relevant (kranjski Janez 'John Doe' -Janez Novak; delati se Francoza 'lit. to pretend to be a Frenchman, meaning to feign ignorance' -Francoz 'Frenchman'). Proper nouns were also considered a valuable source of information on the most typical ways of addressing people, with the caveat that the specific personal name in and of itself is not that relevant (dragi Janez 'dear Janez' -dragi + [personal name]); the key information here is the discourse category.
The arguments against the inclusion were related to longer sequences of collocates of the same type, since this type of information is distracting and does not enhance user experience. This is the case for the selected entries

Participants' Suggestions for Dictionary Improvements
The participants suggested two solutions on the topic of inclusion and presentation of proper nouns in the dictionary.
The proof-readers and translators suggested an introduction of a special button for hiding the proper noun candidates; this would give them the option to choose whether to use it and thus make querying the dictionary more efficient.
Their work is related to the specific nature of various text types and vocabulary, the variety of topics subject to intense linguistic research, as well as time as one of the key components, which is why this group believes that the dictionary should adjust to the needs, wishes, and expectations of its target users as much as possible.
Lexicographers proposed a solution of grouping collocates belonging to the same semantic type under a semantic label (e.g. football, hockey, basketball > sport; dog, cat, hamster > (domestic) animal). This would improve the visibility of collocational behaviour of the word and ease browsing through (long) lists of collocates.

D I S C U S S I O N A N D M E T H O D A S S E S S M E N T
The user evaluation of The Collocations Dictionary of Modern Slovene 1.0 identified the participants' attitudes towards its features, which were grouped in three discrete segments in the research interview. The user evaluation was, to a great degree, positive. In the first segment of the interview, the participants evaluated as positive (i.e. relevant for the dictionary and useful) all the features that they independently recognized. In the guided part of the interview (during which they worked with selected entries), the participants expressed reservations about some (but not necessarily all) data errors, especially mistakes arising as the result of homonymy and ambiguous word inflections. Opinions also differed with regards to the (non-)inclusion of proper nouns (as seen in Section 4). The third and final segment of the interview asked the participants to evaluate general dictionary features; here, also, their opinion was unanimously positive.
The analysis of the participants' attitudes towards errors has demonstrated that even in their initial stage (during which they still contain mistakes), responsive dictionaries represent an invaluable tool-this was a common opinion across all participant groups taking part in the study. In order to understand this degree of positive or permissive attitudes towards data errors, we need to keep in mind that before the publication of the Collocations Dictionary of Modern Slovene, collocation data for Slovene had not been readily available. To a great extent, the participants' enthusiasm is thus a reflection of the newly opened possibilities offered by the dictionary-it is, therefore, safe to conclude that the participants prefer easy accessibility over fully clean data.
The evaluation further demonstrated that: a) it is vital that dictionary users are alerted to the presence of errors with the pyramid icon, which indicates the phase of entry completeness; and b) given the presence of context, the possibility of accessing examples, and links to the Gigafida corpus, it is possible for the users to resolve any ambiguities.
In terms of dictionary shortcomings, special attention should be given to the most "vulnerable" user groups, i.e. teachers of Slovene as a first language and teachers of Slovene as a second/foreign language. Teachers bear the responsibility of choosing the sources used in the classroom with students who as language learners are somewhat less qualified to independently identify and resolve data ambiguities in the manner described above. Didactic use demands precise and unambiguous information, so that the teacher does not lose time by having to correct errors. On the other hand, the teachers themselves found the dictionary to be very useful and of great help, especially as a starting point for exercises, a tool for enriching vocabulary, for checking the correctness and adequacy of phrases; for writing fiction and poetry, for discussing collocations, using idioms, newspaper language, etc. They were excited by the authenticity of the language, the interconnectedness of different resources, and especially by the possibility to observe language as a natural phenomenon across all segments of its use.
What is important is that the study made it clear that many of the characteristics that were deemed problematic by linguists are not necessarily problematic for the users-this was seen, for instance, in the discussion of the participants' attitudes towards the inclusion of proper nouns. Contrary to our expectations, the particpants found proper nouns to be interesting and illustrative despite referring to a specific referent. Whereas the lexicographers' main concern was that the inclusion may result in overcrowding the dictionary (e.g. in cases where the headword is followed by a long, enumerating sequence of collocates of the same type), the participants found such concreteness more intuitive.
The evaluation identified areas of the dictionary and its interface which the participants find adequate and those that need to be re-examined, improved and further assessed. In this sense, the study achieved its main goal and the selected method proved to be successful. Even though collecting, recording and categorizing evaluation data is extremely time consuming, the transcribed opinions offer insight into problems and solutions that significantly contribute to concepts proposed by dictionary developers. The evaluation study has resulted in a number of positive findings, but also revealed possibilities for improving the methodology in case of further, comparable studies.
One of the positive aspects of the study was its multi-stage design (i.e. interviews -transcription -annotation -analysis): on the one hand, it enabled a careful and thorough planning of the entire process of the study; on the other, it increased the time needed to realize individual tasks. The study took place between May and September 2019, with the time span depending on several outside factors: the availability and flexibility of the participants, their willingness to co-operate, collaboration with students, and unforeseen technical difficulties. Apart from demonstrating the need to plan for a longer time span, our experience has also shown the following: • in order to secure participation, it is very important to adopt a personal approach, including personal correspondence, willingness to record sessions in the participants' place of work, etc.; • collaboration with students demands careful and consistent monitoring of their work, including providing clear and understandable guidelines and a detailed examination of the transcriptions and annotations; • a methodological process reliant on the use of recording software and equipment and the use of a digital dictionary should take into account potential technological difficulties and provide for adequate data backup.

C O N C L U S I O N
The user evaluation of the Collocations Dictionary of Modern Slovene has proven to be a highly efficient way to detect (non-)problematic dictionary features and represents a solid foundation for further attempts to improve and upgrade the interface to make it more user-friendly and functional. It presents a model for evaluation and identification of user problems; the gathered results reveal areas for potential methodological improvements and are thus useful for similar lexicographic user studies and analyses.
The findings of the study indicate that the methodology of automatic extraction of lexical data has indeed reached the levels where such data can be immediately presented to the users, something that has been often claimed by authors such as Kilgarriff et al. (2013) and others. Nonetheless, what the study also shows is that the presentation of such data matters, i.e. features are needed that alert the users to the different stages of data validation and that enable data manipulation/filtering. Part of the reason for this need lies in the quantity of automatically extracted data which always exceeds the quantity after human clean up and selection. 10 As envisaged when preparing the study, the user feedback obtained will be used in the preparation of the next version of the Collocations Dictionary of Modern Slovene. First and foremost, we need to acknowledge that no radical changes are needed; to some extent, the aspects of data quality and quantity, as well as clarity of presentation, need to be addressed. For example, we plan to introduce additional options to filter collocates, such as an option to hide proper nouns (as opposed to removing them from the dictionary completely), hiding or downgrading semantically less relevant collocates, and viewing a selection of top collocations (or collocate clusters) regardless of their syntactic structure. In terms of visual improvements, the pyramid icon will be made more conspicuous. In cases where the distribution of collocations over syntactic structures is uneven, structures with more collocations will receive more space in the display. Moreover, an option for downloading entries will be added.
As evidenced by the results of the study, user groups differ in their attitude towards the inclusion of proper names, which makes it difficult to propose universal answers for this issue. Solutions that introduce a choice for the user (as the on/off buttons), seem to be a way to go for such cases. Nonetheless, one feature that seemingly requires a rethink is the option of user participation; to this end, we are already testing other approaches such as gamification, which may help us clean the dictionary data even faster and less obtrusively than existing voting method in the dictionary. And gamification, in combination with improvements to the automatic data extraction method, will make the dictionary even more »responsive«.

A P P E N D I X 1 : E V A L U A T I O N Q U E S T I O N N A I R E First segment: Free use of the dictionary
During the first interview segment, the participants are asked to browse the dictionary freely while thinking aloud. This allows them to form the first impression and get the general sense of the dictionary.

Second segment: Guided work with dictionary headwords
In the second part of the interview, the participants are guided by the interviewer to click on a number of headwords that were pre-selected according to a carefully designed set of criteria. The participant is thus familiarized with the various functions offered by the resource.
The participant is presented with the following headwords: belina 'whiteness' -a non-problematic entry that has already been finalized by lexicographers How do you find this headword? Is it in any way problematic? Do you notice any errors?
Can you identify the various functions available (e.g. the entry phase indicator, sense menus, collocate clusters), the possibility of using various filters, the option to contribute to the dictionary by rating collocations?
pivo 'beer' -an entry with potentially problematic collocates The headword is either a plural noun or appears next to a syntactic structure in the genitive plural; as such, the collocation makes no sense without an additional, quantitative element: pivo po tolarja 'beer for tolar' instead of pivo po 300 tolarjev 'beer for 300 tolars' klop 'bench' or 'tick' -a homonym that has not been disambiguated in the dictionary Do you find anything about the entry distracting? Did you identify the word as a homonym (words having the same spelling but different meanings)? Do you find the ambiguity distracting?
Are you distracted by proper nouns as collocates? Do you find that there are too many errors? usesti (se) 'to sit (oneself) down' -an inherently reflexive verb which is missing the obligatory se pronoun in the dictionary In its initial stage, this resource is compiled completely automatically. This is why, as you may have noticed, it also includes information that should not be here. Do you feel there is too much noise or that there are too many errors? Do you find this distracting? Why (not)?
This resource enables dictionary entry tracking and provides information on the phase of entry completeness, generated by clicking on the pyramid icon.
Did you notice this? How do you find this?
This resource was compiled automatically and as such was made freely and openly accessible as soon as it was compiled. Do you prefer free and open resources with raw data or payable sources with clean data?

Digital-only form
This resource has no printed version. Is that a problem or do you find its digital-only form an advantage?