Computer Aided Analysis of Foreign Language Writing Process

Abstract:
A number of studies have examined the writing process in first language (L1) and English as a Second Language (ESL); however, there is little empirical data with regard to the foreign language (FL) writing process. Syst�me-D, a computer program for writing in French, accompanied by its tracking device, which provides a log of student inquiries made during the writing process, has afforded a novel means of analyzing the FL writing process. Twenty-one students in third semester (intermediate) college French participated in a study designed to examine how the Syst6me-D log provides data about the FL writing process. This study describes the instrument used to evaluate the Syst�me-D log data and suggests how this information can be used in the classroom.

KEY WORDS

Computer aided FL writing, Système-D, FL Writing Process.

INTRODUCTION

The concept of "writing as process" is gaining in importance with regard to foreign language writing (Barnett 1989). For many foreign language teachers, student compositions are no longer evaluated only as a final written product but are also viewed as texts which evolve and develop during the complex process of conveying meaning in writing. This shift in focus from evaluating the written product to analyzing the writing process has also had an impact on the way writing is taught. Before handing in a final composition students are often encouraged to engage in strategies such as planning, brainstorming, writing and revising. Teachers are also likely to offer students

5

the opportunity to revise their compositions before assigning a final grade for their work. While these changes in how foreign language writing is viewed, taught, and evaluated are important, there are very little data to indicate what students actually do as they write. We as foreign language teachers and researchers do not, in fact, know how our students are engaged in “writing as a process."

Research that explores the second language writing process involves primarily student self reporting or investigator questioning (Krapels 1990). These research methods of investigation, which can take place before, during, or after writing, attempt to describe and analyze, or attempt to provide a basis for describing and analyzing the nature of the writing process. However, since students are not always aware of the processes, or strategies, that they are using, the intervention on the part of the investigator may lead them to report their writing strategies and behaviors incorrectly. While these methods are useful in providing certain kinds of information about the nature of the foreign language writing process, the use of computer aided analysis can contribute significantly to the investigation. In order to further our understanding of foreign language writing processes, we devised a method of analysis designed specifically for use in the classroom setting, which provides insight into the strategies students use when writing in a foreign language.

BACKGROUND

Before discussing method of analysis, we should review the research that led to its development and use. The principal tool of analysis is Système-D, a software program for writing in French, developed by Noblitt, Pet, and Sold. This program is a word processor which provides students of French with access to a bilingual dictionary, a verb conjugator, a reference grammar, a vocabulary index, and a phrase index. In addition to its features as a learning and teaching tool, Système-D has a tracking device which provides a log of the inquiries that students make during the writing process.

In a pilot study using Système-D, Noblitt et al. (1990, 1991) designed a tracking system to maintain records, or logs, of student queries. They investigated the use of the tracking device and performed a qualitative analysis of queries as they relate to the learners' mental conception of L2. By studying students' lexical inquiries, the

6

researchers establish three query classes: token matching, type matching, and relexicalization. These studies suggest that teachers can use these different classes of learner inquiries to gain insight into the ways that students process language information with the goal of helping students correct misconceptions about L2. The researchers ultimately suggest that the Système-D tracking device can be an important tool for exploring the language learning process.

This pilot study convinced Noblitt et al. of the value of the Système-D tracking device, and this option was made an integral part of the software program. If teachers choose to use the tracking device, a log of inquiries is printed whenever a student prints out a composition after a writing session. However, the actual classroom use of the logs by teachers and/or students remains undefined. How can teachers evaluate the information provided in the logs and what value does this information have for students learning to write in French? Our study of student logs and corresponding compositions led to the development of an evaluation instrument designed to analyze various aspects of the FL writing process, and gain further insight into the use of the Système-D tracking device. This paper provides a review of how the logs provided data about student writing processes, a description of the evaluation instrument we developed, and suggestions for its use in teaching writing in French.

THE RESEARCH PROJECT

This research project involved 21 students enrolled in a third semester (intermediate level) French course during the spring semester of 1992. In keeping with the general requirements of this course, the students were asked to write a composition in French approximately every three weeks. For each composition, the students were required to revise their work and hand in a corrected version. During the course of the semester the students wrote four compositions and four revisions. All work was to be accomplished on the students' own time using Système-D. Since composition writing was an important part of this course, it counted for 25,Yo of the final grade.

For each writing assignment, the students were provided with specific tasks which included pertinent references to grammar, phrase, and vocabulary information in Système-D (Scott 1990). Table 1 gives an example of a task-oriented writing guideline.

7

Furthermore, prior to assigning each composition, the topics and tasks were discussed in class, giving the students the opportunity to ask questions or have any points clarified. The first version of each composition was due approximately ten days after it was assigned; the revised compositions were due approximately eight days after the first versions were evaluated and handed back to the students.

SITUATION: You are writing an informative article about American children for a French magazine. TASKS: 1 ) Comment est une famille américaine? Describe a typical American family. * PHRASES: Writing an essay; Describing people * VOCABULARY: Family members * GRAMMAR: Adjective agreement; Adjective position 2) Comment sont les enfants américains? Write what a typical American child does each day, telling what they do, don't do, how they look and dress. * PHRASES: Linking ideas; Weighing alternatives * VOCABULARY: Clothing; Games; Sports * GRAMMAR: Negation; Nouns after c'est, il est 3) Quand j'étais enfant ... Describe how you were as a child, what you did, the holidays and traditions that you liked, etc. * PHRASES: Sequencing events * VOCABULARY: Personality; Time expressions * GRAMMAR: Imparfait, Passé composé; Verbs with auxiliary être 4) Les enfants sont ... Conclude with a generalization about children. * PHRASES: Writing an essay * indicates the categories in Système-D

Table 1. Sample task-oriented writing guideline for Système-D

8

METHOD OF ANALYSIS

In evaluating each composition, both the product and the process were taken into consideration. With regard to the product, or the text itself, vocabulary, grammar, expression, organization, and originality were evaluated. Table 2 shows the five categories that were evaluated, each with a maximum of five points, for a total possible score of 25. Errors were indicated by using the code shown in Table 3.

In addition to an evaluation of the composition itself, the students' writing processes were analyzed and evaluated. The primary source of data was the Système-D log which indicates explicitly the following information:

• the time that the student began the writing session;

• dictionary inquiries in both French and English;

• verb conjugation and verb tense inquiries;

• inquiries for examples of words and expressions used in context;

• inquiries to the grammar, vocabulary, and phrase indexes;

• the time of each inquiry;

• the time that the student ended the writing session.

Table 4 shows a sample log with an explanation of the code.

9

10

Using the information provided in the Système-D log, we compared the compositions with their corresponding logs in order to determine if there were any obvious patterns which might provide information about the FL writing process. We recognized that the processes that precede the actual writing, such as planning, idea generation, and organizing would not be documented in the log. Moreover, we realized that the writing environment provided by Syst6me-D as well as the parameters prescribed by the writing task would affect the writing processes to a certain extent. Nevertheless several significant processes that take place during writing could be analyzed by using the information provided on the log.

An analysis of the logs and corresponding compositions revealed recurring patterns. It was clear that students writing with Syst6me-D used certain predictable strategies during the writing process. Table 5 lists the strategies that were observed, including strategies that we considered to be effective for FL writing (section A), as well as strategies which may have a negative effect on FL writing (section B).

11

The first item in section A of Table 5 is adherence to guidelines and simply indicates whether or not the student follows the tasks as outlined for the assignment and then accesses the Système-D information screens listed with the tasks.

Example inquiry, denotes how often the student looks up examples of words or structures used in context when the option is offered on the screen in Système-D.

The Conjugation inquiry refers to when a student checks the verb conjugation screens when looking up the meaning and/or use of a verb.

The French dictionary inquiry describes how often a student looks up words in the French dictionary, presumably to check gender, meaning or usage.

Circumlocution, in both English and French, indicates how often a student looks up a word or a phrase and either abandons the search if a suitable match cannot be found, or tries synonyms, or close approximations, until a word is found that 'feels right."

Browsing signifies the degree to which a student will take the time to look around in Système-D for additional information that is not required in the composition assignment tasks.

Error avoidance involves a student looking up a word or structure and then never using it in the composition, perhaps because the student is overwhelmed with the complexity of the word or structure or just does not like the information given.

A recursive approach, as opposed to a linear approach, is evidenced by a student's efforts to revise or add to any part of the composition at any time during the composing process.

Final revising means that the student accesses information screens in Système-D after having finished the composition, most likely in order to correct mistakes or verify the use of words or structures.

12

Finally, in evaluating the strategies listed in section A of Table 5, students were given a score of one through five. While we recognize that the terms “never," "seldom,” “average," "often," and “very often" are not absolute, we were able to determine, with a good degree of reliability, the significance of these terms for each composition assignment after analyzing approximately twenty compositions and logs.

In section B of Table 5, we include three additional kinds of information pertinent to the FL writing process. First of all, a student who frequently accesses the English dictionary is thought to have a high degree of dictionary dependence. Students who rely heavily on the dictionary while writing may consider that there is a one-to-one match between English and French and they are often translating literally. The quality of lexical inquiries was also noted in order to analyze a student's general awareness of language and culture.

For example, if a student looks up 'went" or "should" in the dictionary, we can infer something about the level of the student's linguistic competence. Likewise, if a student looks up “freshman" or “cheerleader” when no such concepts exist in French, we can recognize a certain cultural ignorance. For the section on time on task, the number of lines of text that a student produces is recorded along with the time spent on the entire writing assignment.

The corrected compositions were returned to the students along with the composition evaluation sheet and the log analysis sheet. They were given approximately one week in which to revise and embellish the compositions. The revision was then evaluated according to the same grading criteria as before, and the scores for the initial and revised compositions were averaged together. However, since students revised their compositions by editing their original work saved on a computer disk, the log for the revision provided very little additional information about the strategies they used during the writing process. The log analysis proved most useful in identifying the different strategies students were using when they wrote the first draft of their compositions.

13

RESULTS

Our interest in this research project lay primarily in determining whether the Système-D log of inquiries could be used to study the FL writing process. We discovered that certain strategies did appear on the students' logs and we developed an instrument which evaluated those strategies. Each of the 21 students in the research project wrote four compositions and four revisions; however only the first draft was used in our analysis of the strategies used in the FL writing process. During the analysis of these 84 compositions and the corresponding logs, we refined our evaluation instrument until it reflected the variety of strategies that we found. The final version of this evaluation instrument, as given in Table 5, is readily accessible to classroom teachers using Système-D to teach FL writing.

An unforeseen result of the research project was the effect of the log analysis on students' writing processes as they wrote their first drafts. All students involved in the project were given copies of their log analysis along with their composition evaluation. They all showed an interest in their log analyses, although some students reacted negatively at first because they saw it as an invasion of their privacy. For example, several students expressed concern at being held accountable for adhering to the specific Système-D task requirements. Others were slightly threatened by the fact that the hour of the day as well as the length of time spent writing were indicated on the log. However, we reassured them that the log analysis was being used strictly for research purposes and had no bearing on their grade. Once students' fears about the use of the logs were allayed, they began to show an interest in using the information provided on the log analysis sheet.

As students became familiar with the log analysis, we were able to identify how they began to incorporate new or improved strategies. For example, students were more careful to follow the prescribed tasks by accessing the recommended information screens in Système-D. In addition, they were more likely to check the “example" and “conjugation" screens for detailed information on correct usage. Also, nearly all the students engaged in a more systematic final rereading and revising of their work. Most notable was the change in dictionary usage. Many students began to rely on their intuition and look up words in French rather than immediately going to English, and general dictionary use declined significantly for all students.

14

A CASE STUDY

Only two students showed evidence of developing recursive writing strategies. We determined that since the compositions and the corresponding logs of these two students were of a significantly better quality than those of students who wrote in a directly linear fashion, they merited further study. The example below includes a portion of B.J.'s log and composition. The numbers 1-14 on the log correspond to the numbers indicated on the composition, thereby illustrating the use of recursive writing strategies.

15

By comparing B.J.'s log and composition, we can see that she puts together the pieces of her prose in a nonlinear way. She finds the words “ville" (1) and “divertissements" (2) at the beginning of the writing process but they don't appear in the composition until the third sentence. The words “entourée par" (3) and “magnifique" (4) appear in the fourth sentence. At this point, she looks up "means of transportation" and probably finds the word “avion,” which is used in the first and second sentences.

To describe her reaction to the plane flight, she finds the imperfect conjugation of “avoir" and writes "j'avais peur" (5) in the second sentence. She then searches for a way to say “to land,” first looking up "decollier" and getting a match, but then choosing to use “atterrir" (6) in the second sentence. She goes back and adds “l'avion a agité" (7), which is grammatically incorrect but very meaningful.

"Visité" (8) and “niveaux" (9) in the second paragraph indicate that she continued to describe what she did before adding the description about "les gens" (10) at the end of the first paragraph. She looks up “aussi” (11) and uses it in the third sentence of the second paragraph and then accesses information on describing weather and uses the information in the last two sentences of the first paragraph (12). We can see that she looks up “grimper” (13) and uses it in the second to last sentence and then finds the word “activité" to use in the first sentence of this paragraph. It is clear that B.J. seeks information in Système-D and uses it to piece together her intended meaning in a completely nonlinear fashion.

In addition to B.J.'s recursive approach to writing, we can also see evidence of other strategies. She circumlocutes in both English and French. She looks up "place" and is not happy with the match “asseoir” and uses 'city" instead. Later she uses her intuition and looks up "decollier" in French and gets the closest match "décoller" but chooses to use “land,” or “atterrir" instead. This may also be evidence of error avoidance.

Furthermore, she read the notes after looking up "city" and “gens” in order to see these words used in context. She is also willing to check the conjugation screens for several verbs to verify that she is spelling them correctly.

16

Ultimately, a careful analysis of B.J.'s composition and log suggests that she has good overall writing competence and that she is using effective writing strategies. She does not feel bound to a linear mode of thinking or writing. She continues to move back and forth from the present moment of composing to a previous moment in the writing process, which suggests that she is in a constant process of editing and revising the form and content of her messages as well as generating ideas while she is writing. She is continuously engaged in exploring the lexical and grammatical features of French, in formulating hypotheses, in making choices and then in incorporating them into her text. Finally, it is very possible that there is a link between her use of effective writing strategies and the quality of her writing in French.

CONCLUSION

This study represents our initial attempt to document the FL writing process. First of all, the use of the Système-D log provided us with a standard method of inquiry. In addition, preliminary analyses of the logs suggested that certain strategies are evident, making it possible to develop an instrument for evaluating students' FL writing process. Furthermore, when students were given the results of the log analysis, they often used the information to improve their written work. This led us to conclude that students could be explicitly taught to develop more strategies for FL writing.

The results of this preliminary study raise many questions about the FL writing process. Each of the strategies that we identified on the log analysis sheet merit further study, particularly in light of the suggestion that students can be explicitly taught to use them more effectively during the FL writing process. More specifically, a systematic analysis of the link between effective writing strategies and good writers might provide insight into improved approaches to teaching foreign language writing. Another issue which deserves further analysis is the effect that computer use has on the FL writing process.

Finally, we believe that this study demonstrates that the Système-D log is a useful tool for analyzing the FL writing process. This kind of analysis provides valuable insight into students' writing processes and points to a method of teaching FL writing which emphasizes both product and process.