The Pitfalls of Implementing TOEIC Preparation Courses
Kevin Miller - The Yoshinogawa Review, Vol. 26, 2003
The TOEIC (Test of English for International Communication) is an English language proficiency test developed and published by the Chauncey Group International, Ltd., a subsidiary of the American testing company, ETS, Inc. TOEIC touts itself as a “highly reliable and accurate” means of assessing a person’s English language ability (TOEIC, 2003). While TOEIC is administered all over the world, Japan is one of its larger markets. In recent years, the number of people in Japan taking the TOEIC test has increased steadily, and many Japanese companies are using TOEIC as a means of assessing their employees’ English proficiency, and some are even basing promotions on high scores (Businesspeople, 2003).
With participation in TOEIC increasing in Japan, there has been a corresponding increase in the number of TOEIC preparation courses offered at Japanese educational institutions. Likewise, an abundance of TOEIC preparation materials have been published. A search on the Japanese bookselling website, amazon.co.jp, reveals upwards of 1,000 volumes with TOEIC in their titles.
The combination of these factors—companies requiring TOEIC, schools promoting TOEIC preparation courses, and publishers printing massive numbers of TOEIC study books—has led many to feel a sense of urgency to jump on the TOEIC bandwagon. Individuals are eager to study for TOEIC, and schools that do not have TOEIC preparation as part of their curriculums are eager to add them, or perhaps convert existing English proficiency courses to TOEIC preparation courses. However, as useful as TOEIC may be, there are several pitfalls pertaining to TOEIC preparation that must be considered. They can be summarized as follows:
Negative washback
Washback (also referred to as backwash) is the effect of testing on teaching and learning. Researchers refer to positive and negative washback, depending on whether the testing has a beneficial or harmful impact on educational practices (Hughes, 1989). Positive washback from TOEIC would be if individuals are motivated by TOEIC to improve their overall English proficiency. Negative washback from TOEIC would be if schools neglect overall English proficiency as a teaching goal in favor of teaching TOEIC test-taking strategies.
To put the issue of washback in the starkest terms, consider the case of random urine testing for professional athletes. The use of random urine testing is based on the assumption that it’s unfair for athletes to take performance-enhancing substances, and that such usage should be discouraged. Positive washback from random urine testing, then, would mean athletes refrain from using banned substances for fear of being discovered by the urine test. Negative washback would mean athletes search for ways to undermine the test by getting false results; i.e., they try to test clean when in fact they have used banned substances.
Studying TOEIC test-taking strategies is certainly not as dubious an endeavor as trying to get false results on a urine test, but the issue of how TOEIC negatively impacts teaching goals is a serious one for educators. Robb and Ercanbrack (2003), quoting Alderson and Wall (1993), have identified four negative effects of testing; 1) a narrowing of the curriculum, 2) a decrease in instructional time, 3) a reduced emphasis on skills that require complex thinking, and 4) test score “pollution,” where test scores go up without actual improvement in skills.
These negative effects can be seen when schools attempt to turn English proficiency classes, such as General English classes found at universities throughout Japan, into TOEIC preparation courses. Instead of focusing on such areas as oral communication or extensive reading, both of which are needed in Japanese education, the focus turns to analyzing discrete grammar items out of context, and to test-taking strategies, such as using the multiple choice format to one’s advantage in eliminating distractors. Japanese education was once criticized for the widespread use of grammar-translation as a method, but by turning to TOEIC preparation instead, it appears that grammar teaching has merely cloaked itself in a more marketable exterior.
To be fair, however, implementation of TOEIC preparation courses can sometimes be considered positive washback. There are four types of listening exercises in the TOEIC format, each of which tests listening in a different way. To score well on TOEIC, students must be able to match oral descriptions to photographs, listen for specific information, listen for gist, and identify appropriate responses to questions. If students can master these skills, they are apt to show a corresponding improvement in listening proficiency. Additionally, since many Japanese university classes feature content that is even less communicative than TOEIC preparation—i.e., they focus on single sentence translation or analysis of obscure poems from classical literature—then TOEIC preparation can be termed a nominal step in the direction of greater communicativity.
There is also at least one legitimate reason why students may wish to take a TOEIC preparation course, and why educators should encourage them to do so—a thorough knowledge of the test format is important for test takers if they are to display their true proficiency. Even individuals of advanced ability cannot be expected to perform at their best if they are encountering a test format for the first time.
These instances of positive washback notwithstanding, educators should be judicious in their implementation of TOEIC preparation courses. TOEIC preparation should not take the place of general English proficiency courses, but supplement them. Additionally, they should not be required of all students, but should be elective courses for those students who have their own motivation for preparing for the TOEIC test.
Inappropriateness of TOEIC preparation for low-level students
TOEIC is appropriately used by businesses to evaluate the proficiency of their employees, often for the purpose of assessing their fitness for overseas positions. Consequently, the language used on the test is very natural, and focuses on business situations as well as daily conversation. The speed of the language on the tape can be considered natural speed for native speakers, which is often perceived as fast for non-native speakers. It also includes advanced vocabulary and idioms that learners acquire only after extensive English study.
Writers of TOEIC preparation materials generally try to duplicate the level and content of the TOEIC test, although they may also include some stepped exercises that build up to the actual level. Low-level students encountering the natural speed, advanced vocabulary, and complex sentence structure of typical TOEIC preparation materials are apt to feel frustrated, and with good reason. They can be better served by taking beginning courses in English proficiency and building their skills to the point where they can make use of authentic TOEIC preparation materials.
Recently, some materials writers have attempted to make low-level TOEIC preparation books, but with mixed results. Low-level TOEIC is a contradiction in terms, since TOEIC is not low-level. The writers try to approximate the TOEIC format, but use only the easiest vocabulary and sentence structures. Such restrictions on language make it extremely difficult for them to write plausible distractors, particularly on the listening section. One solution that some writers adopt is to write absurd distractors that are easier for low-level students to identify. Consider the following distractors from the photo section of two lower level TOEIC preparation textbooks:
Distractors like these would never appear on an actual TOEIC test. Their inclusion in TOEIC preparation materials gives the learner the impression that eliminating absurd answers is a legitimate test-taking strategy, when it is not.
Finding appropriate TOEIC preparation materials for low-level students has been notoriously difficult, but the problem stems from the fact that low-level students have been needlessly directed towards TOEIC study. Understanding this, the Chauncey Group has recently begun a new test called TOEIC Bridge, which purports to assess the language abilities of beginning and intermediate learners. It also aims to help these students find which areas of English they need to improve. While TOEIC Bridge has not yet become a household word, like TOEIC, it’s possible in the future that schools will feel less pressure to push low-level students to take the TOEIC test, but offer the TOEIC Bridge as an alternative. Hopefully, the washback from this test will mean less emphasis on TOEIC preparation for low-level students.
Educators have a responsibility to make effective curriculum design decisions for all levels of students. Promoting TOEIC preparation to low-level students does them a disservice and sends two erroneous messages—that studying test items is the best way to achieve English proficiency, and that the main purpose of English study is to score well on a test. Schools would do better to promote general English proficiency and emphasize that English study is important for many reasons besides test-taking. For students who are personally motivated to do test preparation, however, schools should guide students to choose the appropriate level of test, TOEIC or TOEIC Bridge.
Poorly written materials
When the Chauncey Group develops its TOEIC tests, it employs a vast team of trained writers who submit test items that are then chosen for inclusion on the test or rejected as unsuitable. Test items are also tested before inclusion on actual TOEIC tests. A rigorous selection process is necessary in order to maintain the standard of reliability of the test. The huge TOEIC materials publishing industry, on the other hand, tends to be much less rigorous. Self-study customers can rarely tell by glancing at a book in a bookstore if the practice test items are comparable to the actual test or not. TOEIC preparation course teachers as well are unlikely to know how well the items in textbooks duplicate actual test questions, and most decisions to buy a book or adopt a textbook for a course are based on such issues as cost, availability, and visual presentation. As a result, there is a wide range of quality among TOEIC preparation materials, and there are many instances of poorly written test exercises, particularly on the listening section.
People choosing TOEIC preparation materials should be aware that:
1) No answers on the actual TOEIC listening test can be eliminated because they are absurd or ungrammatical. Two examples of poorly written preparation materials were given in the previous section. Some additional examples are:
2) In the actual TOEIC photographs section, all answers, including distractors, are capable of being represented visually. To put it another way, no answer can be eliminated without checking the answer against the photograph. Poor examples from preparation materials include:
3) The actual TOEIC test does not ask the test taker to predict what people in the photographs will do in the future. Poor examples from preparation materials include:
4) On the actual TOEIC test, technical or rarely used words will not appear in the answer choices. A poor example from preparation materials is:
There is no easy solution to the problem of poorly written TOEIC preparation materials. Often it is not possible to tell how well practice test items and exercises are written until one has gone through the book item by item. One solution is to use materials promoted by the Chauncey Group itself, which can be presumed to be representative of the actual test. Otherwise, one can turn to the answer key of the photographs section of a prospective book and look for items that violate the principles stated above. If the casual reader can identify more than one or two poorly written items, than the book should be considered suspect.
Conclusion
TOEIC has proven to be a reliable test of English proficiency, but there has been negative washback as a result of its popularity in Japan. Educators need not support the TOEIC industry by emphasizing TOEIC preparation courses for all students at all levels. Rather this distracts from the goal of promoting general English proficiency. It’s more important for most students to work on their communicative skills, such as speaking, listening, pronunciation, and intonation. Extensive reading, as well as writing to an audience, are two other important areas that should be emphasized. Schools need to promote the benefits of being able to use English in various walks of life rather than reduce English study to English for test-taking purposes. Finally, students who are really interested in TOEIC preparation should have the opportunity to practice the TOEIC test format, but on a voluntary basis, and with the best, most authentic materials available, and at the proper level. TOEIC is a useful tool of assessment, but educators must take steps to keep it from becoming “the tail that wags the dog.” If so, we will be able to avoid the pitfalls of TOEIC preparation, and instead see positive benefits from TOEIC, namely increased motivation to improve overall English proficiency.
References
Amazon.co.jp, (2003). http://www.amazon.co.jp, retrieved Jan. 2003.
Bailey, K. M., (1998). Learning About Language Assessment, Newbury House.
Businesspeople Confront the English Divide, http://origin.island.lk/2001/12/04/busine07.html, retrieved Jan. 2003.
Cohen, A. D. (1994). Assessing Language Ability in the Classroom, Heinle & Heinle Publishers.
Gilfert, S., and Kim, V. (1996) TOEIC Strategies, Macmillan LanguageHouse.
Hughes, A. (1989). Testing for Language Teachers, Cambridge University Press, p. 1.
Kimura, T., Johansson, M.W., and Kimura, T. (1994) 5-Minute Quizzes For TOEIC Listening, Macmillan LanguageHouse.
Robb, T. N. and Ercanbrack, J., “A Study of the Effect of Direct Test Preparation on the TOEIC Scores of Japanese University Students,” http://www.kyoto-su.ac.jp/~trobb/toeic.html, retrieved Jan. 2003.
TOEIC – The measure of success, http://www.toeic.com/2tests.htm, retrieved Jan. 2003.
TOEIC Bridge, http://www.toeic.com/2_2_4bridge.htm, retrieved Jan. 2003.
Wakabayashi, Y. (1996). The AEON One-Month TOEIC Course, Longman.
The TOEIC (Test of English for International Communication) is an English language proficiency test developed and published by the Chauncey Group International, Ltd., a subsidiary of the American testing company, ETS, Inc. TOEIC touts itself as a “highly reliable and accurate” means of assessing a person’s English language ability (TOEIC, 2003). While TOEIC is administered all over the world, Japan is one of its larger markets. In recent years, the number of people in Japan taking the TOEIC test has increased steadily, and many Japanese companies are using TOEIC as a means of assessing their employees’ English proficiency, and some are even basing promotions on high scores (Businesspeople, 2003).
With participation in TOEIC increasing in Japan, there has been a corresponding increase in the number of TOEIC preparation courses offered at Japanese educational institutions. Likewise, an abundance of TOEIC preparation materials have been published. A search on the Japanese bookselling website, amazon.co.jp, reveals upwards of 1,000 volumes with TOEIC in their titles.
The combination of these factors—companies requiring TOEIC, schools promoting TOEIC preparation courses, and publishers printing massive numbers of TOEIC study books—has led many to feel a sense of urgency to jump on the TOEIC bandwagon. Individuals are eager to study for TOEIC, and schools that do not have TOEIC preparation as part of their curriculums are eager to add them, or perhaps convert existing English proficiency courses to TOEIC preparation courses. However, as useful as TOEIC may be, there are several pitfalls pertaining to TOEIC preparation that must be considered. They can be summarized as follows:
- Negative washback
- Inappropriateness of TOEIC for low-level students
- Poorly written materials
Negative washback
Washback (also referred to as backwash) is the effect of testing on teaching and learning. Researchers refer to positive and negative washback, depending on whether the testing has a beneficial or harmful impact on educational practices (Hughes, 1989). Positive washback from TOEIC would be if individuals are motivated by TOEIC to improve their overall English proficiency. Negative washback from TOEIC would be if schools neglect overall English proficiency as a teaching goal in favor of teaching TOEIC test-taking strategies.
To put the issue of washback in the starkest terms, consider the case of random urine testing for professional athletes. The use of random urine testing is based on the assumption that it’s unfair for athletes to take performance-enhancing substances, and that such usage should be discouraged. Positive washback from random urine testing, then, would mean athletes refrain from using banned substances for fear of being discovered by the urine test. Negative washback would mean athletes search for ways to undermine the test by getting false results; i.e., they try to test clean when in fact they have used banned substances.
Studying TOEIC test-taking strategies is certainly not as dubious an endeavor as trying to get false results on a urine test, but the issue of how TOEIC negatively impacts teaching goals is a serious one for educators. Robb and Ercanbrack (2003), quoting Alderson and Wall (1993), have identified four negative effects of testing; 1) a narrowing of the curriculum, 2) a decrease in instructional time, 3) a reduced emphasis on skills that require complex thinking, and 4) test score “pollution,” where test scores go up without actual improvement in skills.
These negative effects can be seen when schools attempt to turn English proficiency classes, such as General English classes found at universities throughout Japan, into TOEIC preparation courses. Instead of focusing on such areas as oral communication or extensive reading, both of which are needed in Japanese education, the focus turns to analyzing discrete grammar items out of context, and to test-taking strategies, such as using the multiple choice format to one’s advantage in eliminating distractors. Japanese education was once criticized for the widespread use of grammar-translation as a method, but by turning to TOEIC preparation instead, it appears that grammar teaching has merely cloaked itself in a more marketable exterior.
To be fair, however, implementation of TOEIC preparation courses can sometimes be considered positive washback. There are four types of listening exercises in the TOEIC format, each of which tests listening in a different way. To score well on TOEIC, students must be able to match oral descriptions to photographs, listen for specific information, listen for gist, and identify appropriate responses to questions. If students can master these skills, they are apt to show a corresponding improvement in listening proficiency. Additionally, since many Japanese university classes feature content that is even less communicative than TOEIC preparation—i.e., they focus on single sentence translation or analysis of obscure poems from classical literature—then TOEIC preparation can be termed a nominal step in the direction of greater communicativity.
There is also at least one legitimate reason why students may wish to take a TOEIC preparation course, and why educators should encourage them to do so—a thorough knowledge of the test format is important for test takers if they are to display their true proficiency. Even individuals of advanced ability cannot be expected to perform at their best if they are encountering a test format for the first time.
These instances of positive washback notwithstanding, educators should be judicious in their implementation of TOEIC preparation courses. TOEIC preparation should not take the place of general English proficiency courses, but supplement them. Additionally, they should not be required of all students, but should be elective courses for those students who have their own motivation for preparing for the TOEIC test.
Inappropriateness of TOEIC preparation for low-level students
TOEIC is appropriately used by businesses to evaluate the proficiency of their employees, often for the purpose of assessing their fitness for overseas positions. Consequently, the language used on the test is very natural, and focuses on business situations as well as daily conversation. The speed of the language on the tape can be considered natural speed for native speakers, which is often perceived as fast for non-native speakers. It also includes advanced vocabulary and idioms that learners acquire only after extensive English study.
Writers of TOEIC preparation materials generally try to duplicate the level and content of the TOEIC test, although they may also include some stepped exercises that build up to the actual level. Low-level students encountering the natural speed, advanced vocabulary, and complex sentence structure of typical TOEIC preparation materials are apt to feel frustrated, and with good reason. They can be better served by taking beginning courses in English proficiency and building their skills to the point where they can make use of authentic TOEIC preparation materials.
Recently, some materials writers have attempted to make low-level TOEIC preparation books, but with mixed results. Low-level TOEIC is a contradiction in terms, since TOEIC is not low-level. The writers try to approximate the TOEIC format, but use only the easiest vocabulary and sentence structures. Such restrictions on language make it extremely difficult for them to write plausible distractors, particularly on the listening section. One solution that some writers adopt is to write absurd distractors that are easier for low-level students to identify. Consider the following distractors from the photo section of two lower level TOEIC preparation textbooks:
- “They are riding their bikes in the sea.” — Aeon One-Month TOEIC Course, p. 194.
- “The water is upside-down.” — 5-Minute Quizzes for TOEIC Listening, p. 2 of answer booklet.
Distractors like these would never appear on an actual TOEIC test. Their inclusion in TOEIC preparation materials gives the learner the impression that eliminating absurd answers is a legitimate test-taking strategy, when it is not.
Finding appropriate TOEIC preparation materials for low-level students has been notoriously difficult, but the problem stems from the fact that low-level students have been needlessly directed towards TOEIC study. Understanding this, the Chauncey Group has recently begun a new test called TOEIC Bridge, which purports to assess the language abilities of beginning and intermediate learners. It also aims to help these students find which areas of English they need to improve. While TOEIC Bridge has not yet become a household word, like TOEIC, it’s possible in the future that schools will feel less pressure to push low-level students to take the TOEIC test, but offer the TOEIC Bridge as an alternative. Hopefully, the washback from this test will mean less emphasis on TOEIC preparation for low-level students.
Educators have a responsibility to make effective curriculum design decisions for all levels of students. Promoting TOEIC preparation to low-level students does them a disservice and sends two erroneous messages—that studying test items is the best way to achieve English proficiency, and that the main purpose of English study is to score well on a test. Schools would do better to promote general English proficiency and emphasize that English study is important for many reasons besides test-taking. For students who are personally motivated to do test preparation, however, schools should guide students to choose the appropriate level of test, TOEIC or TOEIC Bridge.
Poorly written materials
When the Chauncey Group develops its TOEIC tests, it employs a vast team of trained writers who submit test items that are then chosen for inclusion on the test or rejected as unsuitable. Test items are also tested before inclusion on actual TOEIC tests. A rigorous selection process is necessary in order to maintain the standard of reliability of the test. The huge TOEIC materials publishing industry, on the other hand, tends to be much less rigorous. Self-study customers can rarely tell by glancing at a book in a bookstore if the practice test items are comparable to the actual test or not. TOEIC preparation course teachers as well are unlikely to know how well the items in textbooks duplicate actual test questions, and most decisions to buy a book or adopt a textbook for a course are based on such issues as cost, availability, and visual presentation. As a result, there is a wide range of quality among TOEIC preparation materials, and there are many instances of poorly written test exercises, particularly on the listening section.
People choosing TOEIC preparation materials should be aware that:
1) No answers on the actual TOEIC listening test can be eliminated because they are absurd or ungrammatical. Two examples of poorly written preparation materials were given in the previous section. Some additional examples are:
- “They are harboring wheat.”— Aeon One-Month TOEIC Course, p. 191.
- “The two businessmen are discussing on the table.” — TOEIC Strategies, p. 250.
- “The two boys are sanding the beach.” — TOEIC Strategies, p. 251.
- “There are various baits on the water.” — TOEIC Strategies, p. 252.
- “The man is seeking a good bed for his luggage.” — 5-Minute Quizzes for TOEIC Listening, p. 2 of answer booklet.
2) In the actual TOEIC photographs section, all answers, including distractors, are capable of being represented visually. To put it another way, no answer can be eliminated without checking the answer against the photograph. Poor examples from preparation materials include:
- “The man is a veteran doctor.” — Aeon One-Month TOEIC Course, p. 191.
- “The man is a vegetarian.” — Aeon One-Month TOEIC Course, p. 191.
- “Three bad but wise people are having veal.” — Aeon One-Month TOEIC Course, p. 191.
- “This is a superior market.” — TOEIC Strategies, p. 251.
3) The actual TOEIC test does not ask the test taker to predict what people in the photographs will do in the future. Poor examples from preparation materials include:
- “The woman will not buy the newspaper.” — TOEIC Strategies, p. 252.
- “A man is waiting to rob the bank.” — TOEIC Strategies, p. 347.
4) On the actual TOEIC test, technical or rarely used words will not appear in the answer choices. A poor example from preparation materials is:
- “The gull is standing near the eaves.” — 5-Minute Quizzes for TOEIC Listening, p. 2 of answer booklet. [Neither gull nor eaves would likely appear in the actual TOEIC photographs section.]
There is no easy solution to the problem of poorly written TOEIC preparation materials. Often it is not possible to tell how well practice test items and exercises are written until one has gone through the book item by item. One solution is to use materials promoted by the Chauncey Group itself, which can be presumed to be representative of the actual test. Otherwise, one can turn to the answer key of the photographs section of a prospective book and look for items that violate the principles stated above. If the casual reader can identify more than one or two poorly written items, than the book should be considered suspect.
Conclusion
TOEIC has proven to be a reliable test of English proficiency, but there has been negative washback as a result of its popularity in Japan. Educators need not support the TOEIC industry by emphasizing TOEIC preparation courses for all students at all levels. Rather this distracts from the goal of promoting general English proficiency. It’s more important for most students to work on their communicative skills, such as speaking, listening, pronunciation, and intonation. Extensive reading, as well as writing to an audience, are two other important areas that should be emphasized. Schools need to promote the benefits of being able to use English in various walks of life rather than reduce English study to English for test-taking purposes. Finally, students who are really interested in TOEIC preparation should have the opportunity to practice the TOEIC test format, but on a voluntary basis, and with the best, most authentic materials available, and at the proper level. TOEIC is a useful tool of assessment, but educators must take steps to keep it from becoming “the tail that wags the dog.” If so, we will be able to avoid the pitfalls of TOEIC preparation, and instead see positive benefits from TOEIC, namely increased motivation to improve overall English proficiency.
References
Amazon.co.jp, (2003). http://www.amazon.co.jp, retrieved Jan. 2003.
Bailey, K. M., (1998). Learning About Language Assessment, Newbury House.
Businesspeople Confront the English Divide, http://origin.island.lk/2001/12/04/busine07.html, retrieved Jan. 2003.
Cohen, A. D. (1994). Assessing Language Ability in the Classroom, Heinle & Heinle Publishers.
Gilfert, S., and Kim, V. (1996) TOEIC Strategies, Macmillan LanguageHouse.
Hughes, A. (1989). Testing for Language Teachers, Cambridge University Press, p. 1.
Kimura, T., Johansson, M.W., and Kimura, T. (1994) 5-Minute Quizzes For TOEIC Listening, Macmillan LanguageHouse.
Robb, T. N. and Ercanbrack, J., “A Study of the Effect of Direct Test Preparation on the TOEIC Scores of Japanese University Students,” http://www.kyoto-su.ac.jp/~trobb/toeic.html, retrieved Jan. 2003.
TOEIC – The measure of success, http://www.toeic.com/2tests.htm, retrieved Jan. 2003.
TOEIC Bridge, http://www.toeic.com/2_2_4bridge.htm, retrieved Jan. 2003.
Wakabayashi, Y. (1996). The AEON One-Month TOEIC Course, Longman.
Some online articles that reference "The Pitfalls of Implementing TOEIC Preparation Courses"
- TOEIC: A Discussion and Analysis (2004) - Timothy M. Nall
- How Can We Help Students Improve Their TOEIC Scores Without “Teaching to the Test”? - Cara Phillips
- Learner perceptions of TOEIC test results and language skill improvements:“I don't want to study English, I want to study TOEIC.” (2006) - Howard Brown
- TOEIC: Uses and Misuses (2007) - Alan Bessette
- A Study on Effectiveness of College English-Featured Courses on TOEIC (2008) - Yi-Hsiu Lai
- Enhancing Students Communicative Competency and Test-Taking Skills Through TOEIC Preparatory Materials (2010) - Yi-Ching Pan
- An Overview of the Issues on Incorporating the TOEIC test into the University English Curricula in Japan (2012) - Junko Takahashi
- The Sociocultural Activity of High Stakes Standardised Language Testing (2018) - Dawn Karen Booth