Abbreviations used: CVMT, Continuous Visual Memory Test; MCS, multiple chemical sensitivities; MMPI-2, Minnesota Multiphasic Personality Inventory-2; NART-R, National Adult Reading Test-Revised; NES Battery, neurobehavioral evaluation system; PEA, phenyl ethyl alcohol; SBS, sick-building syndrome; UPSIT, University of Pennsylvania Smell Identification Test; VOC, volatile organic compound; WAIS-R, Wechsler Adult Intelligence Scale-Revised; WHO, World Health Organization.

Introduction
Extensive literature documents the range of neuropsychological tests available to detect and evaluate symptoms of exposure to neurotoxicants. By 1990 approximately 250 different tests had been used to evaluate the effects of neurotoxicants on behavior (1). Numerous articles have been written in which the range of functions assessed and the specific tests used to assess those functions are reviewed (1-5). Tests applied to evaluate symptoms among humans derive from two different psychological approaches: traditional neuropsychological testing to diagnose brain dysfunction and experimental cognitive psychology. Neuropsychology has traditionally focused on tests to identify deficits due to pathology (6), while methods within cognitive psychology have been developed to elucidate normal cognitive processes involved in processing of information and learning (7). This paper will briefly review the range of tests used to assess human responses to neurotoxicants, discuss measurement of individual susceptibility as a factor in establishing risk status among patients with sensitivities to low-level exposures, and consider changes in current methods that will advance measurement of toxic symptoms at low-level exposures.

Review of Current Tests
Traditional neuropsychological tests such as subtests from the Wechsler Adult Intelligence Scale-Revised (WAIS-R) (8) and the Halstead-Reitan (9) proved to be sufficiently sensitive to document cognitive deficits among workers with chronic exposure to neurotoxicants such as organic solvents and lead (10,11). Increasingly, concerns have been raised regarding the sensitivity of these tests for detecting subtle cognitive deficits at lower level exposures (12). Nevertheless, several batteries, incorporating many of the traditional neuropsychological tests, are widely used for clinical and research purposes (13).

While batteries vary in the specific tests used, the cognitive functions assessed are relatively consistent. Table 1 contains a list of the functions assessed and a sample of the tests used to assess these functions. Tests have been somewhat arbitrarily categorized into functional categories; however, as several other reviewers have suggested, any one test typically relies on more than one function for performance (5,6). For example, even relatively simple tasks such as simple reaction time require not only attention and concentration but also motor speed for accurate and quick responding. Thus, tests overlap functional categories.

Overall Ability--Verbal

Tests of overall verbal ability such as vocabulary tests [e.g., Vocabulary-WAIS-R (8)], multiple choice vocabulary (16), or reading scores (e.g., NART-R) (14) are used to estimate premorbid ability. Some studies of chronic organic solvent or lead exposures have suggested that such exposure results in a general dementia affecting all aspects of cognitive function including word knowledge and general information (25,26). However, most studies cite verbal ability tests as methods that are relatively insensitive to neurotoxicants (6,13). A consistent problem in studies of neurotoxicants is the lack of baseline intellectual function before exposure. Therefore, tests of current verbal ability are used as surrogates for preexposure ability.

 
Overall Ability--Spatial Relations

Another broad class of tests of ability are those assessing spatial relations such as block design from the WAIS-R (8) and Raven's Progressive Matrices (15). Block design has been used extensively to evaluate the effects of lead and solvents with mixed results (27,28). Raven's Progressive Matrices (15) was designed to assess overall intellectual ability by presenting visuospatial conceptual problems rather than verbal conceptual problems (e.g., Similarities-WAIS-R) and, although less widely used, it has been sensitive to the effects of neurotoxicants (e.g., mercury) (29).

Concentration/Attention

Tests of concentration and attention assess the ability to orient and sustain attention to either visual or auditory stimuli. This ability is the precursor to learning and memory, two functions emphasized in all batteries of neuropsychological tests. Tasks of concentration/attention range from simple reaction time in response to simple auditory or visual stimuli to more complex tasks in which the individual must sustain attention to the target stimulus when distractors are present, such as the Stroop Color-Word Task (17), or signal detection tasks, such as the Continuous Performance Test (16). Representative tests of attention/concentration and vigilance are included in most studies of neurotoxicants and are a part of test batteries applied to worksite testing [e.g., FIOH Battery (30), WHO Battery (31), London School of Hygiene Battery (32)]. Gamberale (2) cites simple reaction time as the most sensitive test for detecting behavioral performance effects due to solvent exposure.

Motor Skills

As might be expected, tests of cognitive skills are more plentiful than tests of motor skills, particularly tests of gross as opposed to fine motor skills. Tests of motor skills assess speed and dexterity by asking the subject to place pegs in holes while being timed [e.g., grooved pegboard (9), Santa Ana Pegboard (33)] or measure strength of grip by pressure against a spring-loaded device (dynamometer) (9). Finger tapping (9) is another simple test of coordination and speed that has been widely used and found sensitive to the effects of neurotoxicants (34).

Visuomotor Skills

Another category of tests are those designed to assess visuomotor skills. At some level, tests of attention also require motor skills since coordination between the perception of visual stimuli and initiation of motor movement is necessary for response. However, tests of visuomotor skills typically involve more complex levels of motor coordination in response to visual stimuli. For example, the hand-eye coordination test from the Neurobehavioral Evaluation System-NES battery (16) tests the ability to move a computer cursor with a joystick along a sine wave pattern on a screen at a constant rate of speed. Another somewhat more complex, verbally mediated test of eye-hand coordination, widely used in this literature, is Digit Symbol (8). This coding task requires that the subject code symbols with letters while being timed. It has been sensitive to the effects of neurotoxicants (e.g., lead, solvents, mercury) (35-37). This test is also included in most of the test batteries used in worksite testing [e.g., FIOH Battery (30); TUFF (38)].

Memory

Numerous tests of memory have been used to assess neurotoxicant effects. These tests range from those assessing memory for abstract visual designs (e.g., Wechsler Memory Scale-Revised) (18) to tests of memory for verbal materials such as words or numbers [Paired Associates (18), Digit Span (8)]. The methodology for these tests usually involves presentation of the stimulus (e.g., drawing or word list) to be encoded. The subject is asked to recall the stimulus immediately after presentation as well as after a relatively short delay (e.g., 30 min). A more recent development in memory testing incorporates the tradition of cognitive psychology by not only providing a global indicator of memory (e.g., total score based on the quantity remembered) but also by scoring various indicators of memory processes (e.g., slope of learning curve, proactive and retroactive interference) (19). Assessment of learning curves and memory processes may provide more insight regarding subtle effects due to relatively low-level exposures and may help elucidate this frequent complaint that is often not substantiated by global tests of memory.

Sensory

Sensory tests are not as plentiful and have not received as much emphasis in the literature on neurotoxicants as many of the tests of cognition and memory. Tests of audition range from simple tests of hearing with an audiometer to more complex tests assessing the ability to discern speech or rhythmic patterns (e.g., seashore rhythm, speech perception) (9). Tests of tactile perception and vibration sense include simple tactile perception (finger agnosia) (9) and sense of vibration using a device (21) to measure perception of fine vibrations in the finger or toe. These tests have evaluated loss of peripheral sensory perception due to mercury or solvents. Of more recent interest is the finding of color vision loss among solvent-exposed workers (39,40). Finally, altered sense of smell due to neurotoxicants has received more attention recently (28,41). Tests of olfactory discrimination [University of Pennsylvania Smell Identification Test (UPSIT)] (22) and of olfactory threshold detection have been used to evaluate the sense of smell.

Affect and Personality

Numerous questionnaires and standardized tests have been used to assess mood and personality factors that may be affected by exposure to neurotoxicants. In fact, mood is reported to be one of the first aspects of functioning where changes due to neurotoxicants can be observed. Irritability, depression, and lability are mood changes that are reported to occur in the first stages of solvent neurotoxicity (26). Representative scales that have been used include the Profile of Mood States (23), Minnesota Multiphasic Personality Inventory-2 (24), and Beck Depression Inventory (42). Unfortunately, these measures rely on self-report of symptoms rather than any objective indicator of mood. Therefore, they are subject to reporting biases that may be influenced by the circumstances in which the individual is being evaluated (e.g., litigation).

 
Discussion

While numerous cross-sectional studies have been conducted using various combinations of neurobehavioral measures to sample each of the domains listed above, results from these studies have been mixed. Some investigators have attempted to identify patterns of test performance specific to classes of neurotoxicants (43,44); however, it has been difficult to determine a consistent pattern of performance. Numerous reviews have appeared in which the tests and the exposures evaluated are listed (5,45). Some general impressions can be formed from these reviews suggesting that behavioral effects are observed for a number of neurotoxicants. It is difficult, however, to determine from these more qualitative reviews which tests are most sensitive in detecting effects. Further, little information is available to document the predictive validity of these tests for performance in the workplace. Therefore, even if we can say that a test detects differences in performance between exposed and nonexposed groups, the meaning of the performance difference has not been adequately addressed. Information about this issue could provide the most compelling evidence for controlling or reducing exposures.

At this point, there may be sufficient literature on some organic solvents and heavy metals such as lead to conduct metaanalyses of the results across studies. These statistical methods have been used in other literatures to help consolidate disparate findings into a more cohesive picture. These methods could help clarify which tests are most sensitive for detecting effects due to specific neurotoxicants (46).

For the field of neurobehavioral toxicology to make meaningful advances in our understanding of the behavioral effects of neurotoxicants, more refined studies will be needed. Such studies will also require that neurobehavioral methods be improved. For example, rather than continue cross-sectional studies, prospective studies need to be developed in which workers are followed over a period of time to assess changes from baseline. These will require a better understanding of the behavior of neuropsychological tests under repeated measures conditions. Otto et al. (4) found significant practice effects for several of the tests on the NES battery. To avoid ceiling effects on these tests after repeated administration, he suggested that test parameters be altered to make the tasks more difficult and better suited to repeated measures design (4). Similarly, increasing demands are being made for neuropsychological methods to assess subtle effects in acute and unusually low-exposure circumstances. These conditions also require an increased sensitivity in neuropsychological test methods.

In summary, we need to take a more systematic approach toward identifying the most sensitive tests among those cited frequently in the literature. We then need to test the suitability of these tests for the study designs proposed to address present concerns such as low-level exposures. Further consideration of the parameters to be considered in these studies will be addressed in the subsequent sections of this paper.

Individual Susceptibility

The test literature reviewed previously provides an overview of the broad range of tests and functional categories that have been included in the literature on the effects of neurotoxicants. While some tests appear to be more sensitive than others, it is difficult to develop a clear picture due to the large variability in the demographic profiles of the subject groups evaluated, the range of different substances to which these groups were exposed, and the lack of clarity regarding duration and intensity of exposures. Despite these factors, several attempts have been made to develop batteries of tests that can be coordinated across studies [e.g., WHO battery (31), NES battery (16)], thus allowing direct comparisons between studies. This effort to increase comparability is to be applauded. However, in an effort to be broadly applicable, these batteries may prove insensitive for unique populations or exposure situations.
For example, an increasing number of patients have vague complaints, including poor concentration and memory, in response to low-level chemical exposures. This symptom complex, labeled multiple chemical sensitivities (MCS), involves symptoms reflective of multiple organ systems, most prominently the nervous system. The question of whether these patients are uniquely susceptible to chemicals or are a variant of the psychiatric disorder, somatization, is frequently debated (47-49).

MCS patients may present unique susceptibilities to chemicals for several reasons. First, while no epidemiological studies have been conducted to date, most of the investigators observe that approximately 80% of these patients are women (49,50). This is in contrast to the literature on the neuropsychological effects of neurotoxicants, which is based largely on men. In one of the few studies of women, Parkinson et al. (51) reported no significant differences between solvent-exposed blue-collar women and controls on a relatively brief battery of standard neuropsychological tests. However, the highest exposure levels were significantly related to a number of neurologic and somatic symptoms including depression and headaches. When symptoms are reported by women, they are more likely to be attributed to psychosomatic causes such as stress rather than to physiologically based conditions (52). This is particularly true when objective tests do not substantiate symptom reports. However, it is also possible that women may have unique susceptibilities that wax and wane due to hormonal cycles not occurring in men. For example, women can vary in olfactory acuity according to hormonal cycles (53). Alternatively, women may simply be more aware of and likely to report symptoms that occur in response to an exposure or an illness than men. That is, women may be better observers of the early signs of physiologic changes (54). The challenge is to develop methodologies to measure these changes.

Second, MCS patients have a higher rate of psychiatric disorder (e.g., depression, anxiety) concurrent with and before the onset of MCS (49,50). Many use this information to suggest that MCS is not a unique susceptibility but simply a psychiatric condition that is attributed to chemicals. On the other hand, one may question whether individuals with psychiatric conditions are more susceptible to the effects of neurotoxicants. For example, Morrow et al. (55) reported that individuals with higher levels of psychological distress on the MMPI-2 were associated with poorer neuropsychological function at follow-up. From this study it is impossible to know whether the symptoms on the MMPI-2 were a reflection of continuing neurologic symptoms due to exposure or a premorbid personality style. Psychiatric and personality function, as a risk factor for the effects of neurotoxicants, has infrequently been evaluated and needs further exploration.

Only two studies to date have used standardized neuropsychological tests to evaluate the cognitive complaints of MCS patients (50,56). Overall, these cross-sectional studies did not find differences between the MCS and control groups (i.e., musculoskeletal patients, normal controls) on tests of concentration, memory, and visuomotor skills. However, these tests were not administered under controlled exposure conditions. A primary question is how to test the responses of MCS patients objectively. The typical evaluation paradigm in which the patient's neuropsychological performance and physical status is assessed at an arbitrary point in time is not likely to capture the symptomatic response that these patients observe in themselves under exposure conditions.

Studies more directly relevant to investigation of responses among MCS patients are exposure chamber studies with sick-building syndrome (SBS) patients (57,58). These patients are similar to MCS patients in that they are otherwise healthy individuals who report sensitivities in response to indoor air mixtures that other individuals apparently tolerate. Two controlled exposure studies evaluated the effects of a mixture of 22 volatile organic compounds (VOCs) on sick-building syndrome patients relative to asymptomatic controls (57,58). Along with increasing symptom reports of irritation with increasing VOC exposure (0, 5, 25 mg/m3), Molhave et al. (57) reported reduced performance on digit span among SBS subjects. This finding was not replicated, however, when this study was conducted with young, healthy male subjects (4). Kjaergaard et al. (58) also found impaired digit span performance in SBS-sensitive subjects but not among non-SBS subjects with exposure at 25 mg/m3 VOC mixture, which is roughly equivalent to 7 ppm toluene. Otto et al. (4) suggested that differential effects may be due to differential sensitivity of the subject groups as well as relative insensitivity of many of the current neurobehavioral methods.

To test the unique susceptibilities of MCS patients, several factors must be taken into account. First, like SBS patients, MCS patients report responses at exposure levels that most individuals tolerate. For example, in our current protocol we conduct olfactory threshold testing in response to phenyl ethyl alcohol (PEA), a pleasant olfactory stimulant. MCS patients reported significantly more symptoms than normal controls during threshold testing. At suprathreshold levels they reported PEA to be significantly more unsafe and unpleasant than did normal controls. From our estimations, the concentrations at the average olfactory threshold are comparable to 7 ppm, a level well below that expected to produce neurobehavioral performance decrements.

Second, an overriding concern is that symptomatic responses of MCS patients are conditioned responses to olfactory cues (59). Even among healthy individuals, odor has been shown to impact performance (60-62). Neither the studies on SBS subjects nor controlled exposure studies have adequately accounted for the impact of odor on performance.

Ideally, controlled exposures with MCS patients will need to occur below olfactory thresholds to control for psychological expectations due to odor. Detecting effects at such low levels of exposures (as low as 1 ppm) will require highly sensitive behavioral performance measures. Measures such as reaction time and vigilance tasks have been the most sensitive indicators in previous cross-sectional and chamber studies (2). Therefore, use of measures of attention and vigilance similar to those cited in the signal-detection literature may offer the best alternative to detect effects among susceptible individuals and low-level exposure conditions.

Test Parameters

Unlike many previous exposure-chamber studies of neurotoxicants, exposing symptomatic groups such as MCS patients requires some shifts in the methodologies previously employed. First, the questions to be answered depart from the traditional concern for developing workplace standards. These studies sought to establish the upper limit of exposure before objective behavioral effects were detected. While not necessarily applicable even to all healthy individuals working with neurotoxicants, exposure to these levels among hypersensitive groups cannot be undertaken for obvious health and ethical concerns.

As has been suggested by other investigators, one method for detecting effects at lower exposure levels is to vary parameters within the performance test to increase its sensitivity to effects (4,63). Documentation of the effects of varied test parameters has been the subject of much attention within the experimental literature (e.g., signal-detection paradigms) and virtually no systematic attention within the neuropsychological literature or in the literature investigating the behavioral effects of neurotoxicants. For example, in the signal-detection literature, Jansen et al. (64) found that when signal probability was low, alcohol affected stimulus sensitivity and reaction time of hits, but the same dose of alcohol did not affect these parameters when signal probability was high. These findings were not replicated for Diazepam (65). The findings with alcohol were interpreted to suggest that reduced response accuracy to low probability signals would compromise driving performance since low and variable signals are likely. If only one stimulus intensity was used, this differential effect of alcohol would not have been detected.

Detection of effects under exposure conditions will also require that behavioral tests be repeated within a relatively short period of time. Therefore, more information is needed to document the effects of repeated test administration within a brief time period such as before, during, and after exposure. This will require that tests be of sufficient difficulty to allow variability in performance both within and between subjects.

Behavioral tests that focus on process rather than a single summary outcome will be important in the development of research on low-level exposures. Even in cross-sectional studies of exposed working populations, the detection of behavioral effects to objectify symptomatic complaints of poor memory and concentration has been problematic. This difficulty may be due to the fact that many of the neuropsychological tests applied to this field offer a summary score of performance (WAIS-R subtests) rather than assessment of learning curves or variables delineating the various functions that contribute to performance. Thus, several investigators have emphasized the application of information-processing paradigms and tasks to the assessment of neurotoxicants (43,66).

In our experience with MCS patients, the tasks most sensitive to behavioral performance decrements were those in which subfunctions of the task were assessed. For example, obtaining scores on signal-detection parameters for the Continuous Visual Memory Test (CVMT) (67) revealed that MCS patients recognized signals at the same rate as normals (hits) but over responded to nonsignals (false alarms). A summary score for this task would suggest impaired visual memory; however, analysis of the subfunctions suggests that response style may be a more important variable in their performance. Observation of the distribution of scores for this group of patients also suggests that only a subgroup of the total group (approximately 39%) exhibited significant impairment (Figure 1). This finding is consistent with the observation of hyperactive children of Weiss et al. (68). Inspection of individual performance was more important than looking at overall group means, which can mask a subgroup of hyperresponsive individuals. This is particularly important when case definitions for affected individuals such as MCS are not clear.



In addition to test parameters, a complete characterization of host factors such as psychiatric diagnoses and personality traits is important. For example, among the following variables in the MMPI-2--age, reading score, and depression--health concerns was the variable accounting for the highest percentage of variance in performance on the CVMT (Figure 2). Health concerns measures a range of somatic symptoms, some of which can be related to neurologic conditions and some to somatization (69). Previous exposure-chamber studies have not focused on individual difference variables, such as mood or the tendency to somatize, in assessing behavioral response to neurotoxicants. Documentation of these variables may be critical in understanding individual differences in performance among MCS patients, particularly since approximately 25% of MCS patients qualify for a diagnosis of depression (50,56). Little is known about how depression may interact with the effects of neurotoxicants in producing behavioral performance decrements.



Finally, as mentioned above, various odorants may affect behavioral performance (62). In olfactory research, extensive literature documents the psychophysical properties and mechanisms of odor perception. However, this literature does not address the concentration at which symptoms and objective health effects occur. Studies use objective behavioral tests (e.g., digit span) to document the effects of an odorant but relate these effects to properties of the odor (e.g., pleasant vs unpleasant, irritating vs nonirritating) rather than to concentrations in toxicological terms (60). While these odor effects may not impact healthy individuals, the same cannot be presumed in studies of symptomatic individuals such as MCS patients. Therefore, control or measurement of the impact of odor is critical. For example, alternate methods for administering exposures such as dermal routes could be considered.

Conclusion

Documenting behavioral responses to neurotoxicants among highly susceptible individuals places greater demands on the sensitivity of neuropsychological methods. Developing sensitive methods to elucidate the responses of sensitive individuals will also improve our approaches in the entire field of neurobehavioral toxicology.

References

1. Anger WK. Worksite behavioral research: results, sensitive methods, test batteries and the transition from laboratory data to human health. Neurotoxicology 11:629-670 (1990).

2. Gamberale F. Use of behavioral performance tests in the assessment of solvent toxicity. Scan J Work Environ Health 11:65-74 (1985).

3. Russell R. Neurobehavioral time bombs: their nature and their mechanisms. In: Behavioral Measures of Neurotoxicity (Russell R, Flattau P, Pope AM, eds). Washington:National Academy Press, 1990;206-225.

4. Otto D, Hudnell HK, House D, Molhave L, Counts W. Exposure of humans to a volatile organic mixture. Behav Assess 47(1):23-30 (1992).

5. Anger WK, Johnson BL. Human behavioral neurotoxicology workplace and community assessments. In: Environmental and Occupational Medicine (Rom WN, ed). Boston:Little, Brown, 1992;573-592.

6. Lezak MD. Neuropsychological Assessment. New York:Oxford University Press, 1983.

7. Gullion CM, Eckerman DA. Field testing. In: Neurobehavioral Toxicology (Annau Z, ed). Baltimore:Johns Hopkins University Press, 1986;288-330.

8. Wechsler D. Wechsler Adult Intelligence Scale-Revised. San Antonio, TX:The Psychological Corporation, 1981.

9. Heaton RK, Grant I, Matthews CG. Comprehensive Norms for an Expanded Halstead-Reitan Battery: Demographic Corrections, Research Findings, and Clinical Applications. Odessa, FL:Psychological Assessment Resources, 1991.

10. Hanninen H. Psychological test methods: sensitivity to long term chemical exposure at work. Neurobehav Toxicol 1:157-161 (1979).

11. Needleman HL. Lessons from the history of childhood plumbism for a pediatric neurotoxicology. In: Advances in Neurobehavioral Toxicology: Applications in Environmental and Occupational Health (Johnson BL, Anger WK, Durao A, Xintaras C, eds). Chelsea, MI:Lewis Publishers, 1990;331-337.

12. Hernberg S. Neurotoxic effects of long-term exposure to organic hydrocarbon solvents. Epidemiologic aspects. In: Mechanisms of Toxicity and Hazard Evaluation (Holmstedt B, Lauwerys R, Mercier M, Roberfroid M, eds). Amsterdam:Elsevier/North Holland Biomedical Press, 1980;307-317.

13. Hartman DE. Neuropsychological Toxicology: Identification and Assessment of Human Neurotoxic Syndromes. New York:Pergamon Press, 1988.

14. Nelson HE. National Adult Reading Test (NART): Test Manual. Windsor, UK:NFER, 1982.

15. Raven JD. Guide to Standard Progressive Matrices. London:H.K. Lewis, 1960.

16. Letz R, Baker EL. NES2. Neurobehavioral Evaluation Systems. June, 1988.

17. Trenerry MR, Crosson B, DeBoe J, Leber WR. Stroop Neuropsychological Screening Test Manual. Odessa, FL:Psychological Assessment Resources, 1989.

18. Wechsler, D. Wechsler Memory Scale-Revised Manual. San Antonio, TX:The Psychological Corporation, 1987.

19. Delis DC, Kramer JH, Kaplan E, Ober BA. California Verbal Learning Test. San Antonio, TX:The Psychological Corporation, 1987.

20. Visser RSH. Manual of the Complex Figure Test CFT: A Test Involving Restructuring Behaviour Developed for the Assessment of Brain Damage. Amsterdam:Sweta & Zeitlinger BV, 1973.

21. Vibratron II Operating Manual. Clifton, NJ:Physitemp Instruments, 1991.

22. Doty RL. The Smell Identification Test. Haddon Heights, NJ:Sensonics, 1983.

23. McNair DM, Lorr M, Droppleman LF. Profile of Mood States. San Diego:Educational and Industrial Testing Service, 1981.

24. Hathaway SF, McKinley JC. Minnesota Multiphasic Personality Inventory 2. Minneapolis, MN:University of Minnesota Press, 1989.

25. Oberg RGE, Udesen H, Thomsen AM, Gade A, Mortensen EL. Psychogenic behavioral impairments in patients exposed to neurotoxins. Neuropsychological assessment in differential diagnosis. In: Neurobehavioural Methods in Occupational and Environmental Health. Environmental Health Documents 3. Copenhagen:World Health Organization, 1985;130-135.

26. NIOSH. Organic Solvent Neurotoxicity. Cincinnati, OH:National Institute for Occupational Safety and Health, 1987.

27. Bleecker ML, Agnew J, Keogh JP, Stetson DS. Neurobehavioural evaluation in workers following a brief exposure to lead. In: Advances in the Biosciences, Neurobehavioral Methods in Occupational Health. Vol 46 (Gilioli R, Cassito MG, Foa V, eds). New York:Pergamon Press, 1983;255-262.

28. Ryan CM, Morrow LA, Hodgson M. Cacosmia and neurobehavioral dysfunction associated with occupational exposure to mixtures of organic solvents. Am J Psychiatry 145:1442-1445 (1988).

29. Istoc-Bobis M, Gabor S. Psychological dysfunctions in lead- and mercury-occupational exposure. Revue Roumaine des Sciences Sociales--Serie de Psychologie 31:183-191 (1987).

30. Hanninen H, Lindstrom K. Neurobehavioral Test Battery of the Institute of Occupational Health. Revised ed. Helsinki:Institute of Occupational Health, 1989.

31. Johnson BL, Baker EL, El Batawi M, Gilioli R, Hanninen H, Seppalainen AM, Xintaras C, eds. Prevention of Neurotoxic Illness in Working Populations. New York:John Wiley and Sons, 1987.

32. Cherry N, Venables H, Waldron HA. Description of the tests in the London School of Hygiene test battery. Scan J Work Environ Health 10(Suppl 1):18-19 (1984).

33. Fleishman EA. Dimensional analysis of psychomotor abilities. J Exp Psychol 48:437-454 (1954).

34. Grandjean P, Beckmann J, Ditlev G. Relation between subjective symptoms and psychometric tests results. In: Advances in the Biosciences, Neurobehavioral Methods in Occupational Health. Vol 46 (Gilioli R, Cassitto MG, Foa V, eds). New York:Pergamon Press, 1983;301-308.

35. Valciukas JA, Lilis R, Eisinger J, Blumberg WE, Fischbein A, Selikoff IJ. Behavioral indicators of lead neurotoxicity: results of a clinical field survey. Int Arch Occup Environ Health 41:217-236 (1978).

36. Angotzi G, Camerino D, Carboncini F, Cassitto MG, Ceccarelli F, Cioni R, Pradiso C, Sartorelli E. Neurobehavioral follow-up study of mercury exposure. In: Advances in the Biosciences, Neurobehavioral Methods in Occupational Health. Vol 46 (Gilioli R, Cassitto MG, Foa V, eds). New York:Pergamon Press, 1983;247-253.

37. Orbaek P, Lindgren M, Olivecrona H, Haeger-Aronsen B. Computed tomography and psychometric test performance in patients with solvent induced chronic toxic encephalopathy and healthy controls. Br J Ind Med 44:175-179 (1987).

38. Hogstedt C, Hane M, Axelson O. Diagnostic and health care aspects of workers exposed to solvents. In: Developments in Occupational Medicine (Zenz C, ed). Chicago:Year Book Medical Publishers, 1980;249-258.

39. Mergler D, Blain L. Assessing color vision loss among solvent-exposed workers. Am J Ind Med 12:195-203 (1987).

40. Braun CMJ, Daigneault S, Gilbert B. Color discrimination testing reveals early printshop solvent neurotoxicity better than a neuropsychological test battery. Arch Clin Neuropsychol 4:1-13 (1989).

41. Schwartz BS, Doty RL, Monroe C, Frye R, Barker S. Olfactory function in chemical workers exposed to acrylate and methacrylate vapors. Am J Public Health 79:613-618 (1989).

42. Beck AT. Beck Depression Inventory. San Antonio, TX:The Psychological Corporation, 1978.

43. Williamson AM. The development of a neurobehavioral test battery for use in hazard evaluations on occupational settings. Neurotoxicol Teratol 12:509-514 (1990).

44. Hanninen H. The psychological performance profile on occupational intoxications. Neurotoxicol Teratol 10:485-488 (1988).

45. White RF, Feldman RG, Travers PH. Review: neurobehavioral effects of toxicity due to metals, solvents, and insecticides. Clin Neuropharmacol 13:392-412 (1990).

46. Glass GV, McGraw B, Smith ML. Meta-analysis of Social Research. Beverly Hills, CA:Sage Publications, 1981.

47. Brodsky CM. Psychological factors contributing to somatoform diseases attributed to the workplace. J Occup Med 25:459-464 (1983).

48. Stewart DE, Raskin J. Psychiatric assessment of patients with "20th century disease" ("total allergy syndrome"). Can Med Assoc J 133:1001-1006 (1985).

49. Simon GE, Katon W, Sparks PJ. Allergic to life: psychological factors in environmental illness. Am J Psychiatry 147:901-906 (1990).

50. Fiedler N, Kipen HM, DeLuca J, Kelly-McNeil K, Natelson B. A controlled comparison of multiple chemical sensitivity and chronic fatigue syndrome. Psychosom Med (in press).

51. Parkinson DK, Bromet EJ, Cohen S, Dunn LO, Dew MA, Ryan C, Schwartz JE. Health effects of long-term solvent exposure among women in blue-collar occupations. Am J Ind Med 17:661-675 (1990).

52. Heston TF, Lewis LM. Gender bias in the evaluation and management of acute nontraumatic chest pain. The St. Louis Emergency Physician's Association Research Group. Fam Pract Res J 12:383-389 (1992).

53. Filsinger EE, Monte WC. Brief report: commentaries, remarks, and notes pertaining to sex research. J Sex Res 22:243-248 (1986).

54. Leventhal E. Gender and aging: women and their aging. In: Psychological Perspectives on Women's Health (Reddy DM, Adesso VJ, Flemming, eds). New York:Hemisphere Publishing, in press.

55. Morrow LA, Ryan CM, Hodgson MJ, Robin N. Risk factors associated with persistence of neuropsychological deficits in persons with organic solvent exposure. J Nerv Ment Dis 179:540-545 (1991).

56. Simon GE, Daniell W, Stockbridge H, Claypoole K, Rosenstock L. Immunologic, psychological, and neuropsychological factors in multiple chemical sensitivity: a controlled study. Ann Intern Med 19:97-103 (1993).

57. Molhave L, Back B, Pedersen OF. Human reactions to low concentrations of volatile organic compounds. Environ Int 12:167-175 (1986).

58. Kjaergaard SK, Molhave L, Pedersen OF. Human reactions to a mixture of indoor air volatile organic compounds. Atmos Environ 25:1417-1426 (1991).

59. Bolla-Wilson K, Wilson RJ, Bleecker ML. Conditioning of physical symptoms after neurotoxic exposure. J Occup Med 30:684-686 (1988).

60. Knasko SC. Performance, mood, and health during exposure to intermittent odors. Arch Environ Health 48(5):305-308 (1993).

61. Shusterman D. Critical review: the health significance of environmental odor pollution. Arch Environ Health 47(1):76-87 (1992).

62. Rotton J. Affective and cognitive consequences of malodorous pollution. Basic Appl Soc Psychol 4:171-191 (1983).

63. Teichner WH. Carbon monoxide and human performance: a methodological exploration. In: Behavioral Toxicology (Weiss B, Laties VC, eds). New York:Plenum Press, 1975;77-104.

64. Jansen AAI, de Gier JJ, Slangen JL. Alcohol effects on signal detection performance. Neuropsychobiology 14:83-87 (1985).

65. Jansen AAI, de Gier JJ, Slangen JL. Diazepam-induced changes in signal detection performance: a comparison with the effects on the critical flicker-fusion frequency and the digit symbol substitution test. Neuropsychobiology 16:193-197 (1986).

66. Stollery BT, Banks HA, Broadbent DE, Lee RW. Cognitive functioning in lead workers. Br J Ind Med 46:698-707 (1989).

67. Trahan DE, Larrabee GJ. Continuous Visual Memory Test. Odessa, FL:Psychological Assessment Resources, 1988.

68. Weiss B, William JH, Margen S, Abrams B, Citron LJ, Cox C, McKibben J, Ogar D, Schultz S. Behavioral responses to artificial food colors. Science 207(4438):1487-1489 (1980).

69. Alfano DP, Paniak CE, Finlayson ALJ. The MMPI and closed head injury: a neurocorrective approach. Neuropsychiatry: Neuropsychol Behav Neurol. 6:111-116 (1993).