Author's Guide for Preparing a Proceedings Paper
Using Behavior Moderators to Influence
CGF Command Entity Effectiveness and Performance
Philip D. Gillis, Ph.D.
US Army Research Institute
Simulator Systems Research Unit
12350 Research Parkway
Orlando, FL 32826-3276
407-384-3985
Philip_Gillis@stricom.army.mil
Steven R. Hursh, Ph.D.
Science Applications International Corporation
626 Town Center Dr.
Joppa, MD 21085
410-538-2901
Steven.R.Hursh@cpmx.
Keywords:
Behavior Moderators, Cognitive Abilities, Decision Making, Computer Generated Forces
ABSTRACT:. The basic problem addressed by this research concerns the need for more intelligent and realistic command and human performance influenced unit behaviors for Computer Generated Forces. The variability of command and unit behaviors due to human performance factors is basically left out of Army models and simulations leading to too-predictable and unrealistic results. Some constructive and virtual simulations need better methods to represent human performance variability in response to the stresses of war fighting in models and simulations. The ARI “Realism in CGF Unit and Command Entities Behaviors Project” is an attempt to address these deficiencies through the development of cognitive and human performance based models and algorithms that may be utilized by Advanced Distributed Simulation.
1.0 Introduction
The variability of the performance of human beings in response to the stressors of battle and as a part of planning for and reacting to a wide variety of battlefield events is a crucial area for modeling and simulation (M&S) research and development. Human performance in combat is widely acknowledged to be the most important factor in determining victory or defeat; however, there is only indirect evidence of this in current battlefield simulation development practice.
This lack of research, and more importantly, the lack of simulation developers’ implementations of some good research that does exist, impacts the accurate behavior of computer generated forces (CGF) entities, and it also impacts the human resources required for CGF scenario development and the exercise control of CGF entities. These requirements are inappropriately demanding and may be offloaded by more intelligence and realistic CGF behaviors.
This paper will report on the authors’ R&D in this particular area by means of addressing four primary project thrusts:
• Identification of the most relevant research underlying human performance factors and cognition that influence appropriate model behaviors; predict interactions between variables
• Development of human performance and cognitive models from the theory
• The development of the evaluation testbed, the Command, Control, and Communications Simulation (C3SIM), that is used to evaluate the outputs of the models in a military simulation testbed
• The evaluation of the ARI Human Performance Model (HPM) ver. 2.0 in C3SIM and the analysis of the data.
1.1 The Technical Problem
Specific problems with current command entity and individual unit behaviors in CGF within advanced distributed simulation (ADS) domains include the observations that such entities are neither influenced by human performance variables nor are their behaviors grounded in valid cognitive constructs during situational awareness, communications, and course of action activities. Command entities and otherwise animate objects frequently act based upon ground truth, rather than the more realistic perceived truth of a situation. And when such entities do act upon information that has been received, frequently the actions are inappropriately omnipotent. Sometimes a simulation developer will attempt to “degrade” command entity/unit performance, but will do so by means of the application of a random number generator affecting performance. Such “adjustments” are not grounded in appropriate and valid cognitive or human factors research and experimentation; thus, such attempts at varying performance possess no “construct validity.”
2. Background
For Phase I of this research, Gillis [1] reported on the effects of sleep deprivation and stress on a CGF Battalion Command Entity’s (BCE) performance in a recreated battle, using National Training Center data.
This Phase II report depicts both the direct and combined effects of fatigue, stress, time pressure, confidence-building events, intelligence, experience, and aggressive, neutral, or risk averse personality type on a CGF Battalion Command Entity’s effectiveness and performance in a recreated battle, using National Training Center data.
The implementation of these behavior moderators in the Human Performance Model, ver. 2.0. allows a much more realistic observation of the effects of human performance variables on CGF command entities behaviors than that reported for phase I. The current HPM software prototype uses Walter Reed Army Institute of Research data to model the effects of sleep deprivation on performance. HPM ver 2.0 also uses select experimental findings on other independent variables for the production of decision type codes (DTC), effectiveness, and performance variables that influence command entity behavior. Most importantly, HPM 2.0 output variables reflect models based upon current research on the combined effects of fatigue, stress, experience, intelligence, and personality type variables upon performance and effectiveness.
The Phase I BCE possessed limited situational awareness capabilities and course of action selection. For Phase II, the scope of a BCE’s actions has been suitably broadened. In a CGF context, four of the realistic measurable consequences reflecting the appropriate variability of human performance behavior that must be addressed are: correct or incorrect situational assessment, responsive or unresponsive communications, correct or incorrect course of action selection, and the timeliness of actions. The current phase explores the Battalion Command Entity’s enhancement or degradation of behavior in all of these areas, based in a simulated battle, but recreated from National Training Center data.
2.0 Research Methodology
The research and development methodology for this project includes efforts to:
• Obtain and analyze the most relevant research underlying human performance and cognition and predict the likely interactions between variables
• Develop human performance variable and cognitive models with construct validity that are based upon research, empirical findings, and predictions
• Develop an evaluation testbed, the Command, Control, and Communications Simulation, that is used to validate the models
• Evaluate the models and algorithms, analyze the evaluation data, validate the predicted interactions
The structure of this report will reflect research and development in these four areas in sequential order.
2.1 Research and empirical findings from the field
Models and algorithms reflecting human performance variability are primarily gleamed from the extant research data in the field on human performance variable and training effectiveness, and appropriate cognitive models are implemented that apply to simulated entity acts of cognition.
One of the most important precepts that a researcher encounters during an attempt to build valid models of human performance on the battlefield is that humans are generally stressed under battlefield conditions. Therefore, several recent and extensive reviews of the stress literature form the empirical basis for the HPM ver. 2.0 “effectiveness” and “performance” outputs.
Driskell, et. al. [2] and Driskell, Hughes, Willis, Cannon-Bowers, and Salas [3] have conducted two exhaustive meta-analysis reviews of the stress literature. These were conducted specifically to support the Air Force and Navy in preparing for studies to simulate the stressful environment for training. These reports were supplemented by an analysis of material and findings from three recent collections of reviews edited by Driskell and Salas [4]; Klein, Orasanu, Calderwood, and Zsambok [5]; and Flin, Salas, Strub, and Martin [6].
Mullins, Fatkin, Modrow, & Rice [7] found that participants with less experience reported higher ratings of overall stress. Also, several other studies have documented the benefits of experience for cognitive performance under stress in military-context evaluations (Kirschenbaum, [8]; Stokes, A. F. [9]; Klein, Calderwood, & Clinton-Cirocco, [10]; Klein & Calderwood, [11] ). Still other studies have studied the effects of intelligence on decision-making and found cognitive performance to be positively correlated with performance (Whitmarsh & Sulzen [12] ).
All together, these reviews summarize the results of over 1,350 studies of stress, many sponsored by branches of the Armed Forces. ARI contracted Dr. Steve Hursh of Science Applications International Corporation (SAIC) in Joppa, MD to compile and analyze these studies and develop a decision making under stress (DMUS) model that could be used to influence a CGF command entity’s behaviors in a battlefield simulation. Hursh [13] considered a variety of factors that have been found to degrade effectiveness and have been defined as stressors.
3.0 The Human Performance Model
While HPM ver. 1.0 was a decision-making under stress model used solely to compute human effectiveness, ver. 2.0 draws upon other ARI human performance variable findings from the field in an attempt to predict a CGF BCE’s performance in activities in addition to decision making and also not under stressful conditions.
The development of the HPM is thus proceeding in stages, with the incorporation of the DMUS model first, with ARI expanding the outputs based upon stress and non-stress related performance research resulting in HPM ver. 2.0.
For the Phase II completion of the project, the Human Performance Model ver 2.0 utilizes eight types of inputs:
• Sleep Deprivation/accumulation
• Variable Time Pressure
• Performance Use
• Stress Effects
• Confidence Building Effects
• Experience Effects
• Intelligence Effects
• Aggression/Risk Aversion Tendencies
And the HPM ver 2.0 produces four types of outputs:
• Effectiveness, a DMUS quantitative measure
• Decision Response Time, a quantitative measure
• Performance, a quantitative measure that has the option of using DMUS research
• Decision Type Code, DMUS qualitative outputs, consisting of correct, random, aggressive, and risk adverse codes.
Since the effectiveness and performance output variables were the primary areas for concentration for Phase II of this study, they will be examined in more detail for this paper.
3.1 Decision making under stress
The HPM ver 1.0 and refined HPM ver 2.0 DMUS stress model inputs related to the effectiveness output variable were implemented based primarily on the literature reviews by Driskell and colleagues [2] and Driskell, Hughes Guy, Willis, Cannon-Bowers, and Salas [3]. This research suggests the two most salient input stressors, useful for the confines of this study and the development of the effectiveness variable, are time pressure and fatigue brought on by continuous operations.
Thus it was necessary that HPM ver 2.0 outputs primarily reflect the effects of time pressure, fatigue and stress and furthermore, that the HPM reflect the findings that the effects of stress on decision making are strongly dependent on experience. When confronted with time pressure and work overload, the low experience CGF BCE should emulate the less experienced human decision maker, who is subject to a variety of errors that can degrade the quality of decisions in a variety of ways, as summarized by Orasanu and Backer [14]: “decision makers use a small number of heuristics (rules) in making their decisions (Tversky & Kahneman [15] ), fail to consider all possible decisions and outcome options (Slovic, Fischhoff, & Lichtenstein, [16] ), are inconsistent in dealing with risk (Lopes [17] ), ....[are] likely to display premature closure - terminating the decisional dilemma without generating all the alternatives and without seeking all available information about the outcomes (Janis [18] ).”
In contrast, studies of experienced decision-makers under stress suggest that a more streamlined decision strategy is used-- Naturalistic Decision-Making (Klein, Orasanu, Calderwood, and Zsambok [5]; Orasanu & Connolly [19]; Klein [20], in press; Klein & Crandall [21]. This strategy is best suited for settings where the decision task is unclear, the available information is incomplete, unreliable, or continuously changing, and stressors such as time pressure and high stakes are present (Orasanu & Connolly [19] ). Under such situations, it is impractical to adopt an exhaustive prescriptive decision strategy that requires complete data and is time-consuming. Klein [22] has proposed that experienced decision-makers faced with this situation use the Recognition-Primed Decision (RPD) model.
According to this model (Klein [23] ), the experienced decision-maker can make rapid but effective decisions by using experience to size up the situation and to generate and evaluate COAs one at a time (as opposed to comparatively). In the simple case, the situation is recognized as typical of ones encountered before, and a typical COA can be immediately selected. The product of naturalistic decision-making is a decision that is adequate and resistant to time pressure, if not absolutely optimal. Alternative, more exhaustive strategies are disrupted by time pressure and, consequently, yield decisions that are flawed or not timely applied (Hursh [13] ). The studies of naturalistic decision-making “show that experienced decision-makers are able to generate reasonable options as the first ones they consider, and select these options to carry out when performing a stressful task such as flying a complex mission in a simulator (Klein [22]; Klein, Wolf, Militello, & Zsambok [24]; Stokes, Kemper, & Marsh [25] and Yates [26] ).”
Naturalistic decision-making critically depends on a high level of training and experience, and project goals mandate the observation of a simulated, poorly-trained and inexperienced BCE moving teams around on a battlefield as well. The need for the CGF BCE to act based upon variable experience and training levels prompted the development of a decision type code (DTC) as one of the HPM outputs.
3.2 Fatigue and sleep deprivation effects
Modern combat augmented by night vision devices and electronic means of navigation and communication is not constrained by time of day and the cloak of darkness. The flow of battle may be relatively continuous with few breaks for sleep and recuperation. Under these conditions of continuous or sustained operations, sleep deprivation and fatigue may be a natural human hazard. Moreover, studies of sleep patterns in simulated combat at the National Training Center indicate that commanders (Lieutenant Colonels and Colonels) in force on force operations average just over four hours of sleep per day, about half the normal requirement for fully effective performance (Belenky, Balkin, Thomas, Redmond, Kant, Thorne, Sing, Wesensten, & Bliese [27] ).
Furthermore, laboratory studies of sleep deprivation indicate that the most sensitive indicators of sleep deprivation are cognitive operations, such as logical reasoning, mathematical operations, short term memory, and decision making (Thorne, Genser, Sing & Hegge [28]; Banderet, Stokes, Francesconi, Kowal, & Naitoh [29]; Horne [30]; Angus & Heslegrave [31] ). Hence, fatigue can be a strong disrupter of command level performance.
For the implementation of the effectiveness variable and DMUS Decision Type Code outputs used in HPM 2.0, the Sleep and Performance Model , based upon a previous model SAIC had developed in conjunction with the Division of Neuropsychiatry of the Walter Reed Army Institute of Research (WRAIR), was maintained. A comparison of the algorithm, however, with both Air Force and U.K. Ministry of Defence algorithms is ongoing.
3.3 Calculation of the effectiveness variable
At the heart of the ver. 2.0 model is a cognitive reservoir that maintains a balance of effective performance units. During sleep, units are added to the cognitive reservoir according to the sleep accumulation function, which specifies how many units of effective performance are credited for each minute of sleep. The rate of accumulation is responsive to the sleep deficit, the difference between the current level of the cognitive reservoir and its maximum capacity. During time awake, units are subtracted from the cognitive reservoir according to the performance use function, which specifies a linear decrease in the cognitive reservoir with each minute awake (Hursh [13] ).
The resulting effectiveness variable is the sum of three variables: the level of the cognitive reservoir, expressed as a percent of its maximum capacity; the performance circadian rhythm, and the general stress effects.
3.4 Interactions and final DTC outputs
The effects of stress on performance are strongly conditioned by individual factors such as levels of training , experience, and type of personality. The importance of these factors is strongly dependent on the level of stress, fatigue, and time pressure that a human or simulated CE is operates under. The interactions of these factors in HPM 2.0 result in the output DTC’s: correct, random, or situation primed, as diagrammed in Figure 1. Situation primed DTC’s were further categorized as aggressive, neutral, or risk averse. The production of the codes and consequent selection of COA’S by the BCE proceeds in the following manner:
The overall level of effectiveness, E, resulting from the sleep and fatigue model and the stress process determines the first level of control. Based on detailed studies of decision making in stressful emergency
situations, it is clear that the nature of decision making is phasic, depending on time pressure at each stage of the event or battle (Crego & Spinks [32] ). Periods of time-constrained decision pressure are interspersed with periods of time-rich decision opportunity. The first level of the structure in Figure 1 represents the oscillation between these two general states. The outcome of this first branch is based on the level of E and is probabilistic; as E varies based on changes in time pressure, stress, and fatigue, the likelihood of branching to the left or right varies continuously.
The left hand branch is selected most often when E is high based on low time pressure, stress and fatigue. This is a time rich decision opportunity and the greater the level of training, experience and intelligence, the more likely the CE will select the optimal or correct COA. As training, experience, and I.Q. decline, the greater the likelihood that the decision will simply be a random selection from those courses of action that are reasonable for this situation. Note, however, that this is a probabilistic process and only under the conditions of little experience and training will course of action selection actually be random. In practice, given the normal requirements for command, no command entity would ever approach this extreme case.
The right hand branch is selected most often when E is low based on high time pressure, stress and fatigue. This represents time constrained decision pressure and results in situation primed decision making after the concept of recognition primed decision making described by Klein [22]. When a BCE is influenced by variables reflecting these conditions, it will act on prior experience with similar situations, if the BCE experience variable reflects good to high experience.
According to this mode of decision making, the experienced commander, when confronted with extreme time pressure or stress, does not attempt an exhaustive utility analysis of all available options. Rather, the commander looks for features of the
situation that resemble prior experiences and recalls successful decisions from those prior occasions. Once the situation is “type identified”, the commander can apply a “typical” course of action. Obviously, the likelihood of engaging in situation primed decision
making will strongly depend on the level of prior experience required to build this catalog of typical courses of action.
For Phase II, situation primed COA’s were largely coded similar to “correct” COA’s, taken from the decision support template and SME analyses of the situations in the context of the mission itself. Situation primed COA’s were further subcategorized as being typical of aggressive, neutral, or a risk averse personality type.
Time constraints did not allow the implementation for a full range of COA’s reflecting a broad range of experience, hence a broad range of RPD’s that a BCE could enact. During Phase III, these distinctions will be integrated.
For its production of DTC outputs, HPM ver. 2.0 substantially factors in the effects of experience and intelligence in a manner that was consistent with RPD theory and with other findings from the field. Experience and intelligence effects influence the magnitude of a random variable, which is compared to E; the resultant comparison controls the top-level branch selection in Figure 1. The magnitude of the IQ and experience effects for the calculation of the random variable was based upon empirical findings from the literature in the area.
3.4.1 Transforming DTC’s to COA’s
For Phase II a contract was awarded to TRW, in Orlando, FL, for purposes of analyzing a NTC mission for elements useful for evaluating BCE effectiveness and performance. Subject matter experts (SMEs) analyzed one Movement to Contact mission and produced for the mission:
The Decision Support Template
Revised five paragraph OPORDS
Validation of Bluefor and Opfor units
Matrices for the Decision Support Template (DST) actions
TRW SMEs validated the correct actions prescribed within the DST, and within situational contexts, further categorized potential COAs as indicative of the following types:
Correct
Highly to Poorly Trained
Aggressive
Risk Averse
High to Low Experience
SMEs also analyzed the mission for potentially stressful and confidence building events that could occur during the course of the mission. In addition, stress types and amplitudes were correlated with potential events.
3.5 The effects of time pressure
The important finding from the SAIC review of decision making under time pressure is that much larger effects should be expected with inexperienced commanders/CGF command entities as compared to experienced commanders/CGF command entities. For inexperienced decision makers who cannot rely on a recognition-primed decision strategy or who attempt to use an exhaustive prescriptive strategy, the effects of time pressure will be to seriously degrade decision making (Crego & Spinks [32] ).
Driskell [2] has summarized the literature on time pressure and have found that a relatively simple linear equation relates the magnitude of time pressure to the size of the stress effect. Hursh advised that the magnitude of time pressure (MAG) be defined as:
MAG = longer time period/(longer period + shorter period).
Hence, a task that normally is performed with high accuracy in 60 seconds that is required in 42 seconds would have a MAG value of .587 and would predict a correlation coefficient with accuracy (r) of -.3.
For the implementation of the effects of time pressure in HPM, ver. 2.0, Hursh proposed the utilization of three a priori levels of potential time pressures. The levels were categorized, but only two: high and none, were actually implemented and tested for HPM and C3SIM, ver. 2.0:
low (.481)
moderate (.587)
high (.707)
The corresponding changes in accuracy were defined as :
low (r = -.1)
moderate (r = -.3)
high (r = -.5).
Versions 3.0 of HPM and C3SIM will see all three levels implemented.
Driskell did not consider the modulating effects of experience on the magnitude of the time pressure effect [2]. Based on the review of naturalistic decision making (Klein [22 and 33] ), time pressure tends to increase the likelihood that the model will attempt to provide a situation primed decision, and it was this latter finding that was implemented in HPM 2.0.
3.6 Calculation of the stress effect
The effects of stress can degrade cognitive performance as represented in a lower computation of the effectiveness variable. The stress effect (SE), as represented in this simplified model, is designed to reflect the influence of stressful events and time pressure on effectiveness in making decisions.
One key factor in this model of stress is the occurrence of significant events in the battle scenario that may either advance the mission (positive or confidence building events) or hinder the mission (negative or stressful events). The computation of the stress effect depends, in part, on the frequency of those events and their value or severity. For purposes of computing the stress effect, mission advancing or confidence building events ranged in value from 0 to +1; hindering or stressful events ranged in value from 0 to -1. The overall stress effect at any moment in time considers the sum of these values over the preceding time interval. The actual calculation of the effectiveness and stress effect variables are reported in Gillis, 1998.
The ability to process and react to events is modulated by the time available. During a slowly developing operation with events occurring infrequently in time there is plenty of time for a human or simulated command entity to react to events and take appropriate action. This tends to diminish the effects of stressful events. Hence, the value of battle events is multiplied by a factor that represents time pressure. For example, a time pressure value of .5 would reflect a magnitude of time pressure of .707, while a time pressure of .1 would reflect a magnitude of time pressure of .481. Since it is not possible at present to actually measure the magnitude of time pressure in a C3SIM mission scenario, the model was exercised with a range of time pressure values from 0 to 1, that represent a range of time pressure magnitudes.
3.7 The decay of the stress effect over time
The overall value of the stress effect is subject to the decay of memory over time. As time elapses since an event, the value of that event in contributing to the total value of SE declines according to the double exponential shown below, based on classic memory experiments (Ebbinghaus [34] ). The initial term of the expression represents short-term memory and the second term represents long-term memory:
[pic]
The above equation is computed given that Current Value is the percent of the original value at time t since the original value, and time is in hours.
3.8 The performance output variable
The current implementation of the model underlying the performance output variable can be viewed as a modified version of a study by Locklear, Powell, and Fiedler (1988) which examined the impact of individual experience and intelligence of military leaders on decision performance under varying degrees of stress (Guest, 1998, in Gillis, P.D., Hursh, S., Guest, M., & Sweetman, B. [35].) Their experiment incorporated two levels of intelligence (low, high) and two levels of experience (low, high) based on median splits. Three levels of stress were included (low, medium, high). Results indicated that:
intelligence was a benefit at all stress levels;
experienced leaders outperformed less experienced leaders in the high stress condition
at low and moderate stress levels, intelligence resulted in better performance than experience
at high levels of stress, experience resulted in better performance than intelligence.
This study is in agreement with other studies that examined more specific effects of various factors on cognitive performance under stress. Importantly, though, very few studies incorporated aspects of experience and intelligence with the consideration of stress. Therefore, the Locklear, Fiedler, & Powell [36] study is chosen as a representative basis for the current model development. The current model attempts to capture the impact of expertise on cognitive performance under stress, in addition to benefits of intelligence (Guest in Gillis, P.D., Hursh, S., Guest, M., & Sweetman, B. [35] ).
The key characteristics of the cognitive performance model related to individual experience, intelligence, and stress can be summarized as follows:
Intelligence positively affects performance at all levels but is subject to performance decrement, at high stress levels, depending on experience;
At high levels of stress, experience positively affects performance more than intelligence;
Experienced individuals have little or no performance decrement due to high stress;
Novices have a significant performance decrement under high stress.
The current model implements:
a three unit performance decrement for novice individuals from low to high stress, when intelligence remains constant
a 1.5 unit performance decrement for middle-experienced individuals from low to high stress, when intelligence remains constant
no performance decrement for experts from low to high stress
a 1.0 unit increase in performance with a 1.0 unit increase in intelligence, when experience and stress remain constant.
Guest recommended the above findings results in the following regression equation, which was used to compute performance in HPM, ver. 2.0, under high stress conditions:
y’ = 2.25(EXPERIENCE) + 1.0(INTELLIGENCE) - 1.5(STRESS) + 1.5
This equation is currently undergoing minor revision based upon the incorporation of new findings.
4.0 The C3SIM Testbed
The evaluation software requirements for this project are extensive, and essentially call for specialized software to be developed. Early on during the conception phase for the project, the question related to the appropriateness of command echelon necessary for observing the effects of HPM clearly dictated that the most significant effects of the HPM variables would be observed at battalion level or higher, because decision making at such a level possesses a formidable.
The effects of fatigue are, of course, a problem at any level of operations on the battlefield; however, at higher levels of command errors in judgement have great impact on the success or failure of the mission and command experience can moderate the effects of stress and fatigue.
Given that sleep deprivation, accumulation, and performance use were three important variables operating during the study, it was obvious that simulated missions used had to be fairly lengthy. The resulting mission requirements for the project included the necessity to observe a battalion command entity, commanding three or four teams in a simulated battle over a period of time of at least eight hours.
Considering the various study requirements, several constructive simulations were initially considered for purposes of the evaluation of the Human Performance Model developed for this project. MODSAF was, at first, the primary candidate; however, based upon two cost estimates for using MODSAF to test the full range of outputs of HPM at the battalion level of command, MODSAF was quickly dismissed. It was considered too complex and difficult to modify, given the monetary resources available for the current project.
Thus, it was deemed more advisable to develop a software evaluation testbed whereby the effects of the human performance variables acting on simulated command entities at higher echelon levels could be observed. Command actions related to situational awareness, communications, and course of action activities at the battalion level clearly have a potentially greater impact than at squad level.
4.1 C3SIM ver. 2.0 functionality
The evaluation testbed requirements for this project are extensive, and essentially call for specialized software to be developed. The C3SIM evaluation testbed has been in various stages of development for the past five years. The initial functionality of C3SIM was developed by the author, but was turned over to Systems Engineering Associates in San Diego for enhancement and refinement, which resulted in C3SIM ver. 1.0. For Phase II, the Institute for Simulation and Training continued to enhance those portions of the C3SIM functionality that would be used to test HPM.
C3SIM at the end of Phase II is a Win32 application that, in replay mode, is capable of reading temporal and positional data from National Training Center data sets, and consequently replaying the battle. In this mode, C3SIM employs probability of hit and kill algorithms from both the Close Combat Tactical Trainer and the Joint Research Training Center to assess attrition for both Bluefor and Opfor. Results from C3SIM mission replay mode and the actual NTC mission results are published in Gillis [1].
C3SIM, ver. 2.0, is currently configured to run on two, or preferably three, Win32 platforms. It uses Windows Distributed Component technology to allow the three separate applications residing on separate machines to communicate. The three separate applications consist of the C3SIM simulation itself, the HPM, and the Knowledge Base Server (KBS). The KBS provides output in the form of a temporally based knowledge base to C3SIM consisting of battalion level team courses of action (COA); these COA’s are based upon the decision support template (DST) that was developed by the actual Battalion Commander’s staff for the movement to contact mission.
4.1.1 The interaction of HPM variables within HPM, C3SIM, and KBS
HPM ver. 2.0 currently sends the previously discussed effectiveness and performance variables to C3SIM, and these variables affect a variety of conditions and actions within C3SIM.
Effectiveness and performance variables predominantly influence courses of action returned to C3SIM from HPM. Figure 1 demonstrated possible Decision Type Codes that could be returned, consisting of: Correct (aggressive, neutral, or passive); Random (aggressive, neutral, or passive); and Situation primed (aggressive, neutral, or passive
The relative effects of effectiveness and performance variables upon the BCE’s situational awareness, communications, and course of action selections are being investigated during Phase III of this project.
All DTC’s are processed within HPM, and all HPM outputs are subject to stochastic influences. For example, although effectiveness and performance are the result of mathematical equations resulting in real numbers, the actual output produced is partially based upon the influences of chance, as are events on the battlefield.
The actual values of effectiveness and performance are also sent over to C3SIM. These values are also used in a similar stochastic manner in the following situations:
During the BCE’s situational awareness events, such as type and depth of situational awareness activities.
During the BCE’s communication events, such as whether or not it responded to a team situation report (SITREP) or whether or not it issued a fragmentary order (FRAGO) based upon the SITREP.
During select BCE’s course of action, in addition to those returned by the KB server. FRAGO’s consisting of Movement and Fire orders are two examples of COA’s within this category of processing
Two of the most important variables, time pressure on performance and decision response time, have not been fully implemented within C3SIM ver 2.0. Since time pressure has such an important impact upon human performance, much of the movement to contact mission is run with the time pressure variable set to a moderately high setting.
Decision response time is currently computed within HPM and is sent over to C3SIM, yet the variable is not fully implemented with the BCE’s situational awareness, communications, and course of action selection activities, but will be fully implemented during Phase III of the project.
5.0 Data Analysis
A very strong emphasis was placed upon data collection and analysis for Phase II. To date approximately 200 simulation runs have been completed and results analyzed over time.
The data was collected from the simulation runs with the simulations in a time-skewed mode. A comparison of the skewed mode with real time mode demonstrated an approximate 10% difference in attrition levels for both sides. Attrition levels were higher in real time mode because all unit’s fire cycles represented more realistic levels. However, due to the time constraints imposed on this study, it was not possible to complete all simulation runs in real time mode. Neither was it possible to complete multiple runs of each tested permutation of independent variables.
5.0.1 Findings from multiple runs of one permutation
For purposes of the current analysis, it was necessary to demonstrate that a single run at each permutation setting was adequate, i.e., that the stochastic nature of the HPM and C3SIM models used for the determination of actions was negligible; that is, negligible for purposes of the statistical analysis of the data.
Therefore, multiple runs of one permutation of the independent variables were completed and results were randomly divided into two samples. Linear associations between dependent variables were observed, with results indicating high correlations and linear associations between pairings for the key study dependent variables.
Thus, it was deemed reasonable to proceed with the current analysis, based upon the assumption that the stochastic influences of HPM and C3SIM does not produce statistically significant different outcomes between simulation runs for key dependent variables.
5.0.2 Assumptions for the statistical analysis for the single runs of the current set of data
For the approximate 200 simulation runs that have been completed, the Shapiro Wilks’ test for normality was applied to the dependent variables, with results indicating normal distributions for all key variables. Based upon this finding, levels of statistical significance are reported for the Pearson’s Product Moment Correlation Coefficients and for the Student’s t test associated with the linear regression analyses. It should be noted, however, that such significant differences are reported, at this time, for purposes of comparison, only, and cannot be construed as completely valid evidence of significant differences until multiple runs of each permutation of independent variables are completed during Phase III of this project.
5.1 The independent variables (I.V.) for the analysis of HPM, ver 2.0.
The following independent variables have been run in the C3SIM evaluation testbed and analyzed:
Sleep, range 2 to 7 hours/night over 5 nights
Experience, 3 different levels
Aggressive, Neutral, or Risk Averse Personality
Intelligence (I.Q.), 3 different levels
Time Pressure
Experience level of OpFor BCE
648 permutations of the above independent variables are possible. It was not possible to analyze the varying experience levels of the Opfor BCE for this study. Therefore, all simulations were run with the Opfor BCE experience level set to “highly experienced.”
5.2 The dependent variables (D.V.), their means (X), and standard deviations (s’) for the analysis of HPM.
The set of dependent variables that have been used for the current analysis are collected at the end of each simulation run, which constitutes the end of the mission (EOM).
They are:
Dependent Variable X s’
Bluefor Survivability Index 78.2 4.0
Opfor Survivability Index 56.9 81.2
TF4-64 Firepower Damage 25.4 6.4
TF4-64 Mobility Damage 29.2 7.3
TF4-64 FP and Mob Damage 18.8 6.1
TF4-64 Fully Mission Capable 31.6 9.6
TF4-64 Survivability Index 80.1 8.9
TF4-64 in Objective FORD 22.4 30.4
TF4-64 Mission Finish 13:06
Opfor in Furlong Ridge 8.8 10.6
Bluefor Attacks 3706 729
Opfor Attacks 1110 190
HPM Resource Balance EOM 1266 172
HPM Effectiveness at EOM 38.6 17.6
HPM Performance at EOM 30.3 22.7
BCE Communications Failures 28.5 57.4
Stressors Received by BCE 175 192.4
Table 1. The dependent variables, their means (X), and standard deviations (s’) for the analysis of HPM
5.3 The analysis of the results
Four of the dependent variables were considered to be the strongest indicators of Bluefor/Opfor mission success and/or failure and were considered key variables for the purposes of this analysis:
TF4-64 Survivability Index
TF4-64 in Objective FORD
TF4-64 Mission Completion Time
Opfor in Furlong Ridge
This consideration was based upon both Bluefor and Opfor commanders’ intents for the missions, see Gillis [1].
The Commander’s intent for the BLUFOR was “Destroy the Advance MRD in zone and seize key terrain for follow-on defense. The end state is the destruction of the advance guard (destroy Opfor in Furlong Ridge) without penetration of PL FLORIDA, occupy OBJECTIVE FORD with sufficient combat power to establish a defense in sector to defeat follow-on enemy Motorized Rifle Regiments.”
The Commander’s Intent for the OpFor was “Find the enemy and destroy his reconnaissance and lead elements with the Motorized Rifle battalion (MRB) Advance Guard battalions. Success is destroying enemy in zone while maintaining sufficient combat power to seize the regimental subsequent objectives.”
NTC data sets used for the first mission were complete with the exception of HINDs, eight T-72, and ten BMP units for the Opfor and all artillery units for both BLUFOR and Opfor. Therefore, CGF operator-controlled HINDs, missing T-72s, BMPs, Opfor and Bluefor artillery were utilized in order to make up for the missing NTC data. For this particular set of simulation runs, Opfor artillery fires and other Opfor CGF attacks were all similar and orchestrated based upon the Opfor Operations Order for the mission and the Opfor Doctrinal Support Package. Bluefor artillery fires were consistent with Bluefor OPORD fire support matrix for the mission. Time limits for all missions were consistent with actual NTC mission data.
5.3.2 Significant Pearson Product Moment correlation coefficients for dependent variables demonstrating linear association
An observation of the linear correlations, the Pearson Product Moment Correlation Coefficients (r), between variables points toward a useful study of the relations among select variables. This value will be reported in the discussion of each dependent variable analysis.
5.3.3 Multiple linear regression analysis of dependent variables demonstrating significant correlation.
A frequently tested hypothesis for variable associations of this type is that there is no linear relationship between an independent and dependent variable—that the slope of the regression line is 0. Slopes (B1) for significantly different linear relations between dependent and independent variables will be reported. The statistic used to test this hypothesis is Student’s t distribution. The significance of this test will also be reported.
A commonly used measure of how well the linear model between dependent and independent variables actually fits, or the goodness of fit of a linear model, is r2, or the coefficient of determination. The combined r2 for all independent variables effects upon a single dependent variable will be reported in the discussion of each dependent variable analysis.
5.3.3.1 Dependent Variable TF4-64 in Objective FORD
Per the Bluefor commander’s intent, the success of the BCE at moving the four teams up into Objective Ford was a critical mission element. TF4-64 in Objective FORD is reported as a percentage of the task force that actually made it into Objective FORD at the end of the mission.
The Pearson Product Moment Correlation Coefficients between the five BCE/HPM relevant independent variables (all except “Experience level of Opfor BCE”) and TF4-64 in Objective FORD all demonstrated positive r’s, with the r values for I.V. sleep of .29 and for experience of .39 being significant at the .01 level of significance.
The multiple regression analysis for the same combination of variables demonstrated an r2 of .29, thus indicating 29% of the observed variability for the dependent variable TF4-64 in Objective FORD can be accounted for by the five I.V.s. The student’s t for the slopes (B1) for the independent variables sleep, and time pressure, and experience were all significant at the .01 level of significance for this analysis.
This particular analysis constitutes one of the strongest findings for the study; it demonstrates the effects of increased levels of sleep and experience, and decreased amounts of time pressure on the probability for the BCE’s mission success.
The positive, though not significant, correlation between aggression and the dependent variable (r = .07) indicates that the more aggressive BCE in C3SIM was more successful in getting its units to the final objective, though as shall be noted in the analysis of the TF4-64 Survivability Index, a more aggressive BCE lost more of its units doing so.
5.3.2 Dependent Variable Opfor in Furlong Ridge
Again, per the Bluefor commander’s intent, the success of the Bluefor BCE at “destroying the Advance MRD in zone” was a critical mission element. Opfor in Furlong Ridge is reported as a percentage of the Opfor MRD that remained in the main battle zone at the end of the mission.
The Pearson Product Moment Correlation Coefficients between the independent variables of aggression, sleep, experience, and I.Q. all demonstrated negative correlations with the dependent variable, as they should. As the levels of sleep, experience, and I.Q. increased and the BCE became more aggressive, the percentage of the Opfor in the battle zone should, and did, reflect higher attrition rates. The r value for I.V. sleep of -.73 demonstrated significance at the .01 level of significance.
The multiple regression analysis for the same combination of variables demonstrated an r2 of .56, thus indicating 56% of the observed variability for the dependent variable “Opfor in Furlong Ridge” could be accounted for by the five I.V.s. The student’s t for the slopes (B1) for the independent variables sleep and time pressure were again significant at the .01 level of significance for this analysis.
The significance of this particular analysis lies in the observation that the effects of the five I.V.s primarily accounted for the absence of the enemy in Furlong Ridge at the end of the mission. As the appropriate variables increased and time pressure decreased, more of the enemy was attrited in Furlong Ridge.
Notable in analysis of this dependent variable is the lack of statistical significance for the effects of the experience variable for increased attrition on the Opfor in Furlong Ridge. However, the lack of statistical significance does not likely point to a deficiency in the HPM algorithms incorporating the experience effects, but rather a problem in incorporating the effects of greater experience in the BCE in the simulation itself, a known shortcoming, as pointed out in section 3.4 of this paper. This deficiency is being corrected during Phase III of this project by means of the incorporation of refinements to the COA’s, reflecting a greater range or experience levels possible for the BCE.
5.3.3.3 Dependent Variable TF4-64 Survivability Index
The survivability index of all the Blueforces was only in part a consequence of the performance and effectiveness of the TF4-64 BCE. Since another TF (TF1-33) was operating in the arena in support of TF4-64, the survivability index of all the Blueforces was also a consequence of the effectiveness of TF1-33 in attriting the enemy in support of TF4-64. TF1-33 unit movements were based upon NTC data for the actual mission. TF1-33 units fired opportunistically.
The Pearson Product Moment Correlation Coefficients for the independent variable of sleep, only, demonstrated a positive correlation with the dependent variable, with the r value of .47 being significant at the .01 level of significance. Noteworthy however was a negative r value for the aggression I.V.(-.06), reflecting an inverse correlation between aggression and TF4-64 survivability, i.e., a more risk averse BCE in C3SIM was more successful at lowering the attrition level of its own units.
The multiple regression analysis for the same combination of variables demonstrating an r2 of .26, thus indicating 26% of the observed variability for the Bluefor survivability index could be accounted for by the five I.V.s. The student’s t for the slope (B1) for the independent variable sleep again was significant at the .01 level of significance for this analysis.
TF4-64 survivability was also strongly influenced by the communications capability of the BCE; an inverse r of -.69 demonstrates that the more the BCE communicated, the less his TF was attrited.
5.3.3.4 Dependent Variable TF4-64 Mission Completion Time
The outcome for this dependent variable was consistent with expected results. The movement to contact mission was scheduled to begin at 0500, and the last event on the DST was scheduled for 0950, with a possible EOM between 1015 and 1030. A simulation cut off time of 1400 was imposed on BCE’s for all of the simulation runs; if the BCE had not successfully moved its teams to Objective Ford by that time, it was apparent it would not meet this mission element.
All five independent variables demonstrated negative inverse r’s in relation to the EOM dependent variable, except for Time Pressure, which is supposed to show a positive r, or linear relation to the I.V. The r and level of significance is presented in Table 2:
Dependent Variable r significance
Sleep -.30 .01
Experience -.40 .01
Time Pressure .21 .01
IQ -.23 .01
Aggression -.12 .06
Table 2. Pearson Product Moment r’s and level of significance for dependent variables and EOM.
The multiple regression analysis for the same combination of variables demonstrating an r2 of .33, thus indicating 33% of the observed variability for the dependent variable “mission completion time” could be accounted for by the five I.V.s. The student’s t for the slopes (B1) for the independent variables sleep, time pressure, and experience were all significant at the .01 level of significance for this analysis.
The significance of this particular analysis lies in the observation that the combined effects of the five I.V.s significantly accounted for the BCE’s ability to complete the mission is a shorter amount of time. As sleep increased and time pressure decreased, and as experience and intelligence increased, the BCE moved its four teams more quickly into Objective Ford.
6.0 Next Steps
The current phase of this project, Phase III, deals in part with the addition to a training variable, and its interactions with the other extant variables in HPM, ver. 2.0. In addition, HPM ver. 3.0 will be evaluated via C3SIM and, if possible, via an examination of the effects of its outputs in a MODSAF scenario.
In addition, the knowledge representation (KR) schema used for representing human factors attributes of the BCE in addition to effectiveness and performance variables was found to be inadequate. A much more robust KR format must be used that may account for missing or incomplete data during various stages of the mission scenario.
Inherent in knowledge representation issues for a project of this type are issues involving missing or incomplete data when a more robust knowledge representation is attempted. In order to resolve these difficulties, a Bayesian Belief Network will be used for HPM ver. 3.0 in order to correct for problems related to missing and incomplete data.
7.0 References
[1] Gillis, P.D., “Realism in computer generated forces command entity behaviors,” Proceedings of the Seventh Conference on Computer Generated Forces and Behavioral Representation, Orlando, FL, 12-14 May, 1998.
[2] Driskell, J.E., Mullen, B., Johnson, C., Hughes, S., & Batchelor, C. Development of quantitative specifications for simulating the stress environment (Report No. AL-TR-1991-0109). Wright-Patterson AFB, OH: Armstrong Laboratory, 1992.
[3] Driskell, J.E., Hughes, S.C., Guy, W., Willis, R.C., Cannon-Bowers, J., & Salas, E. Stress. stressor, and decision-making. Orlando, FL: Technical Report for the Naval Training Systems Center., 1991.
[4] Driskell, J.E. & Salas, E. “Group decision-making under stress,” Journal of Applied Psychology, 1992.
[5] Klein, G.A., Orasanu, J., Calderwood, R. and Zsambok, C.E. Decision Making in Action: Models and Methods. Norwood, NJ: Ablex, 1993.
[6] Flin, R., Salas, E., Strub, M., and Martin, L. Decision Making Under Stress, Emerging Themes and Applications, Brookfiled: Ashgate, 1997.
[7] Mullins, L. L., Fatkin, L. T., Modrow, H. E., & Rice, D. J. “The relationship between cognitive performance and stress perception in military operations,” Proceedings of the Human Factors and Ergonomics Society 39th Annual Meeting. San Diego, CA, pp. 868-872, 1995.
[8] Kirschenbaum, S. S. “Influence of experience on information-gathering strategies,”Journal of Applied Psychology,7(3), 343-352, 1992.
[9] Stokes, A. F. “Sources of stress-resistant performance in aeronautical decision making: The role of knowledge representation and trait anxiety,” Proceedings of the Human Factors and Ergonomics Society 39th Annual Meeting. San Diego, CA, pp. 887-890, 1995.
[10] Klein, G. A., Calderwood, R., & Clinton-Cirocco, A. “Rapid decision making on the fireground,” Fire Engineering, 149(3), 89-90, 1996.
[11] Klein, G. A. & Calderwood, R. Investigations of naturalistic decision making and the recognition-primed decision model. ARI Technical Report 90-59. U.S. Army Research Institute for the Behavioral and Social Sciences, Alexandria, VA., 1990.
[12] Whitmarsh, P. J. & Sulzen, R. H. “Prediction of simulated infantry-combat performance from a general measure of individual aptitude,” Military Psychology, 1(2), 111-116, 1989.
[13] Hursh, S. , A human performance model for a simulated command entity,” Unpublished Technical Report, US Army Research Institute, publication in process, 1998.
[14] Orasanu, J.M. & Backer, P. “ Stress and military performance,” In Stress and Human Performance, pp 89-125, Driskell, H.E. & Salas, E. (Eds.) Mahwah, NJ: Lawrence Erlbaum Associates, 1996.
[15] Tversky, A. & Kahneman, D. “Availability: A heuristic for judgment frequency probability.” Cognitive Psychology, 5, pp 207-232, 1973.
[16] Slovic, P., Fischoff, B., & Lichtenstein, S. “Behavioral decision theory,” Annual Review of Psychology, 28, pp 1-39, 1977.
[17] Lopes, L. “Some thoughts on the psychological concept of risk,”. Journal of Experimental Psychology, 9, p 137-144, 1983.
[18] Janis, I.L. “The patient as decision maker,” In D. Gentry (Ed.), Handbook of behavioral medicine. New York: Wiley, 1983.
[19] Orasanu, J., Connolly, T. “The reinvention of decision making,” In G.A. Klein, J. Orasanu, R. Calderwood, & C.E. Zsambok (Eds.), Decision making in action: Models and methods pp 3-20, Norwood, NJ: Able, 1993.
[20] Klein, G. (in press). “The recognition-primed decision (RPD) model: Looking back, looking forward,” In C.E. Zsambok and G. Klein (Ed.), Naturalistic decision making. Mahwah, NJ: Lawrence Erlbaum Associates.
[21] Klein, G. & Crandall, B.W. “The role of mental simulation in problem solving and decision making,” In P. Hancock, J. Flash, J. Caird, & K. Vicente (Eds.), Local applications of the ecological approach to human-machine systems (pp. 576-580). Dayton, OH: Human Factors Society, 1995.
[22] Klein, G. “The effect of acute stressors on decision making,” In Stress and Human Performance, Driskell, H.E. & Salas, E. (Eds.) Mahwah, NJ: Lawrence Erlbaum Associates, pp 49-88, 1996.
[23] Klein, G. “Recognition-primed decisions.,” In W. Rouse (Eds.), Advances in man-machine systems research (Vol. 5 47-92). Greenwich, CT: JAI Press, 1989.
[24] Klein, G., Wolf, S., Militello, L., & Zsambok, C. “Characteristics of skilled option generation in chess,” Organizational Behavior and Human Decision Processes, 63-69, 1995.
[25] Stokes, A.F., Kemper, K.L., & Marsh, R. Time-stressed flight decision making: A study of expert and novice aviators , (TR ARL-93-1/INTEL-93-1). Urbana-Champaign, IL: Institute of Aviation, 1992
[26] Yates, J.F. Judgment and decision making. Englewood Cliffs, NJ: Prentice-Hall, 1990
[27] Belenky, G., Balkin, T.J., Thomas. M.L., Redmond, D.P., Kant, J., Thorne, D.H., Sing, M.S., Wesensten, N.J. & Bliese, P.. Report of the Sleep and Performance Work Group Division of Neuropsychiatry, Walter Reed Army Institute of Research, 1993.
[28] Thorne, D.R., Genser, S.G., Sing, H.C., Hegge, F.W. “Plumbing human performance limits during 72 hours of high task load,” p. 17-40, in Proceedings of the 24th DRG Seminar on the Human as a Limiting Element in Military Systems, Toronto: Defence and Civil Institute of Environmental Medicine, 1983.
[29] Banderet, L.E., Stokes, J.W., Francesconi, R., Kowal, D.M., Naitoh, P., “Artillery teams in simulated sustained combat: Performance and other measures,” P 581-604 in DHHS (Department of Health and Human Services) NIOSH (National Institute for Occupational Safety and Health) Report 81-127: The Twenty-Four Hour Workday: Proceedings of a Symposium on Variations in Work-Sleep Schedules, L.C. Johnson, D.I. Tepas, W.P. Colquhon & M.J. Colligan, eds. Washington, DC: Government Printing Office, 1981.
[30] Horne, J.A. Sleep loss and “divergent” thinking ability, Sleep 11 (6) pp 528-536, 1988.
[31] Angus, R.G. and R. J. Heslegrave. “Effects of sleep loss on sustained cognitive performance during a command and control situation,” Behavior Research Methods, Instruments, & Computers, 17(1), pp53-67, 1985.
[32] Crego, J. and Spinks, T. “Critical incident management simulation,” In Decision Making Under Stress, Flin, R., Salas, E., Strub, M., & Martin, L. (Eds.) Brookfield: Ashgate, pp. 85-94, 1997.
[33] Klein, G. “The current status of the naturalistic decision making framework,” In Decision Making Under Stress, Flin, R., Salas, E., Strub, M., & Martin, L. (Eds.) Brookfield: Ashgate, pp. 11-28, 1997.
[34] Ebbinghaus, H. (1913) Memory: A contribution to experimental psychology. Trans. By Ruger, H.A., and Bussenius, Clara E. New York: Teachers College, Columbia Univer., Bureau of Publications
[35] Gillis, P.D., Hursh, S., Guest, M., & Sweetman, B., (1998) Command entities cognitive behaviors for computer generated forces, Unpublished Technical Report, US Army Research Institute, (publication in process).
[36] Locklear, J., Powell, C., & Fiedler, F. (1988). “The effects of individual experience and intelligence on decision performance,” Proceedings of the 30th Annual Conference of the Military Testing Association. Arlington, VA, pp. 135-140.
Author Biographies
Philip D. Gillis, Ph.D., is a senior research psychologist and program manager at the U.S. Army Research Institute, Simulator Systems Research Unit at STRICOM in Orlando, FL. He currently sits as a voting member on the Army Modeling and Simulation Policy and Technology Working Group. Dr. Gillis has been active in constructive simulation development for ten years and coded the initial architecture for the Command, Control, and Communications Simulation. He has published articles in the areas of training, simulation, and artificial intelligence for the past 17 years. His research interests are in the areas of: the utilization and effectiveness of simulation for training purposes, human performance and cognitive modeling on the battlefield, and intelligent tutorial systems.
Steven R. Hursh, Ph.D., was the senior research psychologist for the Army and consultant to the Army Surgeon General for research in the behavioral sciences. Upon retirement from the Army, he became the Program Area Manager for Biomedical Modeling and Analysis at Science Applications International Corporation. Dr. Hursh holds a joint appointment as Professor of Behavioral Biology at the Johns Hopkins University School of Medicine and directs NIH funded basic research. Recently, Dr. Hursh developed the first comprehensive model of the effects of sleep deprivation, work schedule and circadian rhythms on performance, and he is extending this model for use by the Air Force to predict pilot performance for military and civilian applications. This combined experience was applied to assist ARI develop the Human Performance Model described in this paper.
-----------------------
Figure 1. Naturalistic decision making under stress: decision tree showing effects of stress, fatigue, time pressure, training and experience.
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related searches
- author s argument worksheets middle school
- 4 author s purpose for writing
- 4 types of author s purpose
- examples of author s purpose
- 3 types of author s purpose
- guide to being a man s man
- author s purpose examples for kids
- guide to preparing financial statements
- author s purpose for kids
- a man s guide to women
- java a beginner s guide pdf
- author s purpose for writing