next up previous
Next: 6.8.2 Performance Up: 6.8 SUO Evaluation Previous: 6.8 SUO Evaluation

6.8.1 Quality of alerts

Figure 7 presents the total number of alerts by type at each echelon during a typical run. Flash is the highest of four priorities, and ``immediate" the second highest. Flash alerts are generally life threatening (first contact with adversarial entities and fratricide), while lower-priority alerts are only plan threatening.

We analyzed and evaluated the alerts generated from our first and most challenging scenario. Analysis by SRI and our domain experts indicates that all important situations were alerted. Less than 10% of alerts were judged to have such low value that they should not have been issued, and no Flash alerts were so judged. Judging the VOA for each alert is subjective: different domain experts may have different alerting preferences, and each alert will have some new information. We have no firm data on the number of unwanted alerts that would lead to performance degradation for a typical user (or that would cause a user to shut off his EA). It is clear that the 86% false-alarm rate found in a pediatric ICU [40] would not be acceptable on the battlefield. In our judgment and that of our domain experts, the rates of low-value alerts we achieved are acceptable. The number of alerts in Figure 7 is reasonable for a 90-minute interval of fast-paced action, and further elimination of alerts risks missing a high-value alert. We purposefully erred on the side of not missing any alerts.

Figure 7: Number of alerts by type at each echelon. The number of missions for each echelon indicates the size of its plan. Only the last 90 minutes of the 13.5-hour scenario was simulated at a high fidelity. Of the 78 Battalion alerts, all but 5 were issued over the last 90 minutes.
\begin{figure}\begin{center}
{\small\begin{tabular}{\vert c\vert l\vert}
\hline
...
... alert\\
0 & all other alerts\\ \hline
\end{tabular}}
\end{center}\end{figure}

We have compared the alerts generated by EAs operating at different echelons (running on different machines on the SAIM network) on the same simulation. Our analysis shows that they detect the same threat at the same time from the same tracks, when the threat is relevant to their plans. The alerts show plan-specific and mission-specific behavior as expected. Because of the nondeterminism inherent in our asynchronous agents, the alerts do not always show the exact same strength, bearing or location of a threat. Figure 8 shows one example, the BN and A CO alerts near 08:05. At this time, 2nd PLT, A CO is moving outside its unit boundary specified in the plan, and a hostile force appears to the north of A CO moving south. Note that both EAs issue flash fratricide alerts at 8:05. However, the other alerts are different, and specific to the plan and owner of that EA, as we would expect.

Figure 8: BN and A CO alerts around 0805 on Day 2. There was only one A CO alert from 8:03 to 8:06, during which time there were several BN alerts.
\begin{figure}{\small \centerline{\textbf{From BN EA} 0803-0805\\ }\hspace*{.15i...
... 2 PLT} is out of position near 3-3-B-2-66\\ }
\vspace*{-.15in}
\par\end{figure}

The plan called for an Attack-By-Fire mission if tracks are observed at location DP2 (a decision point at which hostile activity calls for a human decision). This immediate alert appears only on the BN EA (because the contingent fire mission is in only the BN plan) and notifies the user that hostile entities have entered DP2, triggering the contingency. AA-Diamond is a route, defined in the BN plan, along which adversaries are likely to approach. The second alert notifies the BN user (only) of activity on the route and reports the number of entities detected.

Both EAs independently identify the fratricide risk at 8:05, as would the EAs of the two platoons involved. The message details the two platoons to facilitate a quick response. Next, the BN EA issues a distance alert after detecting tracks 450m SE of the Recon PLT, which was subordinated to the BN earlier in the plan (so only the BN EA alerts). These tracks are now closer than when an earlier first-contact alert was issued. Finally, the out-of-position alert at 07:58 indicates that 2 PLT is 1 km south of the route specified for its move mission. (The 2 PLT EA simultaneously alerts that one or more of its subordinate squads are out of position.)


next up previous
Next: 6.8.2 Performance Up: 6.8 SUO Evaluation Previous: 6.8 SUO Evaluation
Pauline Berry 2003-03-18