Project:	IST-1999-11562	
Acronym:	NESPOLE!	
Title:	NEgotiating through SPOken Language in E-commerce	

Title:	Assessment and Evaluation Plan	
Version Number:	1.4.1 	
Deliverable:	D21	
Author(s):		
Organisation:	ITC-irst	
Participation:	----------	
Date:	4 April 2000	
Status:	DRAFT	
Security:	Internal	
Keywords:	Monitoring progresses, Risk analysis, Recovery Strategies	

Index

Index	2
Assessment and Evaluation Plan	3
1.	Introduction	3
2.	Monitoring the project's progresses	3
3.	Checking the achievement of the Milestones	3
4.	Assuring the quality of the documents	4
5.	Testing and verifying technical results	5
6.	Carry out risk analysis and assessment	5
7.	Fallback and recovery strategies	8
7.1	Managing the Risk	8
8.	References	10

Assessment and Evaluation Plan

1.	Introduction
The Evaluation and Assessment Plan aims to provide the procedures suitable for implementing the following processes:

•	monitoring the project's progresses;
•	checking the achieving of  the milestones;
•	quality control;
•	testing and verifying the technical results;
•	carrying out the risk analysis and assessment;
•	fallback and recovery strategies.

In the following the different phases of the Evaluation and Assessment Plan will be analysed.

2.	Monitoring the project's progresses
Each WP leader is responsible for monitoring how the work proceeds, so as to assure a smooth implementation of the planned activities, by taking care of both technical and organisational matters. 
In order to accomplish these goals, the Project's Co-ordinator has to maintain frequent contacts with the WP leaders. Particularly in order to monitor the project's progresses, each WP leader has to prepare a brief document that will be useful to produce the progress reports.
This document has to be made available monthly by each WP leader and it will be structured in the following descriptive parts: 
•	Covered period.
•	Involved workpackage and task.
•	Involved persons and resources.
•	What it has been done.
•	Eventual problems came out during the considered period.
•	Evaluation of the advancement status compared to the Milestones.

3.	Checking the achievement of the Milestones
This process aims to check the achievement of the Milestones, by going through one of the following procedures:
•	In case the Milestone is a paper-document by means the quality control process;
•	In case the Milestone consists of a system or piece of software by setting up and running suitable tests to verify it, according to the decisions and procedures established in WP7.

4.	Assuring the quality of the documents
In order to assure the quality of the documentation, the Quality Control has to be integrated with the organisational set-up (see the Figure 3.1) in such way that:
•	The WP leader submits the deliverable to the Project Director's supervision and assessment;
•	the Project Director sends it to the Project Management Committee (PMC) for approval;
•	the PMC Co-ordinator forwards the deliverable to the EU Commission. 

DELIVERABLES

Emission

Supervision and Assessment

Approval

Forwarding
 to the Commission

WP leader

Project Director

PMC

Co-ordinator

Information flow

Procedure of quality control

Figure 4.1 Process of quality control

In order to facilitate quality control, other practices could be adopted:
•	a quick and frequent updating of the NESPOLE! Web site;
•	a detailed and informative annotation of draft documents and deliverables, to be provided at the beginning of each document.

By quickly and frequently updating the Project's Web site, information and document exchanges among the partners is made easier. A suitable document-initial annotation helps to classify the document with respect to:
•	identification number (the first figure identifies the workpackage involved, the second is a progressive number that identified the document inside the same workpackage, the third identify the document version);
•	deliverable number;
•	author(s);
•	organisation;
•	participation (to indicate the partners involved in the workpackage);
•	date;
•	status (draft or deliverable);
•	security status (internal, public, or different status);
•	keywords (some words suitable to identify the document content).

5.	Testing and verifying technical results
Tests and activities for verifying technical results is planned and managed by WP7 "Testing and evaluation". 
Its objectives are to test and evaluate: 
•	individual components of STST systems with respect to robustness, sentence-based evaluation, coverage and accuracy.
•	whole STST systems (both single- and multi-engine) with respect to task-based evaluation, sentence-based evaluation, coverage and accuracy. Comparative evaluation.
•	showcase evaluation on the basis of user’s satisfaction.
•	multimedial and multimodal features.

The results of various evaluation activities are made available through D9 and D18.
The former document describes the results of the evaluation of: single HLT modules, STST systems and first showcase.
D8 reports on the results of the evaluation of the second showcase, including the evaluation of multimodal features.

6.	Carry out risk analysis and assessment
After the requirements' definition, the specific procedures to carry out the process of risk analysis will set up. The risk analysis will consider both matters related to systems and technologies, and those relating management. To evaluate the risk to be beyond schedule a control process based on the direct communication among partners, e.g. e-mail messages or memo published on the web site, will be adopted.
Methods for carrying out risk analysis at the level of system requirement are more articulated and complex and require established procedures and high level methods that allow both a systematic approach, and flexibility while operating.
The method proposed is based on the "Comparative Risk Assessment" or CRA.
CRA is an approach for organising, analysing, evaluating and placing into prospective information about the risks, costs, and benefits of technological alternatives. It can be used as an aid to decision making while doing risk-management.
A generic CRA approach describes the methodological steps rather than prescribing specific procedures. This way it allows for flexibility at the level of requirements.
The following diagram reports on the risk analysis and assessment procedures for the NESPOLE! project.

Diagram of the risk analysis and assessment

4. Ascertain institutional
factors

2. Identify assumption

1. The scope of the assessment is defined from time to time

3. Define decision objectives and options in accordance with project main objectives

6. Summarise and report findings.

5. Analyse and compare positive and negative aspects related to the alternative decisions

1	During each phase of the project's life cycle a conflict could occur because of problems with alternative solutions. 
2	The criteria for choosing should be individuated by the WP leader with the consensus of the Technical and Administrative Directors. If consensus can't be reached, then the decision will be taken by voting (the procedure is available in the Annex 1).
3	The decision objectives and option should be defined in accordance with the main objectives of the NESPOLE! project.
4	A diverse assortment of institutions, practices, and relationship usually affect the feasibility of implementing alternative risk-related options. In the NESPOLE! Project many partners have already worked together. This suggests that procedures and routines used in the past, which have proven to be effective ,can be re-used. 
5	In the case of NESPOLE! the analysis and the evaluation of positive and negative aspects related to the alternative decisions are carried out mainly considering technical alternative solutions, and monitoring the market trends. The users groups help in this activity.
6	All the presentation's and dissemination's tasks are foreseen in the Annex 1 inside the Deliverable and Milestones lists.
The following figure represents the decision process under two points of view (involved knowledge and ensuing values).

Figure 6.1 Decision Process and risk evaluation.

Knowledge

Description of alternatives in terms of decision criteria

values

RISK

Evaluation of decision outcomes

Specification of decision criteria

Technical description of alternatives

Specification and alternative actions

7.	Fallback and recovery strategies
Fallback and recovery strategies have to be defined basing on of the risk analysis findings.
The following check list  could be useful in risk managing.

7.1	Managing the Risk
The relevant questions focus on the risks underlying the project. 
They have to be dealt with in order to ensure that risks are clearly identified, that the impact of any potential failure has been evaluated, and that suitable strategies are in place for dealing with risk. The following is a sample of the relevant issues:
•	What can go wrong with the project, and why, i.e.: proven technology? in-house methodology experience? resource constraints? robustness of cost estimates? mission critical process? time deadlines? 
•	What would happen if it does go wrong, and how disruptive would it be? 
•	What can be done about this, and is it enough? 
•	Has an action plan been prepared detailing the "who, when and what"? 

In order to manage the risk a Fallback and Recovery Strategy has been set up.
This strategy has to be adopted after the risk identification (foreseen risk).
The strategic phases are:

1.	Risk identification and evaluation
2.	Priority communications
3.	Routine communication
4.	Emergency Response Team
5.	Emergency Phase
6.	Recovery Phase

More in detail:
1.	All the Project participants and also some external subjects can contribute identifying possible risks (technical, organisational, schedule-related, or any other kind of risks).
2.	Priority Communications have to be made to the Project Director or in his absence to the Technical Directors.
3.	Routine Communication are directed to the Project Director or Project Technical Directors, who consult the Project Management Committee (PMC) and the Project Technical Committee (PCT) to set-up the specific procedure related to the risk management.
4.	The Emergency Response Team becomes operative if there is neither time, nor possibility to consult the Project Management Committee and the Project Technical Committee. The team is constituted by the Project Directors and by Technical Directors.
5.	The Emergency Phase involves all the resources capable to prevent the risk's negative effects and consequences.
6.	The Recovery Phase is planned by the Project Director and approved by the Project Management Committee and Project Technical Committee. It is adopted to plan recovery procedures.

Figure 7.1 Fallback and Recovery Strategy

Emergency Response Team

Communication to the Project Director

Routine communication

Technical Project Directors

Risk identification and evaluation

Communication to PMC and PTC

Recovery Phase

Emergency Phase

8.	References

•	Covello V.T. - Menkes J. - Mumpower J., Risk Evaluation and Management, Contemporary Issues in Risk Analysis, Volume 1, Plenum Press, New York, 1986.
•	Covello V.T. - Lave L.B.-Moghissi A.-Uppuluri V.R.R., Uncertainty in Risk Assessment, Risk Management, And Decision Making.
•	IFAC, International Federation of Accountants, Information Technology Committee Information & Communications Technology Projects (Final) International 
http://www.ifac.org/Committees/ITC/projsandpubs/9812/ITC-ICtechproj-final.htm
Information Technology Guidelines, 1998. 
•	Vlek C. - Cvetkovich G., Social Decision Methodology for Technological Projects, Theory and Decision Library, Series A: Philosophy and Methodology of the Social Science, Kluwer Academic Publisher, Netherlands, 1989.