(Message inbox:10398) Return-Path: Received: from N3.SP.CS.CMU.EDU by B.GP.CS.CMU.EDU id aa12986; 10 Dec 95 10:29:40 EST Date: Sun, 10 Dec 1995 10:10-EST From: Thomas.Gross@N3.SP.CS.CMU.EDU To: "/usr0/trg/priv/review/23isca/list.dst" Subject: ISCA 96 Message-Id: <818608227/trg@N3.SP.CS.CMU.EDU> With this mail you receive a) Norm Jouppi's (PC Chair) "Instructions to reviewers" b) review form I'm sorry it took a while to get these to you, but my mail system was broken. Remember: Jan 5, 1996 is the target date for the reviews. Thanks again a lot for helping out, and have a nice holiday season tg ---------------------------------------------------------------------------- INSTRUCTIONS TO REVIEWERS The review process is as important as the research and writing that goes into papers appearing at a conference. Without a good review process, good work can be lost and erroneous work widely propagated. There are usually several sources of error and uncertainty in the review process. This year I would like to significantly reduce these sources. REVIEW CONTENT The most important part of a review form is not the numerical recommended action for the paper, but rather the comments that can be communicated to the authors and the program committee. If a review has nothing written in the comments section, its recommendation may be largely ignored during the review process if other reviews with conflicting recommendations have comments. Moreover, all papers should benefit from being submitted to a conference. Authors who receive little or no constructive feedback from the comments section have little direction on how to improve their work. In your comments please summarize and critique both evaluation methods and new ideas. If you do not plan to write at least a paragraph in the comments section, please contact the person who sent you the paper about finding an alternate reviewer. REVIEWER EXPERIENCE If you have received a paper to review, it was because the person sending you the review thought you were qualified to review the paper. If you feel you are not qualified, please contact the person who sent you the paper about finding an alternate reviewer. Please do not forward papers to other reviewers without notifying and getting the approval of the person who sent you a paper. If you forward the paper to someone who may be relatively inexperienced with the review process (e.g., a new graduate student), please make sure you read and agree with the review before it is returned. New reviewers need to learn to be better reviewers, and without feedback learning is rarely accomplished. Please make sure these directions are forwarded along with the paper. MEAN AND VARIANCE OF REVIEWER SCORES Although the written comments are a primary determinant of a paper's fate, the numerical scores are used for an initial ranking of papers. It is important that as much as possible all reviewers use the same scale for their reviews. Unfortunately, every year there are reviews of the form "This idea could never be implemented and the analysis was hopelessly flawed but I liked the paper -> strong accept" and "The authors stated that this nit (which is not material to their results) is X, but it is really x -> strong reject". The calibration of a review is especially difficult if an inexperienced reviewer reviews only one paper. Reviewers can gain a better context by reviewing more than one paper. Also please look over papers in prior year's proceedings as a reference in scaling your scores. Reviewers are encouraged to assign scores in tenths precision (e.g., "3.8") within the range [5.0 .. 1.0]. When recommending an action for a paper, please keep in mind the severity of any problems with the paper. Misstatements in the text can easily be corrected in the final version of the paper; flaws in the methodology cannot. Also, please consider the importance of any new ideas in architecture or methodology versus the strength of the evaluation. A paper which proposes a revolutionary and obviously important new architecture idea or evaluation methodology should be ranked higher than a paper which simulates every instruction of all the SPEC benchmarks without providing any new insight. Thank you all for your help in improving the review process! Norman P. Jouppi ISCA '96 Program Chair ---------------------------DO-NOT-DELETE-THIS-LINE-------------------------- 1996 ISCA Electronic Referee's Report Please return this report by Friday January 5 to jouppi@wrl.pa.dec.com with a carbon-copy (cc) to thomas.gross@cs.cmu.edu Please do not alter any existing fields/labels in this form. Paper Number: Paper Title: ---------------------------DO-NOT-DELETE-THIS-LINE-------------------------- Place a number for each item below corresponding to your evaluation. (5=outstanding, 4=high, 3=medium, 2=fair, 1=poor) Confidence in your evaluation: Interest/Importance to ISCA: Quality of presentation: Technical contribution of paper: ---------------------------DO-NOT-DELETE-THIS-LINE-------------------------- (5=strong accept, 4=accept, 3=maybe, 2=reject, 1=strong reject) Recommended action for paper: ---------------------------DO-NOT-DELETE-THIS-LINE-------------------------- Regardless of the recommendations, please state specific comments and suggestions that can be communicated to the author(s): ===========================DO=NOT=DELETE=THIS=LINE================== Comments on this paper for the Program Committee, TO BE WITHHELD FROM THE AUTHOR(S): ---------------------------DO-NOT-DELETE-THIS-LINE-------------------------- Referee's name: Referee's affiliation: +++++++++++++++++++++++++++DO+NOT+DELETE+THIS+LINE+++++++++++++++++++++++++