http://mir.cs.illinois.edu/issta2014/artifacts.html ISSTA 2014, Call for Artifacts ISSTA 2014 will allow authors of accepted research papers to submit artifacts along with their papers. An artifact can be any kind of content related to a paper, e.g., detailed experimental data, complete experimental setup, test suites, or tools. Submitting an artifact is optional. Two members of the Artifact Evaluation Committee (AEC) will review an artifact with respect to the following criteria (if applicable): How easy is it to use the provided artifact? (Easy to reuse) Does the artifact help to reproduce the results from the paper? (Consistent) What is the percentage of the results that can be reproduced? (Complete) Does the artifact describe and demonstrate how to apply the presented method to a new input? (Well documented) Accepted artifacts got an official badge of approval (courtesy of Matthias Hauswirth) which can be shown in the proceedings and in presentations It will be up to the authors whether they make the results of the evaluation process public or not. We encourage authors to make accepted artifacts available online! Important Dates Submission Deadline April 18, 2014 (Friday), extended from April 11 Notification May 26th, 2014 (Monday) Submission Instructions All submissions have to be made through: https://www.easychair.org/conferences/?conf=issta2014ae. We expect the authors to submit only a link to a single file containing all the information needed to evaluate the submitted artifact (see the following section). The artifact evaluation chairs will download the artifact and distribute it to the reviewers (in order to hide the identity of the reviewers from the authors). The authors should additionally make an effort not to learn the identity of the reviewers, e.g., through logging. Packaging Guidelines High quality packaging of an artifact is as important as the quality of the artifact itself. Please keep in mind that the committee members will have limited time to review each artifact. We have some requirements for the artifact submission that will expedite the review process. Please submit a single zip archive, containing a README text file with instructions, the paper, and the artifact itself. Additionally, you may submit a video with a demo and/or instructions. The name of the file should be .zip. We provide a template for the README file. Fill it out without modifying the lines starting with "#". In case you want to submit a running setup of your tool, please use the virtual machine image that we have prepared (and include it in the zip archive). Please read our instructions on how to run the virtual machine. You can install any software that is needed to demonstrate the artifact. Assume that no Internet connection will be available for the review! Previous Artifact Evaluations ISSTA is not the first venue to offer an artifact evaluation process but continues a recent trend in Software Engineering conferences (cf. FSE 2011, ECOOP 2013, and OOPSLA 2013). The motivation for such an extension of the submission and review process is twofold. Submitting artifacts (e.g., tools or detailed experimental results) allows for a more thorough assessment of the claims made in a paper. Publishing artifacts (e.g., test suites or an executable experimental setup) makes it easier for other researchers to reproduce results from a paper or to compare their own techniques with the one presented in a paper. Previous artifact evaluation committees have put some thought into how to introduce artifact evaluation without scaring potential authors from submissions. Detailed accounts can be found here and here. We generally follow these considerations and hope that the authors will submit their results for the evaluation. Artifact Evaluation Chairs Milos Gligoric, University of Illinois at Urbana-Champaign, USA. Falk Howar, Carnegie Mellon Silicon Valley, USA. Artifact Evaluation Committee Alessandra Gorla, Saarland University, Saarbrucken, Germany Wei Jin, Georgia Institute of Technology, USA Temesghen Kahsai, NASA Ames, USA Baris Kasikci, Swiss Federal Institute of Technology in Lausanne, Switzerland Aleksandar Milicevic, Massachusetts Institute of Technology, USA Sasa Misailovic, Massachusetts Institute of Technology, USA Kivanc Muslu, University of Washington, USA Johannes Neubauer, TU Dortmund, Germany Gustavo Soares, Universidade Federal de Campina Grande, Brazil Linhai Song, University of Wisconsin-Madison, USA Andrei Stefanescu, University of Illinois at Urbana-Champaign, USA Shin Hwei Tan, National University of Singapore, Singapore Yida Tao, The Hong Kong University of Science and Technology, Hong Kong Xusheng Xiao, North Carolina State University, USA Lingming Zhang, University of Texas at Austin, USA Sai Zhang, University of Washington, USA