This page links to the individual files produced during the open card sorting process for the survey analysis. For a description of the format used see the Survey Analysis Process and Datafile Description.
| Code | Survey Question | Labelled Answers |
|---|---|---|
| G3 | Please elaborate on your previous answer: (How helpful do you find these criteria when reviewing artifact?) | AEC-Survey-G3.xlsx |
| G4 | In your words, what is the purpose of artifact evaluation? | AEC-Survey-G4.xlsx |
| AE1 | What is your minimum requirement to accept an artifact in an artifact evaluation process in general? (e.g., reproducibility of article results, easy setup, documentation) | AEC-Survey-AE1.xlsx |
| AE2 | What is your minimum expectation for the code portion of an artifact? (e.g., code quality, documentation, packaging, size) | AEC-Survey-AE2.xlsx |
| AE3 | What is your minimum expectation for the proof portion of an artifact? (e.g., completeness, understandability) | AEC-Survey-AE3.xlsx |
| AE4 | What is your minimum expectation for the data portion of an artifact? (e.g., raw data, data format, format documentation, size) | AEC-Survey-AE4.xlsx |
| AE5 | Do you have any other comments on the requirements for accepting an artifact? | AEC-Survey-AE5.xlsx |
| AE7 | Please elaborate on your previous answer: (Do you think that the effort of artifact evaluation is justified?) | AEC-Survey-AE7.xlsx |
| AE8 | What are the reasons why you have recommended to accept or reject an artifact? | AEC-Survey-AE8.xlsx |
| AE9 | Which arguments of your fellow artifact evaluation committee members for the acceptance or rejection of an artifact do you recall? | AEC-Survey-AE9.xlsx |
| AE11 | Which reasons for the acceptance/rejection of your artifact do you recall? Please indicate for each reason if you consider it justified or not. | AEC-Survey-AE11.xlsx |
| AU2 | Please elaborate on the (un-)met expectations towards the code: | AEC-Survey-AU2.xlsx |
| AU5 | Please elaborate on the (un-)met expectations towards the proofs: | AEC-Survey-AU5.xlsx |
| AU8 | Please elaborate on the (un-)met expectations towards the data: | AEC-Survey-AU8.xlsx |
| AU11 | Please elaborate on your previous answer: (Have you noticed a difference between artifacts with and without a successful evaluation?) | AEC-Survey-AU11.xlsx |
| AU12 | Do you have any other comments on artifact usage from the perspective of a researcher? | AEC-Survey-AU12.xlsx |
| F01 | Please elaborate: (Are you satisfied with the current artifact evaluation process?) | AEC-Survey-F01.xlsx |
| F02 | In your perception, has the quality of accepted artifacts improved compared to earlier years? Please elaborate. | AEC-Survey-F02.xlsx |
| F03 | What factors influence the quality of an artifact (e.g., the research group, the domain, the type of artifact)? Please elaborate. | AEC-Survey-F03.xlsx |
| F99 | If you have any comments that did not fit into our survey questions and that you wish to share, please elaborate here: | AEC-Survey-F99.xlsx |