/acmmm-artifact-evaluation

name: acmmm-artifact-evaluation
description: Use when packaging code, models, datasets, or media as ACM MM (ACM Multimedia) artifacts — building the anonymous review package versus the public release, and choosing between the Open Source Software Competition, the Dataset track, the Reproducibility track, and main-track supplementary evidence, each with its own blinding and expectations.

ACM MM Artifact Evaluation

Use this to turn an ACM Multimedia project's code, models, media, and data into the *right* artifact for the *right* track. ACM MM has a track economy around artifacts, and the choice determines blinding, format, and what reviewers judge.

Which track is the artifact?

| Artifact is primarily... | Route to | Blinding | Judged on | |---|---|---|---| | A reusable software system/framework | Open Source Software Competition | Single-blind | Adoption, quality, license, docs | | A new dataset/benchmark | Dataset track | Single-blind | Scale, quality, ethics, usefulness | | A reproduction of published results | Reproducibility track | Single-blind | Whether results rebuild; ACM badges | | Supporting evidence for a method paper | Main-track supplement | Double-blind | Whether it backs the paper's claims |

The named single-blind tracks exist *because* the artifact's identity cannot be hidden; a main-track method paper's artifact, by contrast, must be **anonymous** through review.

Two artifacts, two audiences

Plan both from the start:

• **Anonymous review artifact** — what reviewers see during double-blind review: an

anonymized repository, an anonymous data mirror, stripped media metadata, and a README that reveals no author identity.

• **Public release artifact** — what ships at/after camera-ready: the de-anonymized

repository, a permanent archive (DOI), the license, and the final dataset/model.

review/    -> anonymous repo, anon data mirror, no names in code/media, run instructions
release/   -> public repo + DOI, LICENSE, model weights, dataset card, citation

Open Source Software Competition

• The bar is a system others will *use*: clear install, documentation, examples, an

OSI-approved license, and evidence of quality or adoption.

• Reference models and reproducible examples matter more than a single benchmark number —

this is the lane exemplified by community frameworks and portable libraries.

Dataset track

• Ship a **dataset card**: collection method, size, splits, license, consent, and known

biases or limitations.

• Address ethics and rights explicitly, especially for user-generated or scraped media; a

dataset a reviewer cannot legally use is not a contribution.

Licensing and rights decisions

• Choose a code license (permissive vs. copyleft) and a **data** license separately; they are

not the same choice.

• For media, confirm you have the right to redistribute; where you cannot, provide a

retrieval script or agreement path instead of the raw files.

• Record third-party asset licenses so the release is clean.

Ethics and consent for media artifacts

Multimedia artifacts carry people's faces, voices, and content, so the ethics review is not a formality:

• **Consent and rights** — confirm you may redistribute the media; user-generated content often

cannot be re-hosted, so ship a retrieval script or agreement path instead.

• **Privacy** — remove or justify identifiable individuals who did not consent; a dataset of

scraped faces is a rejection risk regardless of its scale.

• **Documentation** — a dataset card that states collection method, consent, license, and known

biases is part of the contribution, not paperwork.

Timeline: review artifact, then release

before paper deadline:  anonymous review artifact ready (repo + data mirror, no identity)
during review:          reviewers/AC access the anonymous artifact
on acceptance:          build the public release (de-anonymized repo + DOI + license)
by camera-ready:        release replaces the anonymous mirror; dataset/model final

Plan the public release early even though it ships late — a scramble at camera-ready is how projects end up with a broken anonymous link and no working public archive.

Output format

[Track] Open Source / Dataset / Reproducibility / main-track supplement
[Blinding] correct for track / mismatch
[Review artifact] anonymous + runnable / gaps: <list>
[Release artifact] archived + licensed / gaps: <list>
[Rights] code+data+media licenses set / open questions: <list>
[Top fixes] <ordered>