CCS Experiments

Use this before submission when the attack demonstration, defense evaluation, or measurement story is not yet locked.

Experiment audit

• Map each security claim to a specific artifact: an exploit run, an overhead measurement, a

coverage number, a false-positive/false-negative table, or a measurement dataset.

• For attacks, demonstrate the exploit against a realistic, named target (software version,

platform, configuration) and report the resource cost to the attacker.

• For defenses, evaluate against an adaptive attacker built with knowledge of the defense,

and report performance overhead, memory cost, and any compatibility breakage.

• For measurements, validate sampling: document the population, the vantage point, coverage

and blind spots, and ground-truth checks against known cases.

• Include baselines that represent the state of the art in attack or defense, not strawmen.
• Report variance for stochastic results and audit for leakage, selection bias, and any

mismatch between the threat model and the tested configuration.

What experiments are for at this venue

• CCS experiments exist to make a security claim undeniable to a skeptic, not to top a

benchmark. One clean end-to-end exploit against a real target outweighs a table of micro-benchmarks.

• The strongest defense design triad: the attack it stops, an adaptive attack that knows the

defense, and a deployment-cost measurement. Missing the middle element is the classic CCS defense reject.

• Reviewers, often practitioners, check whether the evaluation environment matches the threat

model. A defense claimed for production but tested only on a toy in a lab invites the relevance question.

Attack-and-defense evaluation table

| Security claim | Matching evidence | Reject pattern avoided | |---|---|---| | Exploit is practical | End-to-end run on named target with attacker cost | "Works only in a lab against a strawman" | | Defense stops the attack | Detection/prevention rate on the original attack | "No numbers, only a design argument" | | Defense resists adaptation | Adaptive attacker with defense knowledge, degraded results | "Only the non-adaptive attack was tried" | | Deployment is feasible | Overhead, memory, compatibility on a realistic workload | "Security claimed, cost never measured" |

Vignette: evaluating a control-flow-integrity defense

Suppose the paper proposes a fine-grained CFI scheme. The matching plan: reproduce a known code-reuse attack and show it blocked; construct an adaptive attacker that respects the CFI policy and search for surviving gadget chains; then measure runtime overhead and binary-size growth on a standard benchmark suite. Every claim ties to a numbered table, and the adaptive result is reported even when it dents the headline.

Reporting floor

• Name every target's exact version and configuration; "a popular browser" is not a target.
• Report the attacker's resource budget (queries, time, samples) and the defense's measured

overhead rather than vague "negligible cost" language.

Output format

text
[Experiment readiness] strong / adequate / weak
[Claim -> evidence map] <claim: exploit run / overhead table / measurement>
[Missing security evidence] <adaptive attack / baseline / cost / validation>
[Threat-model mismatch] <where the setup breaks the stated model>
[Decision-critical next run] <one experiment>