Engine Expert

Engine Expert

Reference for making changes to the Zeebe workflow engine with confidence. The engine processes thousands of commands per second on a single thread, state is rebuilt from the log on every restart, and partitions communicate over an unreliable network. Mistakes here cause leader/follower divergence, upgrade-path breakage, or production performance regressions. This skill exists to prevent the well-known classes of mistake.

Iron rules

These are summaries. The reference files (linked below) are authoritative — they carry the rationale, exceptions, and worked examples.

• No state mutation from processors. State changes only through events applied by event appliers.
• Released event appliers — and any method on a Mutable*State interface or anything transitively called from an applier — must not change in logic. Add a new version/method instead. Cosmetic changes (formatting, imports, comments, behavior-equivalent renames) are fine. No golden file protects state class methods — extra care required.
• New event applier versions must reach every newer minor before its initial release. A version that ships only in an older minor's patch breaks upgrade replay — dead partitions, unrecoverable without an ad-hoc patch. See event-appliers.md.
• Processors must end the command's processing by appending an event or a rejection. Exceptions are a last-resort rollback mechanism and are expensive — prefer pre-validation.
• Generated keys must be used as the record key of at least one appended record. Otherwise the key generator can't be rehydrated on replay and may hand out a duplicate.
• Inter-partition command receivers must be idempotent. Prefer reject-redundant-command over re-emitting events.
• Hot paths log at trace level only. INFO/DEBUG on a hot path is a performance regression at engine throughput.
• Values returned from ColumnFamily.get(...) / state reads are backed by a shared, mutable buffer reused on the next read of the same column family. Never cache them or hold them across another read — copyFrom(...) if you need to keep the value, or use get(key, valueSupplier) to allocate a fresh instance.
• Non-transactional side effects (metrics, post-commit hooks) run only after all state and follow-up command writes that could throw. Exceptions roll back state, but already-executed side effects don't roll back — counters drift.
• No unbounded recursion unless documented why it's safe. Especially for process trees, call activities, and other user-provided input. A StackOverflowError would ban the executing instance. Bound recursion or use iteration/trampolining.

Where to read next

Local checks before commit

Run only the tests relevant to your change — running the full engine test suite locally takes a long time and is what CI is for.

# 1. Format (mandatory before commit when touching Java/markdown/pom.xml).
./mvnw license:format spotless:apply -T1C

# 2. Tests scoped to your change (single class or small pattern). Use
#    -Dtest=YourTest, comma-separated classes, or a glob like '*UserTest*'.
./mvnw verify -pl zeebe/engine \