avizmarlon/reactive-integration-design

Observe Facts, Not Promises — Reactive Integration Design

When designing a reactive integration to an external system event (container deployment, Kubernetes state change, queue message, OS-level file change), prefer primitive event streams (kernel-level or runtime-level — docker events, inotify, Kubernetes watch API, fanotify, journald) over application-level callbacks (webhooks, SDK callbacks, HTTP API polling).

Why This Matters

Primitive events describe facts. They are generated by the kernel or runtime and represent something that actually occurred in the system. The contract between your integration and the event source is stable across versions because it is enforced at the infrastructure layer, not the application layer.

Webhooks describe promises. They are contracts the application makes to notify you when something happens. Promises break silently:

• When the application upgrades, the webhook payload format may change without notice.
• When the application loses permissions or encounters errors, events get dropped without notification.
• Under high load, the application may batch, skip, or defer webhook deliveries.
• The webhook endpoint itself may be removed or behavior changed without backward compatibility.

The fragility gap: a webhook depends on the application continuing to implement and maintain the notification contract. A primitive event stream depends only on the infrastructure remaining functional—which it must, or the entire system fails anyway.

Examples

• Container deployment monitoring: Instead of polling the Kubernetes API every N seconds or relying on a platform's webhook delivery, consume the Kubernetes watch API directly. The watch API is a runtime primitive; a platform's webhook system is an application-level contract that can change.
• File system integration: Instead of consuming an application's "file changed" callback, use inotify (Linux) or fanotify (Linux, more recent) to observe kernel-level file events. These are OS primitives.
• Container lifecycle: Instead of relying on a container orchestration platform's webhook, consume docker events directly. docker events is a runtime primitive exposed by the Docker daemon.
• Journal events: Instead of polling application logs via HTTP API, follow journald on the host. Journal is kernel-level infrastructure.

How to Apply

When designing a reactive integration:

1. First question: "What is the most primitive event that captures this requirement?" Primitive means kernel-level, runtime-level, or OS-level.
2. If a primitive exists: use it. Build around it. It will outlast application version changes.
3. If no primitive exists: then use webhooks. But when you do:

• Version the payload contract. Expect the webhook payload to change and plan for breaking changes (add a version field, include a timestamp, allow optional fields).
• Monitor the integration. Webhooks are inherently fragile; log every delivery attempt, every missing event, every parse error.
• Have a fallback. If the webhook fails or gets behind, have a secondary polling mechanism (even if inefficient) so the integration can self-heal.
• Document the dependency. The webhook format is not a guarantee—it is a snapshot of the application's current behavior. Document what behavior the integration depends on and what breaks if it changes.

Anti-Patterns

• Relying exclusively on a webhook when a primitive event stream exists.
• Assuming webhook payload format is stable across application versions without explicit guarantees from the vendor.
• Building integration logic that silently fails if a webhook is dropped or delayed.
• Polling an application's HTTP API every N seconds when a primitive event exists at a lower layer.

Testing and Validation

When you choose a primitive event stream:

• It should survive application upgrades unchanged. If the application updates tomorrow, the event stream should still work without code changes.
• It should have explicit documentation. Primitive events are documented as part of the kernel, OS, or runtime, not as a bonus feature of the application.
• It should be version-stable. If there is a version number (e.g., Kubernetes API version), it should be explicitly bumped if the contract changes, not silently changed in a minor release.

References

• Kubernetes Watch API: observing resource changes at the runtime level
• Docker Events API: kernel-level container state changes
• Linux inotify / fanotify: kernel-level file system event streams
• systemd journald: kernel and service logs as a primitive journal
• Message queue primitives (RabbitMQ, Kafka, etc.): consume from the queue directly rather than relying on the producer to re-emit events

Applies to: all AI agents, all projects, all reactive integration design tasks.