Azure Batch runs work that nobody watches, which breaks the feedback loop by design. A node lost mid-task doesn't page anyone. The consequence appears later, in whatever process expected the output.

Ambiguity is what makes it expensive. Teams often can't tell after the fact whether a task completed, restarted, or produced partial results, and reconstructing that from logs takes longer than the job did. If tasks aren't idempotent, an automatic retry compounds the problem by writing the same work twice.

Gremlin gives you those answers while no deadline is attached. Install the Gremlin Agent on your VM pool nodes, then run Shutdown experiments during an active job to see whether the pool reschedules cleanly and whether tasks resume or start over. Process Killer experiments simulate a crashed task process without taking the node with it.

The answer determines how much manual verification your team performs on every run from here on, which makes it worth establishing once rather than assuming forever.