Gremlin and Azure Data Lake Storage

Test how your application handles a Data Lake Storage outage. Run Gremlin network experiments and reliability tests to validate pipeline resilience.

Why Data Lake Storage reliability is important

Analytics pipelines built on Azure Data Lake Storage assume the storage layer is always there, right up until a batch job fails partway through and nobody is sure how much of the output is complete.

That ambiguity is worse than a clean failure, because it can't be resolved by re-running. Downstream consumers rarely account for it either: a job that reads whatever files exist will happily process a partial dataset and produce results that look finished. Nothing errors. The numbers are just wrong, and they stay wrong until someone notices they don't reconcile against something else.

Gremlin lets you interrupt a pipeline on purpose. A blackhole experiment against the storage endpoint cuts access mid-job so you can see whether your jobs fail atomically or leave partial state behind. A latency experiment slows large reads to find the operations with no timeout at all.

For anything feeding reporting or machine learning, silently incomplete input is the failure that costs the most to discover late.

Building resilience on Data Lake Storage with Gremlin

Data Lake Storage is a dependency: a managed service that your service connects to. You can run tests by deploying the Gremlin agent or Failure Flags sidecar to a service that consumes Data Lake Storage*. Using the experiments shown to the right, you can prepare your service for Data Lake Storage failure modes, including:

  • Network outages making Data Lake Storage unavailable
  • Slow performance due to network latency
  • Expiring TLS certificates
You can also run these expert-built workflows designed to replicate real-world failure modes on Data Lake Storage:

* Testing a dependency doesn't require installing anything on it. Gremlin runs the experiment from the service that calls it, so compatibility depends on that host rather than on the dependency itself. Check our compatibility documentation for supported operating systems and platforms, or get in touch if you don't see yours.

resources

Learn more about Gremlin and Data Lake Storage

All product names, logos, and brands are property of their respective owners. AWS is a trademark of Amazon.com, Inc.; Azure is a trademark of Microsoft Corporation; Google Cloud is a trademark of Google LLC. Use of these names is for identification purposes only and does not imply endorsement or affiliation unless otherwise stated.

Avoid downtime. Use Gremlin to turn failure into resilience.

Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.

Product Hero ImageShape