December 7, 2018

Kriss Rochefolle: "How to Convince Your Boss to Say "Yes!" to Chaos Engineering" - Chaos Conf 2018

Let's break things in production it's gonna be fun! It's really not the best way to convince your boss. Maybe Earth, like my feet. Sorry. So, let's pick a better way to convince your boss. This made up, in France, we call this the big clogs, and I think here you would call it 'A Bull in a China Shop'. It's really, really not a good way to do it. So, a bit about me, even if everything, I will just write a book about their reps and I'm an agent of chaos.

My background, I work from the French National Railway Company, about trains, we are the first internet website in France to sell tickets from train, and I've working the VSC technology which is a small company who build tools for the best experience in travel, in France. A bit of our story, in Chaos Engineering, we start to think about it in 2015, try our first experiment in 2016. It was not in production, to be able to do it in production we try game day and then we chose Friday 13th of January, a great date, great date, and we start to change the culture and to be able to do it in production.

Our first chaos monkey were in May 2017. Our second day of chaos based on 12 monkeys movie, and this year we are working to put monkeys quite everywhere on our critical reputation. Play a lot of game day, and begin a Chaos portal, a tool for us to follow all our experiment and what we done about it. Like last week, when we simulated the loss of a data center, and tried to see what we can learn about it. The good thing we know it's no impact of business, yeah, the bad thing it's take a lot of time for people to detect what's happened, what the real cause of the problem, so we need to improve observability of our tool.

But, let's go back in time. How we first convince our boss to make them say, "Yes!" To Chaos Engineering? Every time I speak about it everyone respond, answer, which on Earth investment? Okay, but between a project that can make you earn a lot of money, and one which maybe a way to loss money everyone want to do the first. So, region of investment is not enough. Rational way is not enough, you need to speak to the Earth, and when I say to convince your boss, it's all the C-Level not only one people. Is a CIO, CFO, CTO, and so on, so you need to convince everyone of them to do Chaos Engineering. That's why you need to divide up influence strategies. I won't speak about technology and tools, I would only speak about influence strategies.

First steps, make it sound familiar. You know when you use the word Chaos Engineering, everyone of our C-Level people heard Chaos. It's afraid, it create fear. So, in front of fear of unknown our own brain have three choice: fight, "are you crazy? We already have enough critical incidents in production." Or freeze, "Maybe let's talk later." Or fly away, "Oh, I have a call, see you." So, no, you need to make them feel okay with the name of Chaos Engineering. You need to share a lot of articular pause video, the Technology Radar from ThoughtWorks. Everything you need to make them comfortable with the term Chaos Engineering.

My current boss, is a new boss from six months ago. I take around two months to speak about Chaos Engineering but not with enough details. Until one day she said, "Oh, you speak a lot of Chaos Engineering but I don't know what it is. Tell me more." The first step was won, she wants to know more. Second step, around your c-level, you need to know who they are, what are their feels. Personally I use socio dynamics. It's a way to put people in place to identify who that's open, and who don't like you or what you do. Who are influencer, waverer, passive, and you need to adjust the story to every one of them.

The first group, the fools. Which in moaners, opponents. Don't take too much energy on them. The only thing they want to do is to block what you want to do, so if you let them speak a lot they will try to convince everyone that it's not a good idea, so don't use a lot of energy. They will come later when everyone will be convinced. The friends. Zealots and influencers are the good people, they're your friends. The people who think engineering can be interesting and you may use them as promoters. You will make them part of your strategy to influence all the system because the most part of people will be passive and waverers. They will be on fence, so you need to convince every one of them to do your job, to do Chaos Engineering and to give them the right reason, the right emotion, to do that.

So, you need to tell them the right story. I speak about survival instinct, fight, freeze, fly away. So we are there, in the first part of our brain. We need to go in the connective, the concept way to speak about Chaos Engineering. For that, we need to use feelings and emotion. I don't know if there is no scientist in the place. I know this is very simply view of our view of our brain and I apologize but it was enough for me today. So play with emotion. Of course, the first one, fear. Major outage happens some time so fast, and so they cost a lot of. The first thing I did when I arrive in my company it was to put money in front of every critical incident. If you say you have incident of five minutes, for some people it's big, for some other people it's only five minutes, but if you speak about one million incident everyone agree it's a big incident. So let's put money, dollars, euro, in front of your incident.

And be opportunistic. We all have critical incident. In my company we in 2008 30 hours of work our website. I was not there but it was a beginning of a strategy to improve our service, and when I arrive in 2016 we lost a room in our data center, and right after this outage I come to my boss and say, "Hey, maybe it's time to try a new thing." So be opportunistic.

I love this guy from Amazon quite everything about him. Like he said but this one is interesting too, "Everything fails all the time." You cannot avoid incident, you cannot avoid failure. What you can have is to limit the impact of those failures. When you tell them to your CEO, he can understand, because you don't like when there is impact, there is inevitability. If you tell him, "No it's okay, we know that we cannot avoid failure but we can limit the impact." He will understand and it speak to him or her.

To your CTO you can speak about resilience. The way to limit the impact. We use this phrase, this sentence, "Did not even hurt me." every time we have an incident we try to recover quickly from it and to limit the impact. For our head of operation if you speak about sleeping. I was head of operation, and it was hard to sleep sometime because you have incident day after day after day. Not always, but, if you speak about sleeping and to do experiment in production to be able to recover quickly I think he will heard you. I will heard you.

In your c-level you may have some people who are fighter, because a fighter I use a Red Queen hypothesis. It's an evolutionary hypothesis about you need to evolve to survive, you need to divide up survival skills to keep your business running, and Chaos Engineering is a good way to divide up survival skills. You may ask also people who are afraid of change. They don't need news, they need to do the same thing as ever. You can say, "It's all new but, well, not really." I dreamt of it about right before we did disaster recover incident management, crisis management. So it just a new way to do it, to go to the cloud. Don't be afraid.

We are not alone, there is a lot of people and company around the world who do it. That's Netflix, of course, Google, Amazon, a bank like investment. Even a French national company did it, so everyone can do it. Some for the people they want to do new things, to explore new territories. For them, you can tell them it's a beginning. It's beginning to happen mode for the socials ladder, and it's new territory to explore. We see today there is a lot already done, but there is much more to do in the future about this.

So, all of this story you don't have to tell them to every one of you people on the executive committee. I use a lot this tale of the blind man six by nine on an elephant. It's a story about six blind men who come to village, they never meet an elephant and they say, "I want to do, what it is, what is an elephant?" So every blind man touch a part of the elephant, the first one say, "Oh, it's a rope.", "No, it's a wool." "No, it's a trunk." Everyone see a part of it. It's the same with your c-level executive committee. You don't have to tell the whole story, you have to find the right thing that will convince them. For some, will be follow Netflix, Amazon. For other it's innovation. For other it's stability, resilience, defense protection, avoid ordering the next outage. Try to find every part of your story that talk to them.

Maybe you still not convinced. Maybe you think it's all bullshit. Okay. Why not? Just I want to share some numbers. Numbers coming from leadership in organization from Gary Yukl about our influence strategy engage people. You don't want to only convince them to let you do it, you want to convince them to become sponsors. To work with you, to help you do Chaos Engineering and help to improve resilience of your system. The best way to do it is inspirational, making sense, make them dream. You will have 90% of engagement. If you speak about written investment, well, it's rational way, it's 23%. Not very good. Maybe best thing to do is to work about friendship, have a beer with them. You will have around 31, 35%. The way you don't have to use this traits to say, "Oh, if you don't do that we will be dead." Because pressure is three person, and in France we have an expression to do that. We say that, "[French 00:13:57]." We do not suit for pressure, we drink it, because pressure in France is draft beer. So, pressure triples engagement, draft beer 31%. So, let's have a beer together.

Join us in France, we have a lot of meeting, meet up. Maybe you did come once. We hope that everyone here will come once in Paris, share documents, experiment with us. We have a lot to say, it's all in France sorry, but, you can speak English in our meetup, everyone will understand you. That's all for me, I was a bit quick but it's okay, we have much time to lunch. Thank you.

Related

Avoid downtime. Use Gremlin to turn failure into resilience.

Gremlin empowers you to proactively root out failure before it causes downtime. Try Gremlin for free and see how you can harness chaos to build resilient systems.