This was originally proposed by Eric Brewer back in 1998, and states that any distributed system can only achieve two of the following three properties:

• Consistency (C) – this means the system will always return the most up-to-date information e.g. any read of a record after it has been updated, must return the updated information
• Availability (A) – this means the system is available so that a non-failing node will return a response within a reasonable amount of time
• Partition Tolerance (P) – this means the service can survive a communication breakage that results in some nodes in a cluster being unable to communicate with other nodes. This is known as split brain

One of the fallacies of distributed computing is that networks are reliable – they are not. As such, any distributed system must be designed to handle network partitions. This means when designing a system, we will need a trade off between consistency and availability.

In the example above, data is distributed across 4 nodes. To be fully consistent, when Client A updates a record, Client B must be able to read that record, even when the read is made from a separate node. The problem occurs when there is a network partition, which prevents the updates being synchronized or replicated to the other data nodes.

In its simplistic form, there are two basic options. A system that mandates consistency will fail when Client B attempts to access the record. A system that mandates availability will succeed but return stale data when Client B attempts to access the record.

It is crucial to understand that this is not an either/or binary decision. Instead, there are patterns which allow a sliding scale of availability or consistency. A system as a whole will also consist of many services, and each of these may choose a different preference. This is based on factors such as risk and probability. In my current role, the view is to be consistent when we can, but when not possible we will choose availability. This is because the probability of multiple transactions taking place almost simultaneously on a specific record are extremely rare. When choosing to use local data to improve availability, a staleness indicator will be included. This would allow a transaction to be rejected if the perceived risk was high enough if transacting against stale data.

1) Ability to Innovate
2) Command and Control versus Mission Command
3) Continuous Improvement
4) Value Stream Mapping
5) Role and Importance of IT
6) IT Best Practices
7) Multiple Strategies

1) Ability to Innovate
The long term value of an enterprise is related to its ability to increase the value it provides to customers – and to create new customers – through innovation. We are in a rapidly changing world, and no business can afford to stand still, as no business model is indefinitely sustainable. It was fascinating to read the average expectancy of a Fortune 500 company has declined from around 75 years half a century ago, to less than 15 years today. Professor Richard Foster of Yale University even estimates that by 2020, more than three-quarters of the S&P 500 will be companies that we have not heard of yet.

The example was given of Kodak who invented the digital camera in 1975. However, this was seen as competing against their existing business model optimised around developing photographs. As a result of not recognising future trends, and the unwillingness to cannibalise (or disrupt) their existing business model, they ended up filing for bankruptcy in 2012. Other examples include Blockbuster vs Netflix, HMV and Tower Records vs iTunes, YouTube and Spotify. This all adds to the evidence that you need to constantly scan the horizon, and restlessly reinvent yourself.

2) Command Control vs Mission Command
This struck a chord after spending many years working for a global services organisation.

Command and Control is the idea that people in charge make the plans and the people on the ground execute them. This is thought to be modelled on how the military operates, but is outdated. My own experience was working for an organisation in which any change needed to be escalated to get approvals at a continually higher level. What seemed worse, was that an American-led organisation, the strategies and setup were based on the US market, which in many cases was different than in the UK. There were plenty of people who in my mind “got it”, and had some great ideas. However, they were never entrusted to act on it, without spending months escalating to the core central team.

The organisation itself was so large, that lots of formal processes were put in place. However, this placed focus on pre-defined ways of working. It meant that innovation was stifled, which was a real shame. I can fully understand the rationale behind it, but the primary measurement for employees was utilisation. This also prevented an environment where learning new skills was encouraged, as all focus was on utilisation, and most quarters I was there (which was many) training budgets were slashed and courses cancelled.

Counter to Command and Control is Mission Command. With this approach, the general direction or intent of the mission is communicated from the centre, but those below have the authority to make decisions as situations change without waiting for approval through the chain of command. This also means the centre can specify the outcome they are looking to achieve, whilst allowing others to derive their own plans to achieve this based on their own knowledge of the local market.

3) Continuous Improvement
A number of methods where discussed that all had similar characteristics:
Eric Ries Build-Measure-Learn loop
Colonel John Boyd created the OODA loop (Observe, Orient, Decide and Act)
Deming Cycle Plan-Do-Check-Act

The key is to run cheap and quick experiments and building in feedback loops. Much of this thinking is already incorporated in the Government Service Design Manual with the service design phases moving from Discovery to Alpha, Beta and Live.

4) Value Stream Mapping
This was another aspect of the book that I could relate too. Many features take a small to develop and test, but take a long time to deliver end-to-end as a result of waiting time. The best way to understand where problems start is by performing an activity called value stream mapping. A value stream is the flow of work from a customer request through to the fulfilment of that request. Each value stream crosses many functions or lines of business within an organisation.

I have spent a long time working on various client sites, many of which heavily weighed down by red tape and process. There are often governance boards of one variety that sit infrequently, and rely on submissions being well in advance. The cost of missing one of these submissions can add weeks in and of itself. Then you read about companies such as Amazon, Netflix and Etsy where teams in many cases do not need to raise tickets and have changes reviewed by an advisory board to get them deployed to production.

5) Role and Importance of IT
The 2015 State of DevOps Report found that high performing IT organisations experience 60x fewer failures and recovery from failure 168x faster than their lower-performing peers. They also deploy 30x more frequently with 200x shorter lead times. An MIT Sloan Management Review report found that companies whose engineering teams did a good job of delivering their work on schedule and simplifying their systems achieved better results with much lower code bases, even if their IT investments aren’t aligned with business priorities.

6) IT Best Practices
Decouple deployment and release: Deployment is the process of installing a piece of software, Release is the process of making the new feature available to the end user. Release is a purely business decision. There are different techniques available for this. For example, using blue-green deployments which requires two separate production environments. There is also a concept called ‘dark launching’. Developers can protect new features using ‘feature flags’ so they are only accessible to a particular set of users. This also allows for different flavours of a feature to be tested to see which is more popular, a technique known as A/B testing. Companies such as Amazon and Microsoft typically run hundreds of experiments in production at any one time and test every new feature using this method rolling it out. The same goes for Etsy, Bing, Facebook and Netflix
Perform Real Failure Injection Exercises – Amazon, Google and Facebook inject faults into their production systems on a regular basis to test their disaster recovery processes. Netflix run a set of services known as the Simian Army, led by Chaos Monkey.
Automate Everything – make sure all builds and deployments are automated to reduce risk of releases
Reduce Batch Sizes

7) Multiple Strategies
It is important for most organisations to have a balanced portfolio. It is also clear that exploring new opportunities and exploiting existing ones are fundamentally different strategies. The ‘Three Horizon’ model can be used to look at these different horizons.

It had been a strange few weeks leading up to the race. With a house move, a change of job and a heavily pregnant wife, the thought of lugging myself around 140.6 miles really didn’t appeal. Still, the 400 notes spent on entering the race, and the fact I was doing it with my best mate, was motivation enough to head over to Tenby.

The alarm went off shortly after 4am. There was no spring in the step or excitement, just a sense of trepidation, and that this was going to be a long tough day. A quick bowl of porridge, and it was time to head into Tenby, park up, and make sure the bike was all set up in transition. This turned out to be a rush job, as transition was about to close. Still, I didn’t need long, and then myself and Jeremy started the long slow march down to North Beach. The good news was that this year was a rolling start. There were over 2000 people entered, and the view from the organisers was that the mass start would have been intimidating for a lot of people nervous about a sea swim. Personally, I could have lived with either approach, but it was going to make it easier.

I put myself into the pen for the 1:25 – 1:35 swimmers. Before long, the Welsh National Anthem boomed out of the loudspeakers. This was the first of many special moments on the day. Finally, shortly after 7am, I was walking into the waves, and the worst part of the day was about to start. The swim around the first buoy was tough. Although the sea looked relatively calm, there was a large swell, and it was a completely new experience for me. I was struggling with my pool stroke, and kept on taking on sea water. I battled on, and was surprised when I finished the first lap in around 43 mins, as it felt much slower.

Onto the second lap, and the swell had got worse. As I got close to the first buoy, some water I took on had the distinct taste and smell of boat fuel. I stopped and hung onto a kayak. Initially I started wretching and then was violently sick a number of times. My throat was sore, my head ringing, and I just wanted out. Still, I decided to just try and carry on. After 1hr 27 mins, I finally exited the swim. I felt beaten up by the sea and pretty low. I took my time back to transition, and wondered long this day was going to take.

It was a relief to get onto the bike and start the second discipline. I still didn’t feel good, and had to make another impromptu sick stop. My power felt low, my throat was still sore, but the good news was I was moving, and it was at a reasonable pace. There are not too many things I remember about the bike. The weather was surprisingly good which was a relief. The two hilly laps were hilly, but manageable. The support was great.

The most memorable part is the climb up St Brides Hill coming out of Saundersfoot. There is a mass of spectators, many in fancy dress, most with cow bells and other instruments. First time up the crowd parted just enough to let me up, and it is the closest I will ever get to experiencing the Tour de France. It really was incredible and is something I will always remember. I started to have a few bike problems over the last 30km when I struggled shifting between rings, but at least it took my mind off the race.

Finally, it was time to get off the bike, and get ready for the marathon. I’d had a tough experience at the Outlaw, where I started at 4:50 min/km pace, and kept slowing down until I was shuffling at 6:20 min/km pace at the end. My quads had been on fire and that was on a flat course. My plan for IM Wales was just to take it gently and try and enjoy it. I set off almost a minute a kilometre slower than at Outlaw, but didn’t mind and didn’t feel the need to pick it up. I was only expecting to run a 4.5 hour marathon, so was happy to be running almost an hour for each 10km. The run course was better supported than the bike. It was an unbelievable experience running down tiny thronged streets in the town centre being cheered all the way. I saw a couple of friends and stopped to chat to them, and I walked 20-30 seconds at feed stations. It started to get slightly tougher but the pace stayed the same. As I came round to complete my third lap, I had the most unexpected surprise of seeing Niamh, Cian and my folks shouting me on. This was by far the highlight of the day, and I couldn’t believe they’d made the journey. I met up with Jeremy on my final lap and we ran it together which went in a flash.

And that was that. I crossed the finish line to have completed my first official Ironman race.

In terms of my own health, I’ve picked up a few illnesses and niggles along the way. I should have listened to my body and taken more rest, but was conscious of IM Wales looming on the horizon. It definitely took a couple of weeks before I could run with little pain. However, after that I started to try and pick the pace up on the runs and have picked up a knee injury which makes it sore to run.

The first two weeks after Outlaw I was living off the buzz of having achieved an amazing life goal. However, as the time drifts by, and the elation fades, the overwhelming feeling of tiredness has kicked in. This was always going to be the case, but I wish I’d been more prepared.

As it is, my best mate Jeremy came over this weekend. We ended up doing a 140km bike ride on the Saturday, and a 20km jog on the Sunday morning, with a night off on the beer in between. I felt a bit stronger on the bike, and other than the bad knee was comfortable on the run. It’s made my mind up that it’s all systems go for IM Wales in 2 weeks. It’s not going to be pretty. It’s not going to be enjoyable. There’s a very real chance I could do my knee some serious damage. But when have I let common sense get in the way ….

Now, almost 16 years since I first started work, I’m about to wave goodbye to the consultancy lifestyle and become a civil servant at the DVLA. On the face of it, that may seem like a crazy decision. I was doing well for myself at IBM. I’d received the highest grade given to only a small percentage of people in 3 of the last 4 years. I’d been elected to the Technical Consultancy Group (TCG), the top cross-brand technical community in IBM UK and Ireland, and I was part of the Public Sector CTO team.

The perception of the civil service is one of faceless individuals in suits and bowler hats. Worse still, for someone like myself with a huge interest in technology, public sector is associated with big old archaic computer systems. The truth though, is that the times are changing.

DVLA are on the cusp of a major transformation. Government is still faced with a large deficit meaning departments need to find ways to save money. It’s well known that legacy technology is costly and inflexible, with vast sums of money spent just to stand still. As a result, there is a fantastic opportunity to completely re-engineer the existing outdated processes whilst focusing on user need. The exemplars delivered at DVLA, for all their shortcomings, demonstrated that the agency were capable of delivering new capabilities in the cloud, built on open standards and open source, embracing DevOps, utilising multi-suppliers and delivered using an agile methodology. I firmly believe this is the approach all industries will adopt.

At the same time, DVLA are actively involved in building a local digital community through their partnership with TechHub, and are forging closer alliances with universities. More importantly, it allows me to get more hands-on with technology.

Personally, I am exciting about the next couple of years. It’s up to me to make the most of it, and that’s something I intend to do.

Day Before

After struggling with illness the week before, I had a great nights sleep on the Friday night and woke up feeling half human. I packed the car, said my temporary goodbyes, and headed down to Nottingham to register and rack the bike.

After all the hassle with the logistics around the Staffs 70.3, the organisation and layout for the Outlaw were top notch. Although the nerves had set in, my legs were finally feeling better, and small amount of confidence returned. I went out for a meal with Ben, James and Joe who were all racing, and then back to the hotel for the expected fitful sleep of a condemned man.

Pre-Race

The alarm went off at 03.10 and I felt half-human considering the time. We all went down to the lobby to eat, and then made our way to the venue.

I checked the bike, and decided to borrow a track pump. Unfortunately, the valve on the rear wheel didn’t stick out much from the deep rim, and I only succeeded in letting all the air out. In a state of panic, I also wasted a CO2 canister, until finally I was able to borrow another track pump that somehow attached to the valve and could reinflate the tyre. Phew … a major panic over.

Finally, I went back to the change tent and got into the wetsuit. This was the point of no return and I was committed (though looking a little apprehensive)

The Swim

I set myself up at the front of pen 3 (target time 80 – 100 minutes). When the horn went off, I spent the first few seconds trying to start my stopwatch, which for some reason wasn’t working. I quickly decided to give up and just swim. The first 10-15 minutes were the most bumpy I had experienced in any triathlon. I got punched twice, and had my goggles kicked off, none of which I’d experienced before. The crucial thing was that I just kept plodding on and didn’t let it affect me.

It seemed to take an age to get up to the turn around point. I kept encouraging myself to keep going, and telling myself that this was it. Just reaching the top of the lake was a big psychological boost, and the return leg seemed quicker. I was definitely overtaking a number of people in the last quarter.

I came out of the water with an official split of 1:16:31. I couldn’t believe it. I was expecting to swim around 1:25, though I knew the swimming had improved out of all recognition the past two months. I also had no cramps, swam front crawl the entire way around, and felt good.

Transition

Normally, I wouldn’t include anything about transition, but this was the second race in a row I messed up. I grabbed my transition bag and got changed in the tent. I ran out of the tent to my bike, but as I got there, I realised that I had left my number behind. I went back to the tent and searched around, but with no luck. In a panic I spoke to a marshall, who pointed me to a race official. Basically, what had happened is I had left my race number in the transition bag, which was now in the back of a van being sorted. The race official was great. He said he would let me go out on the bike, and would look to retrieve my number and put it in my run bag. I’d wasted quite a few minutes, but at least I was good to go.

The Bike

Ultimately, the bike was uneventful. I settled into a steady pace, and just concentrated on ticking off the kilometers. I was so worried about blowing up at some point, and I wanted to keep something in reserve for the run, that I rode well within myself. I focused on eating and drinking and trying to make sure I kept hydrated. The rain held off till towards the end of the bike, but then the heavens opened. It really was a miserable last hour making my way back, but the thought of the bike being over kept me going. It’s only in hindsight that I realised I had no saddle soreness at all on the whole ride. I think the TT bike puts me in a more forward position that just suits me better. Saddle soreness was something I was really worried about, but it never crossed my mind on the day.

In the end, I came into T2 with an official split of 5:42:07. This was a decent solid split without being spectacular.

The Run

For someone who had previously run 9 marathons, it seemed strange that the run was the part I was most worried about. I had done some decent long runs in training, but none of them straight off a long bike. I guess this was always going to be the real journey into the unknown.

I set off at just under 5 min/km pace and felt comfortable. I knew this almost certainly wouldn’t be sustainable, but I figured it was best to get some miles in the bank and worry about what would happen later. I took the strategy of walking each feed station (approx every 1.5 miles) as a way of getting a short recovery.

The first 5km went quickly in sub-25 minutes, but the pace was already starting to slow. I didn’t mind, and I definitely didn’t want to force the pace. The first 10km went in around 51 minutes which I was delighted with, and I ended up going through halfway in around 1 hours 50 minutes, though the pace was already around 5.30 min/km. This kept going up until by 32km I was pushing 6 min/km. My quads were on fire at this stage, and it was a case of just surviving. It was also somewhere between halfway and 32km that I started to think about finish times. I knew I was comfortably on track for sub 11.30, but I couldn’t afford to blow up.

I started walking slightly longer at the feed stations, and the run which had turned to a jog was now a shuffle. It was a case of doing whatever I could to keep moving. I had two stops with stomach issues, but this wasn’t getting any worse. By the time I started the final lap of the lake I was telling myself to just enjoy this. I didn’t care about the pain at this stage. Somehow, I managed to pick the pace up slightly over the last couple of km’s by not stopping at the feed stations, and before I knew it, I was coming down into the finisher’s chute. Niamh passed Cian to me, who decided he didn’t want to run down the red carpet. Still, I picked him up and held him close, and doing my best to keep emotions in check, I stumbled through the tape with the announcer confirming over the tannoy that … “Matthew Lewis, you are an Outlaw!”

My official run split was 3:55:56, a time I thought I could only dream of. My official finish time was 11:11:11. For someone who’s birthday is 11.11 there is something weird and magical in that.

It started on the Saturday night not long after my last blog post. I had a thumping headache and felt unwell, but I just put it down to tiredness. Instead, at around 1am, I woke up and rushed to the bathroom. There followed over 24 hours of some V&D bug that just completely drained me. Even on Tuesday, I was still in no condition to complete an ironman. I tried a swim on the Wednesday evening which saw me cramping – probably a sign of dehydration from the last few days. I managed a jog on the Thu evening which felt alright, but was at least 5bpm higher heart rate than it should have been. I did a very short bike yesterday, and the legs still felt heavy and achy.

On top of the that, Niamh has also been struck down by illness. She had the bug I got the week before, but came down with what may have been food poisoning during the week and was crippled for 24 hours but has started to recover.

And as if all that was not enough, I got a phone call on Monday saying I had till the end of the week to decide whether I wanted to stay at my current employer and TUPE across to the client I am working on. I knew I had to make a decision but expected this to be a few weeks off yet.

With all the above, the stress levels have been through the roof. I have spent many hours walking around with a knot in my stomach and feeling sick with worry and nerves about some of the big life decisions to be made. As you’d expect, none of this has helped with getting lots of sleep and coming into the race feeling well rested and full of energy.

I guess second thoughts are what happen as you get close to major events. I wish it was a week later to allow me chance to recover, but realistically I am definitely fit enough. It’s now just a case of whether I am over the illness and my body doesn’t just shut down. It couldn’t have gone much worse this week, but I can’t change that now. Hopefully, by the next blog post I will be an Outlaw and be proud of my achievement, whatever the actual result on the day

Anyway, that was all exciting but deep down it felt like the wrong approach and too much custom coding involved. Looking around, there are a number of options that utilise configuration options in Dropwizard, or existing libraries.

The initial simple challenge was to show how you can expose performance metrics from Dropwizard and I chose Graphite as a monitoring tool to collect the data and render graphs on demand.

The first challenge was how to install Graphite locally on Mac OS. Graphite has a whole load of dependencies e.g. Graphite Web, Carbon, Whisper, Ceres. This was going to be a pain, but luckily there is a handy Docker image for Graphite and Statsd. Firing up Boot2Docker, I had a running instance of Graphite in no time at all.

Configuring Dropwizard to output metrics to Graphite was just as simple. All it took was:
1) Adding the following config to the yaml file:

metrics:
reporters:
- type: graphite
host: localhost
port: 8080
prefix:

2) Adding the @Timed annotation the relevant resources
3) Adding the ‘dropwizard-metrics-graphite’ dependency to the maven POM file

The training has gone really well lately. I have focused on the swim for the past few weeks. This has meant trying to swim 3 times a week. This has involved two “swim for tri” sessions at the National Pool, and a lake swim to get used to the wetsuit. I am still very slow, but I am definitely swimming more front crawl and it is feeling easier. I did my final lake swim a couple of days ago. I wanted to swim 3.6km but I ended up stopping after 3km. I could feel my neck still sore from the rubbing, and I didn’t want to make it any worse. The great news was that I swam front crawl the entire way around, and still felt strong at the end. I am the most confident I have been of getting through the swim, which was always going to be my worst discipline.

Without having the chance to commute to work on the bike lately, my time in the saddle has been going down. I am starting to get more comfortable on the TT bike, and I love the extra speed. We did a recce of the Outlaw route last weekend, and it’s a perfect course for a TT bike, with plenty of time down on the bars. My main concern is around nutrition and pushing it too hard early on and suffering as a result. I guess both of these are in my control. I practiced today trying to eat more natural food rather than gels. This means I will be using a cycling jersey as it has 3 pockets rather than the 1 on my triathlon top. Pushing too hard will be harder to stop. The key will be to hold back on any drags or when the wind is against.

The run is the one I expect will hurt the most. I know holding back 5-10 minutes on the bike could make all the difference. There is nothing I can do now, so it will be a case of getting through the early miles and just trying to hang on.

When I think about the race I get nervous, but I am also so excited at the prospect of it being over and having achieved a major goal on a bucket list. It’s going to be mentally incredibly hard, and I know there will be plenty of low points on the course when I will just want to quit and I have to push through.

I have my fingers crossed that my next blog post will be one where I have achieved a massive goal!!