Saturday 23 March 2019

Don't be a hero :: How to build successful software in a skills-deprived world

I used to be a nice guy.

In the early days of my career I did a lot of training, both business and technically-focused. One of the most regular bits of feedback I got from course attendees was that I showed boundless patience with people who really struggle with the material.

I appreciated those comments and it is something I strive for every day.

But lately, I haven't really succeeded.

Lately, and by lately I mean the last couple of years:
  • My temper has gotten shorter and shorter.
  • My overall frustration levels have been increasing steadily (like my weight)
  • I have been sleeping less and less
  • I have gone on fewer holidays
And perhaps most importantly, while I've experienced a lot of "success" in my professional career, my overall job satisfaction has dropped significantly.

These days, I VERY frequently contemplate selling everything I own and going to stay in the mountains for a number of months.

To quote Bilbo Baggins: "I want to see mountains again. Mountains Gandalf"

So what happened?

Well, you can probably describe it as burnout in some sense, but the short answer is: I became a hero, and I got addicted to it. I got addicted to be the linchpin guy that the project depended on. 

But saying I got addicted to being the hero is a radical oversimplification. 

In reality, I would give almost anything to have a normal life. To go cycling on Saturday mornings. To improve my landscape photography. To fix the irrigation in my vegetable garden. 

In reality, I don't want to be the hero on projects. And given my specific role in my company I actually should NOT be getting involved in the nitty-gritty of projects at all.

In reality, I just often found myself in a position that I literally HAVE TO step in and become the hero, else projects fail. And I simply cannot handle failure.

Of course, it isn't all bad, and I have had many successes over the years. This article explores my journey, looking at:
  • Some of the things that cause projects to need a hero
  • What you can do to prevent it
  • Why you should focus all your energy on not becoming the hero

A teenie bit of context might be useful here: 

I play a leading role in the bespoke software development area of a large South African IT company. Which means we are often involved in projects where there isn't a pre-baked solution that can be purchased off the shelf.

Most of our projects start off on day 1 being under pressure, because:
  • It took the customer ages to get through the business case and get budget allocated
  • The procurement cycle took ages
  • And then more time was spent getting the project team assembled and ready to rock and roll
  • And of course, requirements are never defined or well thought through at the start.

But despite all of that, some big-shot exec has already made a delivery date commitment to the board of directors in order to get the money for the project.

So at the best of times, most projects have to start by wriggling themselves out of a hole dug by business people.

Understanding the different role-players in projects

Of course it differs from project to project, but I tend to see the following breakdown on most of the projects we get involved with (note: most projects tend to be a joint effort with the customer and sometimes even including other vendors)

  • Heros - People who REALLY know their stuff. Passionate about delivering the project - 15%
  • Divas - Passionate technologists who think they are too good to be on the project -  5%
  • Talkers - People who add little value except for the fact that they're old or seem to talk a lot in meetings - 10%
  • Workers - Worker bees who do one thing well and put in a solid 6-8 hours of work - 40%
  • Strugglers - People who want to work. And think they do, but they never seem to deliver anything - 20%
  • Dead wood. People who are literally just on the project to fill a seat - 10%
So based on these (admittedly very anecdotal) numbers, it turns out that most projects are actually delivered by about 50% of the team. 

But remember, when you originally do proposals and quotes, you can't double your estimations simply because you expect a bunch of team members to not really pull their weight. If you're in my business - the business of making a profit out of software development - then your Go To strategy must be to change the layout of project teams to something like the chart on the left.

It's probably obvious that we've killed off the Divas, the Talkers, the Strugglers and the Dead Wood, but what is also interesting is that:

We didn't remove Heros entirely - Every project, no matter how well planned will need some instance of heroic programming at some point. Let's not live in a dream world.

Building your pack

The best analogy that I can come up with for a project team is a pack of sled dogs. I'm not sure if it's just me, but I loved watching the Discovery series on the Iditarod, the epic dog-sled race in Alaska. In typical Discovery channel fashion they constantly tried to manufacture intrigue and surprising plot twists. But what I loved about the show was not so much the intrigue, but rather learning more about how these racers assembled their packs of dogs prior to the race and how they cared for them during the race.

A key measure of success for a pack of dogs is that they work together. If one of dogs has an issue and needs to slow down, there are two options: 
  • The whole pack slows down until the slow dog recovers
  • You remove the slow dog from the pack
But as lead developers, people with built-in Type-A personalities, that isn't often our instinct. My natural response, if I see members of my team struggling, is to run harder myself - Thinking that by running harder, I will balance out the slow team member(s).

Although this approach can sometimes counteract a performance problem with one team member, it has a significant unintended consequence - It pulls the WHOLE TEAM off balance. And once the team loses balance it is very difficult to get your delivery velocity back.

Depending on the project, there are two possible outcomes:
  • Small projects become a one-man show with other project team members feeling useless and just sitting on the side-lines
  • Large projects fail outright, because it is impossible for one person to carry a 10 or 20 person project.
So our goal is to go from an inefficient project make-up, like the one on the left, to an efficient make-up, like the one on the right.

And there are many tricks of the trade to achieve that. This article will focus on one specific side of the problem an how to fix it: How can we CONVERT THE STRUGGLERS INTO WORKERS?

We can probably debate for hours about how we should fix the education system in South Africa, or how we should improve recruitment to ensure strugglers don't get onto our projects, or why we should move to San Francisco to give ourselves the opportunity to work with the best teams of our industry.

Those are interesting discussions, but it is not the discussion I'm having today.

My question is: How do we convert the STRUGGLERS who are already on our projects into WORKERS.

And there really is only one answer: Teach them.

Accepting the Kruger Dunning Effect

So let's talk about South Africa for a bit. In South Africa we have people from many different languages, many different cultures, males, females, different religions, different races, different economic statuses.

And yet, there's one thing that brings us together as a Nation: We all believe we are ABOVE AVERAGE DRIVERS.

And that's a key observation I've made with strugglers on software projects. They don't consider themselves strugglers. They actually think they are experienced software developers.

Most CVs here in South Africa include a skills-matrix of some kind where the candidate lists their skills along with a rating from 1 to 5. In every single CV that I see from a young software developer, they drastically overestimate their own abilities. I've observed this over many years.

When I started to investigate this problem, I came across a phenomenon called the Kruger Dunning effect. But to really appreciate the Dunning Kruger effect, you need to hear the story that inspired it.

In 1995, a 45 year old guy walked into a Pittsburg bank and robbed it at gunpoint. Later that same day he robbed another bank. What was most interesting about this bank robbery is that the perpetrator didn't wear any mask or use any method to conceal his identity.

As could be expected, he was quickly identified and arrested. When the cops entered his house and placed him under arrest, the first words out of his mouth was: "But I wore the juice"
You see, McArthur isn't very smart. He had heard about the fact that you could use lemon juice to write in a type of "invisible ink" which is undetectable to the naked eye. He then reasoned that if he rubs lemon juice over his face, people in the bank, and the security cameras would be unable to see his face. Master plan!

What is interesting about this story is not just the complete ignorance of the perpetrator (which, let's be honest, is pretty interesting), but rather how utterly convinced he was that the lemon juice plan would work.

Psychology researchers Justin Kruger and David Dunning were inspired by this story to explore the idea that people who have low levels of competence in a specific field seems to greatly overestimate their own abilities.

The Kruger-Dunning Effect describes this phenomenon where someone with a low level of competence in a subject, let's say Programming, also lacks the knowledge to accurately assess their own competence. As a result, they greatly overestimate their own capability.

You can summarise this as: A little knowledge is a dangerous thing.

The graph below depicts this phenomenon by showing how someone, once they start to get experience in a subject, slowly begins to realise how little they really know.

This is what happens to a Struggling junior developer if left to his/her own devices. They literally don't know that they aren't adding any value to the project. They actually think they are experts in their field.

So let me give a practical example of this effect from my day to day life: On a recent project, we had a junior developer who was very clearly VERY Afrikaans. For some reason, however, he was under the impression that his English grammar was impeccable and that it was his duty to correct people at every opportunity. It was so bad that I once had to privately address the issue with him, when he "corrected" the client's product owner in a meeting.

Here's one of the classics he added into JIRA as a defect:

Yes, perhaps a whimsical example, but it applies to most aspects of young developers who fall into the "Struggler" category. It's not just that they lack the required competence, they also lack the knowledge to realise their lack of competence.

Your goal as a senior developer should not be to criticise their lack of competence, but rather guide them along the path, helping them realise what they don't know and then encourage them to learn it. We'll touch on some approaches just now.

The Curse of Knowledge

But first I want to address another cognitive bias that makes it difficult for experienced developers to teach younger ones. It's called, the "Expert Paradox", or sometimes the "Curse of Knowledge"

As a senior technologist, many things may seem easy or self-evident to you. This is function of your competence as well as years of experience. This knowledge and experience makes it difficult for you to actually empathise with the position of the inexperienced person. Your knowledge actually becomes an impediment to teaching.

I generally like to use a puzzle analogy: Think of your whole IT carreer as building a puzzle. It's the puzzle of knowledge. Each piece of the puzzle is a new bit of isolated knowledge - It only becomes valuable when you can look at it in context. As an experienced practitioner, your puzzle is already well established, so when you find a new piece, you have a much better overall picture and you can more easily understand where this fits in. Even if you find a difficult piece of blue sky or ocean water, you can more easily find where it belongs because you can compare it with the other patterns you've already constructed.

To a young and inexperienced person, they have barely begun assembling their puzzle. Each piece is a brand new experience and is often very difficult to see in context.

Practical mentoring

So how, in a practical sense, do you mentor inexperienced developers? I've seen ways that work and ways that definitely do not work.

Here's one way that absolutely does not work: You pick a small, apparently easy bit of work and give it to the new guy to implement. And then you wait. This fails for a number of reasons:

  • You wait too long
  • By the time you receive something back the learning opportunity is lost
  • You probably implemented it yourself, so the new person loses conffidence

I learnt a better way from one of my colleagues, William Gadney. I'm not sure if it is his own method or whether he read about it somewhere, but the process is both deceptively simple and yet deceptively difficult to execute.

But it is almost guaranteed to work. I refer to it as the 3-Stage Journeyman approach and it looks like this:

The idea is simple:

  • First, you find a piece of functionality that you will need to implement at various points in your project. A good example would be if you had to build a number of different REST services.
  • Stage 1: When building the first one, the inexperienced developer sits next to you. While coding you "Think out loud", explaining what you're doing as you proceed.
  • If it seems that the new person understands, you move on to stage 2, else you repeat stage 1 
  • Stage 2: At this point you switch seats. The inexperienced developer does the coding, but the mentor is right there, guiding, correcting and helping the whole time.
  • If the learner gets the hang of things, you move onto stage 3, else you can repeat stage 2, or even go back to stage 1.
  • Stage 3: At this point, the inexperienced developer goes away and writes the code by themselves. Followed by a detailed code review.
  • If all went well, the rookie has become a journeyman in this particular skill, and you can pick the next one.

Help the team see the bigger picture 

The next bit of advice I use to help inexperienced developers escape the Kruger-Dunning trap is by constantly assisting them with seeing the bigger picture. Effectively, I overshare information about the project so that they are always 100% aware that there is more to the project than the bit they are involved with.

This has two consequences:

  • They are frequently reminded that there is stuff they don't know, but
  • They have full access to learn that stuff - It isn't hidden from them

My strategy includes the following tactics:

Tip 1: Err on the side of a mono-repo
A multi-repo scenario might have many benefits, but it has one critical flaw: It entrenches a fragmented view of the solution. Even the most astute learner will struggle to fully understand how a solution fits together if he doesn't even know that he is looking at a partial picture.

A mono repo helps greatly with this. In fact, I prefer using a single repo that includes Documentation, BDD Testing code and Load testing projects all into one Git repo. If you do prefer a multi-repo scenario, then ensure that the repos are clearly documented and referenced in a single Master project repo.

Tip 2: Get the documentation into the repo
Add the project documentation into the single repo (or the project master repo if you follow a multi-repo approach). I use Sphinx, which makes this a breeze. You document right there in your IDE making it quick and intuitive. And you can generate and publish the docs as part of your CI/CD pipeline which means that the docs and the code are always in sync.

If your company uses Confluence, you don't have that option, so you just need to work hard to make the documentation accessible.

Irrespective of which approach you use, you will always need to work hard to make sure developers access the documentation. Trust me, it is a never-ending battle. 

Tip 3: Make the architecture visible
Architects tend to create their diagrams in isolated CASE tools that nobody else has access to. This is one of my most fundamental critiques against the discipline of Architecture, and is a problem that I have been fighting against my entire life. 

My mantra is simple: "An invisible architecture is just as bad as no architecture." 

All you need to do is to add the architecture diagrams into the project documentation. I use the 4+1 Architecture framework on almost all projects, and it provides a great skeleton for the project documentation as well. 

By following this approach you make the architecture accessible to the masses.

Tip 4: Create an Architecture Overview Diagram
Sometimes people get lost in the details of UML and they forget that architecture is about communication. The most successful approach I've followed is to create a bastardized UML diagram called the AOD - An Architecture Overview Diagram. It's like a component diagram, but it also includes Actors.

The result is a diagram that can depict the main Solution Building Blocks (as defined by TOGAF) as well as the context of the solution in a single diagram.

The goal is to create a single diagram, at a high-enough-level that it can depict your whole project on an A4 page, but still shows the salient aspects of the entire solution. Creating a proper AOD is an artform - it forces you to think hard about which details to include and which to exclude. It's almost like the battle Picasso went through while deciding which details to include in Guernica.

But if you get it right, the result is amazing and the resulting diagram serves as a reference point for everyone for years to come.

And, of course, the inexperienced developers can use it to assemble their own puzzles (Almost like looking at the picture on the box of the puzzle).

I have a few other tips as well, that have helped me in the past, I'll briefly list them here:
  • Add a GETTING STARTED section in the docs. Include reference links and introductory notes for each of the technologies used on the project.
  • Do VERY FREQUENT code reviews. Not just at the end of a piece of work, but while they are busy.
  • Establish a formal avenue for knowledge sharing. In our company we call it Lunch Time Learning - Every fortnight we have a team learning session on a Wednesday during lunch.
  • Introduce project team sharing sessions. We do it after retro each sprint, where one team member explains a piece of the project to others in detail - A level of detail that doesn't make sense when the product owner is in the review session

The Benefits

All of these tips and tricks take time and effort, so it is important to be patient. If you rush it, for example, by skipping stage 2 of the 3-stage journeyman process, you are dooming yourself to failure.

Also, it is important to be aware of the benefits that await you. So let me help by summarising the medium term and long term benefits.

In the medium term you will create a situation where you DON'T ALWAYS HAVE TO BE THERE.

  • You can afford to take a week off and go hike in the mountains where there is no cellphone signal. 
  • You don't have to be online with each deployment
  • You don't have to prepare fully for each sprint-review
Also, you create an overall reduction of the Bus Factor of your project - So if you leave (by choice or otherwise), the project will survive without you.

But the long term benefits are the most rewarding, and as someone who's been in this business for many years, I've now experienced it a few times. It's when one of the junior developers you had under your wing becomes a master in their own right.

It might take years, but the day will come when you can hand over an entire project to someone and know ABSOLUTELY that it will be a success and that you don't have to get involved at all.

That's the greatest feeling in our business.

And of course, if you have aspirations to begin that Tech Start-up you've always dreamt of, you are finally ready to get it started as well, because you now have people you can trust to get the job done.



Blog activity mostly moving to

Just a quick note that, after realising that I should focus my attention fully on the skills problem, I am focusing all blogging activity at...