Introductory thoughts on managing projects

Over the course of my career I have had people ask me which book will give them a quick toe dip into the waters of software project management. I cannot say that I know all books out there of course but I really didn’t have an answer to this. Frederick R Brooks’s Mythical Man-month, Kent Beck’s XP Explained, Mike Cohn’s User Stories Applied and Agile Estimating and Planning, Ken Schwaber’s Agile Project Management, Esther Derby’s Agile Retrospectives all come to mind as seminal readings in and around this topic. I have learnt a lot from these books. They are all must-reads. Each technique, each process, each philosophy, each principle described in these (and other) books is invaluable to making the art of building software work.
What I am setting out to do is to provide a gentle overview of all these topics as an introduction so that readers get a taste of what this is about. I try to explain what I have learnt and how sometimes practice differs from theory.
What is it that makes managing software projects hard? How is it different compared to managing a construction project or managing the manufacturing floor? Why do software projects have such a low success rate measured by schedule, quality and budget?
No two projects are alike and they can never be. What we learn as we grow older building software is a sense of comfort in doing things a certain way backed by the techniques, processes, philosophies and principles that have worked. When we enter a project mid-stream we look at everything that we feel is not right and ask “Why is this done this way?”. Without valid reasons we attempt to guide teams to do things different or do it the way we have seen it work in the past.
That is all Project Management is. Adapting to situations. Building relationships. Understanding Risks and mitigating them. Tweaking the way we do things to meet the demands of that situation. Keeping an open mind by understanding that we need to change our thinking process often.
What I have tried to do is to take a vanilla situation and try to explain how I would do it. I also try to explain why so that if those reasons are not met we know we need to adapt.
Background
Many years ago I worked on a project that had a team of 20 (PMs, BAs and TLs) analyzing and architecting the requirements for the migration of a large piece of software. I was part of the offshore team. The client was somewhere in the midwest. Many things about that project come to mind. The first was the client being a stickler for process. We literally had to have a template for Use Cases that did specified everything from CAPS, COMMAS, DOTS, SPACES defining what was right and what was wrong. The sense of helplessness where you feel you don’t understand the whole picture. The fact that that after 7 months the project was stalled, the client spent many millions on us, and what we had to deliver was a bunch of Word documents. What a waste I thought. Of course we know the tyranny of distance. But knowing what I do now we would have done it differently. Right?
Thankfully Martin Fowler did a conference in Bangalore, I joined Thoughtworks and learned a lot about an iterative approach to Project Management.
One other thing to note. As I mentioned before every single project that I have worked on has been different. I have worked on one migration of an e-commerce platform where we religiously followed story point estimation and velocity and another where we totally ignored it. One project I worked on had everyone in the same room and another where all developers were remote. Some variables you have no control over. Others we do. I have found that there are other aspects of projects that have a bigger impact to the success and failure of projects including building solid relationships, being completely transparent, open and honest to everyone and servant leadership. I once commented to Roy Singham the founder and chairman of Thoughtworks that one of the most difficult traits of a good manager (read that as leader even) is to be able to work with and bring together people who have very extreme, different views from yourself and one another and still succeed. Working with yes-men is completely different from those that are opinionated.