Making a Super Team

Turning “programmers” into “product engineers”

For the last couple of months we had a DevOps guy in our team. This is still unusual in our organization, most DevOps aren’t assigned to specific teams, they just work in a separate DevOps department that does work for various projects. The approach we took — which I think worked out really well— is that the DevOps guy did not handle all operational issues and deployments. Instead, the main thing he did was build the infrastructure and tools to have developers do this themselves.

Part of everybody’s on-boarding in this team (including mine) was setting up SSH keys and Ansible to run deployments yourself, and becoming familiar with the documentation describing how it all works. The result is that everybody in the team can, and does, deploy new versions. Developers deploy, QAs deploy, even I deploy stuff from time to time.

This has some nice side effects:

1. Because the team doesn’t only write the code, but also operates their own products — the sense of ownership is very high. Other teams may have a “throw it over the fence” culture — _we_ write the code, you keep it running. This team certainly does not.
2. The versatility of everybody in the team is increased. Beside their own expertise, everybody knows at least the basics about infrastructure, deployment and debugging production issues.

This versatility has the added advantage of subconsciously influencing the code developers write. For example, if you know multiple instances of an app will be deployed on different servers for HA purposes, you will handle state differently. Similarly, if you know you will have to debug issues on production, you will take good care of good log messages.

The role of DevOps in the team is to share knowledge, build infrastructure and tools for the team to deploy and operate the services by themselves, and maintain company standards. Company standards could be e.g. standards about security, back-up policies, high-availability requirements, software choice (DB engines to support) etc.

As an aside, this approach has the added benefit of taking a job that’s plagued by exclusively negative connotations (“this broke, that doesn’t work”) to a focus on positive contribution and appreciation by the team, with an aside of heroic savior (“thanks for helping bring system back up, we had no idea how it broke!”).

Product

I think this knowledge and tools idea is very much in line with the Agile idea of pushing more product knowledge to the team as well. Rather than a product owner pushing a low-level “to do list” to specific people in the team: “you have to write method A with these inputs and those outputs,” requirements are shared with the entire team at a higher level, with heavy focus on the Why in the shape of user flows, explaining what the user is trying to achieve. Here too, the product owner is very open to suggestions for tweaks that may make things much cheaper to implement, and prioritization based on estimates.

This has some nice side effects:

1. Having deeper understanding of the product users and requirements allow the developer to make many decisions themselves. In fact, the more the development team knows about the roadmap, the better choices they can make at the code level. For instance, if something they’re going to work on today is going to be the basis for many more features over the next year, they know to spend more time architecting it thoroughly.
2. Product investment is higher, and as a result motivation is higher. We all want to work on stuff we understand and care about. If we’re involved in this creation process and understand why we do what we do, that goes a long way. Especially if we have at some level of influence on what’s developed.
3. Like the DevOps example before, understanding another domain in addition to the domain of programming, adds to the development team’s versatility and as a result, better code.

The role of a product owner (or product manager) is to interact with users, and make decision on where to take the product next. The PO then works with the team to provide them the “tools” to make this vision a reality. Tools in this context include user requirements, usage data and a prioritized product roadmap.

Like other roles, POs are part of a larger company-wide product organization that aligns on product strategy, best practices etc.

Quality

Since the team is involved in product aspects as well as operational aspects, the other obvious next candidate is quality. In a sense this one comes for free — if you are the one to push stuff to production and the one getting the call when stuff breaks, you will implicitly start to care about quality more. You’d assume, anyway.

In many companies, including ours, teams have QAs part of the team. So, what do QAs do? In many companies QA is a fancy word for tester, they:

1. Maintain test plans and perform manual tests to ensure everything works as specified.
2. Run manual regression tests before every release to ensure nothing broke during a development cycle.

But I don’t like this model. Why? Because it’s boring, slow and prone to failure. Who wants to perform the same repetitive task over and over again, again and again? In some teams regression cycles take two weeks, that’s way too slow of a feedback cycle. And are you sure you didn’t miss anything during two weeks of full-time manual testing?

The alternative is automation. So, in many more modern companies, QAs now write automated tests. But when QAs are responsible for writing tests there’s still a split responsibility within the team. It may lead to a mini-“throw it over the fence” culture between developers and QAs. A developer writes some code, and the QA covers it with tests and finds bugs.

It’s healthier if developers also take responsibility of writing their own tests, and the QA focusing on sharing best practices and building infrastructure and tools for developers to use to make this easier.

This has some nice side effects:

1. If developers have to write their own tests, they will automatically write their code in a more testable way. As a result, code quality improves.
2. Because developers will be the ones maintaining test suites, there’s high motivation to find a good balance between unit, integration, functional and user acceptance tests.
3. Because a QA can dedicate most of his or her time to creating infrastructure, the barrier to writing the first test (which is always a challenge) can be lowered significantly, and new ways of writing tests can be experimented with.

So, similar to DevOps — rather than doing the (automated) testing work, it’s the QA’s job to whip the team into shape in terms of quality. The QA builds the infrastructure and tools, and acts as a consultant suggesting additional tests to write, e.g. by participating in code reviews and making suggestions how unit tests or integration tests could be written.

Like for DevOps and product, it still makes sense to have a bigger QA department in the company that sets standards, e.g. regarding environments to be used for testing, how to setup continuous integration, testing frameworks to standardize on, and possibly metrics (e.g. code coverage) to hit.

I believe that the best developers are developers that have the full picture — they get the product, they get the context in which it has to operate, and they understand the importance of quality assurance.

Yes, this a lot to worry about for an individual, or group of individuals. But that’s what modern software development is all about.

In this model, specialized roles like DevOps, PMs and QAs exist to support the development team to deal with this complexity, and making it an as smooth ride as possible.

What else?

In my teams this hasn’t become relevant yet, but perhaps the tools and knowledge, rather than doing the work model could work for other roles too — designers perhaps (focusing on developing base pallet of reusable controls, rather than ad-hoc design), UX, perhaps other roles I don’t even know exist.

What does “programmer” even mean anymore?

Considering all this, the term “programmer” almost sounds derogatory. When working in an environment where you have this unique perspective and responsibility: a combination of product, operations, quality, combined with the traditional writing of the codez — perhaps we should start calling this role differently.

Product engineer?