Skip to main content
Product Manager at Arity
Aug 24, 2017

In this guest post Jenny Wanger, Product Manager at Arity, gives an overview of the most important developer experience KPIs (key performance indicators).

 

I spoke with various Developer Experience product managers about how they measure success–what KPIs they use and why. It became apparent that the responsibilities of each DX team shaped the KPIs they use. There are four potential domains that developer experience teams are responsible for, with different KPIs for each domain.

the four domains
  Get people to the developer portal Convert people on the developer portal to customers Retain developers Operate the API gateway or program
Primary KPIs Increasing quality people landing on the dev portal Conversion rate Retention rate Uptime and latency
Leading Indicators
  • Site analytics
  • Time on site
  • Click-thru rates
  • Bounce rates
  • Usability testing and developer satisfaction
  • Time to first hello world
  • Developer satisfaction
  • Adoption and use of tools
  • Deflected queries
  • Pull requests to open source projects
 

 

Of the four areas, everyone I spoke with was responsible for the middle two sections. Depending on the company, the team also had one of the responsibilities on the ends, but no team I spoke with was responsible for all four domains. While not everyone tracked the same KPIs, I will highlight the ones that I found to be the best fit. When discussing the KPIs below, I define them as primary KPIs and leading indicators. I do this because each domain has one KPI that is the ultimate measure of success. Because the KPI can be hard to measure or slow to gather, many teams use leading indicators as secondary metrics to support their work.

 

Get people to the developer portal

Not every DX team is responsible for marketing to developers, but plenty of them have responsibility for developer evangelism. As Cristiano Betta explained to me, if you're responsible for traffic to your website, then you're really responsible for getting potential customers to the point of sale. Given that, your KPI should be focused on increasing the number of quality visitors that arrive on your site. This requires defining your target audience, figuring out how to target them, and serving the right content at the right time. A good analytics program is critical at this point.

 

Convert people on the developer portal to customers

Once you have people on your site, it's time to convert them and, just as in ecommerce, your main KPI becomes your conversion rate. The KPI is easy to understand but hard to break down, and so leading indicators are especially helpful here. Bandwidth uses usability testing, time to first hello world, and site analytics together as leading indicators. The combination of qualitative and quantitative measures here seems especially helpful.

 

Retain developers

Once you've got your customers, it's time to retain them. As with the conversion piece, all the product managers I spoke with talked about this domain in one way or another. Some phrased it as "supporting the developer community" or "working with customers", but it comes back to retention. The main retention rate is straightforward enough, but leading indicators are less concrete. Possible metrics include adoption and use of tools, an increase in volume of API calls, and pull requests on open source projects. Returning to the ecommerce analogy, it's equivalent to size of shopping cart.

 

Operate the API gateway or program

The least common responsibility of a DX team is actually managing the operations of their developer program. In one case the product manager explicitly told me that this is a DevOps role, and it was not the primary focus of any DX team I spoke to. However, for those who need to operate the program as a part of their responsibilities, standard DevOps metrics apply: uptime, latency, etc. The general consensus was that the more mature the company, the less likely this role would be part of the DX responsibilities.

 

Developer Satisfaction

While each specific area has corresponding KPIs to match, the one piece that touches the whole spectrum is developer satisfaction. Given that it’s an emotional goal, it’s very hard to measure with anything concrete. Some use net promoter score, but that gets fuzzy—it’s hard to separate out satisfaction with your developer experience program versus the company overall. Slack creates a few leading indicators for developer satisfaction by keeping an eye on documentation quality and adoption of tools. They’re also great at the most important part of measuring satisfaction-- listening and talking with their developer community, whether through online tools or events, which allows them to really capture the qualitative side of this metric. Others mentioned deflected questions—questions that get answered through a knowledge base or support flow without requiring a person to intervene to help the developer find resolution.

 

Choosing your KPIs

Defining your KPIs is an opportunity to define the scope of your team's responsibilities. Take the time to connect across your organization to figure out who is taking care of your customers and at what point; once you know your responsibilities, the KPIs should follow.

alt
Information Architecture as the 1st step in your DevPortal journey What information will help you outline what content and functionality should be in your portal? When should you integrate with 3rd party systems? Why is information architecture a vital base for your DevPortal?

 

alt
Contact us for more information on how an Information Architecture workshop can help you to identify and deliver on your developer portal goals.

Product Manager at Arity

I'm a product manager who believes that a user-centered mindset can be applied to any process, and love to bring agile and user-centered practices to unusual spaces like service design. My experience designing digital products and tools, structuring new business models, and facilitating design thinking workshops at some of the world's largest companies has provided me the skill set that can help a company sustainably grow and scale.

Newsletter

Articles on devportals, DX and API docs, event recaps, webinars, and more. Sign up to be up to date with the latest trends and best practices.

 

Subscribe