James Noes - Optimizing Dev Portals with Analytics and Feedback

This talk was presented at API The Docs Virtual 2023 Feedback, Metrics and Analytics event series on 22 February. We are glad to present the video recording, slide deck, talk summary, and the panel discussion below. Enjoy!

Visit the talk listing page for an overview of all presentations!

James Noes

Product Manager at Kroger

“To optimize a product with analytics and feedback, we must follow practices that emphasize the importance of analytics and feedback.”

Opportunities & prioritization

Building a Dev Portal involves bringing many different things together (accounts, documentation, RBAC, catalog, support, analytics, etc.) to create a seamless experience for the users.

This is an opportunity-rich base where prioritization is needed: create a roadmap using the available data and the needs of the business as a basis.

• What are the needs now? What to include as part of the MVP Dev Portal?
• What are the things that are needed next?

In the stage of prioritization, it is easy to develop bad practices and start witnessing the side effects of having many opportunities.

Side effects of having a lot of opportunities

• Creating a detailed long-term roadmap which becomes “a list of features” to be developed;
• Falling in the habit of “checking things off the list” (results in not questioning the return on investment and shifting the focus away from the value of the delivered features);
• Meeting the stakeholders’ requirements as output versus the stakeholders’ requirements as input to the output;
• Focusing on the output (delivering more features), and not on the actual outcome of the delivered features;
• Putting less emphasis on analytics and feedback.

Traditional vs. modern product development

Traditional product development (“feature factory”): once a feature is designed, implemented and delivered, it is checked off the list.

Modern product development: once a feature is designed, implemented and delivered, the outcomes are assessed and measured, resulting in iteration if necessary.

“For optimizing Dev Portals, it’s really essential that we take a more modern approach rather than just turning out features over and over, and over.”

How to avoid the side effects

After the MVP phase, spend time on the foundation and on new features, but also allot time for experiments.

Foundation & new features:

• have minimal uncertainty, and their development results in achieving an outcome, so they can be checked off the list;
• can be larger features that take weeks, months, or years;
• are only revisited when impacted by another feature.

Experiments:

• have a moderate or high uncertainty, and certain results are expected, but not always achieved;
• are timeboxed (emphasis on return on investment);
• can help refine “the list” of features with data; the data informs the decision about what should be on the list, or what should be done next;
• are structured to emphasize assessing outcomes, learning, and iteration.

An example experiment

1. Have a hypothesis - the belief that by doing something, you can achieve some result.
2. Set a scope or requirements - set a time frame for the experiment, specify what is measured.
3. Decide on key measures - outcomes that will prove the experiment is successful.
4. Set the success criteria - look at existing data to set realistic targets.

• Focus more on the outcome rather than the output.
• Assessing results is critical to any experiment: measure & assess, then iterate if necessary.
• Use analytics to make a data-driven decision, and optimize the experience by measuring and assessing the impact of the outcome.

Recommendations

• Determine how much time you can spend on experimentation.
• Bring the experimental mindset back into new feature development.
• Always have multiple key measures for any of your experiments.
• Focus on experiments as a learning opportunity (embrace learning & avoid output as a measure of success).
• Unsuccessful experiments are a success if you minimize investment and learn.

At this event, the presentations were followed by a panel discussion, where the speakers shared further thoughts and insights.

Panel discussion

Participants: James Noes, Matthew Revell, Laura Vass.