Kumar Dhanagopal - Making sense of analytics for documentation pages

This talk was presented at API The Docs Virtual 2023 Feedback, Metrics and Analytics event series on 25 January. We are glad to present the video recording, slide deck, talk summary, and the panel discussion below. Enjoy!

Visit the talk listing page for an overview of all presentations!

Kumar Dhanagopal

Staff Solutions Writer at Google

The presented views and idea is entirely Kumar’s, which is based on his experience over the years. These statements do not necessarily reflect his colleagues’ opinions.

How analytics can (and can’t) help

Analytics can:

• show who the users are (e.g. Where are they from? How do they use the published information?),
• reveal which pages the users visit (e.g. How do they navigate? How much time did they spend reading the content?).

The collected information can help to improve the quality of decisions that a company makes when they plan to write and maintain documentation. But analytics can’t provide all the needed information.

Analytics can’t:

• show information on user sentiment, intents, experiences (e.g. Were they satisfied or frustrated?),
• tell if the users skim the content or read it carefully.

To answer these questions, we might need to use other sources (e.g. user ratings, comments), but even these sources might not help to fully understand the user’s sentiment.

We have to be careful with data, because it is easy to manipulate them. Keep in mind: What are the sources? What is the agenda? Correlation does not necessarily imply causation.

Overview of key metrics

Bounce: a visitor enters a website through a page and then exits the website without going to any other page.

If a user:

• exits the website and then returns in a certain predefined time period, the second visit might not be counted as a separate visit.
• enters a website and stays there more than a certain period of time, then the analytics tracker can terminate the visit at the end of the predefined period.

The visit limits and timeout settings can be different for each analytics tool. They ensure that the frequent exits, entries and delayed exits do not skew the visit-related metrics.

The analytics process

1. Define the problem or goal: before setting to use analytics data, find out the business goal/problem.
2. Get the required data: we need to know what metrics attract, understand how each metrics are measured, make sure that the analyzed data is from the right time period, and it represents the customer base.
3. Prepare the data for analysis: carefully scrub the data that is not relevant, find and fix inconsistencies (e.g. some metrics are missing from some pages), and aggregate connected data (e.g. a page might be served through multiple URLs » aggregate the data from all related URL variants).
4. Analyze, explore, visualize: what we do depends on our goals, the quality and quantity of data. We need to be careful with going too far with interpreting.
5. Describe, diagnose user behavior, prescribe changes, and predict the effects of the changes.

Descriptive analytics describe the key features of a given data set. E.g., “What are the top 5 popular pages on the site?” Diagnostic analytics shows why data is the way it is. E.g., “This page is extremely important but it gets relatively low views. Why is that? Is this even a problem?” Prescriptive analytics indicate what we can or should do in the future. Predictive analytics predicting or projecting future events based on the current and historical data.

Interpreting metrics

Recommendations:

• validate the data,
• avoid cross-product comparisons (there are several factors how users consume the published documentation),
• look at trends, not just absolute numbers,
• wait metrics to mature and become meaningful (e.g. wait until the docs have been in the field at least for a few months),
• use visuals to tease out trends and stories,
• keep in mind: offline and second-hand usage is not tracked,
• consider the length, type, format and the structure of the content,
• use analytics data as supplementary input for decision making.

“Analytics data can tell us how our users interact with documentation, but it might not tell us what our users really need or how they feel about the experience of consuming our information.”

Interpreting low page views

Page views are not equivalent to a page’s popularity. We should take a look at trends, the age of the page, and whether the page is discoverable. Zero-view pages: these pages will not show up in analytics reports.

Interpreting bounce rates

A high bounce rate is not necessarily a problem: on a documentation site, it could mean that most visitors actually found what they wanted. If a user’s session times-out, it might be counted as a bounce.

Interpreting time spent on page

The interpretation can be difficult: does somebody spend much time on a site, because the content was useful or it was hard to understand? In addition, time spent is not calculated for the exit pages.

Analytics “targets”

Many companies use KPIs (key performance indicators), which makes sense for certain websites, but not for technical documentation sites.

Analytics can help understand how the users consume the information » use the understanding to make decisions about how we plan and prioritize the projects.

Consider the availability of reliable data and whether the organization has the history and culture of making decisions based on data.

At this event, the presentations were followed by a panel discussion, where the speakers shared further thoughts and insights.

Panel discussion

Participants: Serah Njambi Kiburu, Kumar Dhanagopal, Laura Vass, Kristof Van Tomme.