Editorial Services: a Central Utility for API Teams

Co-Founder. Editor, Research & Knowledge Sharing.

Jun 03, 2022

How do you set up the processes to make your own API documentation?

To transform your business by making APIs an integral part of how your organization works you need to distribute API ownership throughout your organization. For a continuously developed or supported product you will need a central API team to do more than prove the business value of the API. API teams need to think about the maintenance and support of the APIs they are making.

That is why we think central API teams should transition from an “API pilot team modus” to an expertise center that supports developers throughout the organization. Scarce skill sets like documentation expertise and developer experience best practices should be shared. This is the best way to transition an organization into the API-first mindset, necessary to capture the value of internal agility.

Editorial services don’t happen in a vacuum, they need to be adjusted to your company culture. Companies are not machines where you can drop a new piece in and it will just work: you need to look at the whole system and the culture that informs its behavior.

We think there are patterns applicable to documentation scaling, and at the end of the presentation I will return to the idea of patterns and show you an innersourcing pattern template. The aim of this presentation is to give you a scaffold for your own learning, and your own research into centralizing your documentation efforts.

Take a flock of starling birds: they become something different because of their interactions. It is not the simple sum of birds and there is no central orchestrator. Companies are similar – or at least there is a cultural shift happening to urge companies, even very large enterprise companies to move like these murmurations of starlings.

For example, an organization with lots of technical writers starts downsizing. They introduce an automated editorial system to replace existing technical writers. This won’t work, because you cannot at the same time give a signal that docs are not valuable, and yet expect developers to start doing documentation. Instead you could set up automated editorial services as an additional unifying force to standardize the editorial practices across business units. On the other hand take Organization B, that doesn’t yet have a well worked out documentation practice. They could introduce a centralized editorial service as a tool that supports developers when writing documentation, as a first step towards a better documentation infrastructure. The key is to nudge and enable the documentarians and the software engineers to iterate the setup together.

A central content team can be the composer and the supporter of these user journeys. In the case of documentation, one has to look at the whole docs lifecycle and cannot entirely separate editing from writing. The type of developer portal also plays a role. One of the strongest driving forces behind API programs in large multinational companies is to increase their internal agility. Most successful public APIs started out as private, internal APIs.

Public-facing API projects, which can be considered products, can afford to invest a lot more money in documentation and user experience driven developer portal improvements. Internal APIs rarely have that luxury. Usually there are a few external APIs; and then there are a lot of internal APIs , those are what connect everything together within the organization.

Pain points of documenting internal APIs

There are numerous problems with the execution of internal API programs, these are the kind of frustrations that make people change their jobs. How do these problems evolve?

A lot of teams are making one-off internal APIs, because often a very particular API is necessary to do a specific API. Realistically, the first team to use the API may be the only one for a while, and so often the team creating the API just sets up the API with the other department and they do not document the process well or at all.

This results in very poor discoverability, confusion in domain language across departments and the lack of API design standards keeps it all inconsistent.

The API’s description is often bad, written only from the creator’s point of view, and it may be only six words. You are already lucky if you get the API’s reference docs. No tutorials, no conceptual docs. Maybe a bad, short description that is part of the Open API specification, because a short description is a compulsory field for submission.

Onboarding is often only happening in live workshops. This in turn allows a fragmented and complex authentication process to linger on.

Distributed API ownership

Most organizations never consciously address the experience of the developers teams that create APIs. This is only logical, Developer eXperience doesn’t always need attention: a team that is dedicated to APIs will overcome friction on its own terms. In large enterprise organizations, however, where API development might be a secondary priority of a team, DX can make the difference between a failed and a successful digital transformation initiative. If your objective is to transform your business, and to make APIs an integral part of how your organization works, then you need to distribute API ownership throughout your organization. A central API team is great to prove the business value of APIs, but if you build APIs as a one-off project instead of a continuously developed product, your APIs will soon fall into disrepair.

At Pronovix we developed a developer journey model that helps us to think about how to improve the experience of the developers who build the API. This is especially important for internal-use APIs because you will not have the budget to iterate on docs to polish them after publication. Traditionally, it would be at QA and publication where an editor comes in, but a central editorial team can look at all the steps to reduce the need for interventions.

For example:

If you can step in at the enterprise-level inventory as editors to make sure the names and short descriptions make sense outside the departments, you can reduce discoverability problems and duplication of work.
If you can get your engineers to agree on style-guides for API design, the result will be more consistent APIs that in turn allow for docs re-use.
You can adjust the submission process to the developers’ tools and toolchains.
Templates and guidelines for tutorials and conceptual docs can get you at least a first draft.
Quality assurance and publication is the field for implementing docs as code practices and extensive automated testing.

All this will have a positive effect on your feedback and support options, and you can further push the envelope with gamification elements.

Innersourcing for your APIs

There are patterns that are proven to work to help innersourcing and the cultural change that the practice of innersourcing requires. Innersourcing is the adoption of the open source development paradigm in organizations.

Developers are always keen on patterns that they can reuse. The InnerSource Commons is a community around this, they hold events and conferences about their results.

We need to find a way to break down the barriers and nudge people to collaborate.

Adopting inner sourcing practices can help you with

software reuse across teams
improving the quality through more reviewers with more eyes
opening up innovation to a shared, broad expertise across your whole organization
potentially faster time to market
standardization and better inter-department mobility of developers.

It is key to understand that innersourcing is not a defined methodology but more of a psychology and adoption of those behavior practices that fit your company.

InnerSource Pattern Template

There are great research efforts going into capturing and analyzing these patterns of innersourcing.

Hewlett-Packard, Alcatel, Phillips Healthcare, IBM, SAP and BASF are in the vanguard of these efforts, you can find several intriguing case studies from them.

Every company has its own development culture and practices. There are no cookbooks for InnerSourcing yet. Open-source projects usually rely on a product to drive the circle of develop/publish/use, test/contribute. The InnerSource Commons uses patterns to serve the same purpose: these patterns provide a level of abstraction showing a proven solution to a specific problem within a given context.

“We have found that top-down doesn’t work but bottom-up doesn’t scale.” Riona Macnamara

An innersource project is not owned by a team or a department. You can contribute or use others’ contributions wherever you are in the organization. For the same reason, it can be a challenge to start, and to evaluate your progress.

Patterns can help to change an organization’s culture and behavior, to make sure departments use the same standards and to nudge people to do something the same way without forcing them.

How can a centralized editorial utility address these issues with internal APIs? You have to model the developer journey, identify its stages and standardize at all stages possible.

At Pronovix we developed an upstream developer journey model that helps us to think about how to improve the experience of the developers who build the API. This is especially important for internal-use APIs because you will not have the budget to iterate on docs to polish them after publication. Traditionally, it would be at QA and publication where an editor comes in, but a central editorial team can look at all the steps to reduce the need for interventions.

For example:

If you can step in at the enterprise-level inventory as editors to make sure the names and short descriptions make sense outside the departments, you can reduce discoverability problems and duplication of work.
If you can get your engineers to agree on style-guides for API design, the result will be more consistent APIs that in turn allow for docs re-use.
You can adjust the submission process to the developers’ tools and toolchains.
Templates and guidelines for tutorials and conceptual docs can get you at least a first draft.
Quality assurance and publication is the field for implementing docs as code practices and extensive automated testing.

All this will have a positive effect on your feedback and support options, and you can further push the envelope with gamification elements.

Experience design for your API documentation

Introducing gamification by applying the elements of game design as human triggers to increase engagement are a good way to encourage good documentation practices.For example, in a large enterprise company you can have internal dashboards for showing how many APIs you have, the development status and quality of those APIs. You can have an overview of the APIs with 5-star ratings depending on what criteria they have passed.

In UX design, we create engagement flows with engagement loops. If you have a community around your APIs, it is a good idea to spotlight the latest contributions, to enable immediate interactions with the documentation, to let users follow up on the state of your API.

You can go beyond dashboards and dig deeper into human psychology.

The Octalysis model was developed by You-kai Chou. It’s a design framework, that lays out the structure for analyzing the driving forces behind human motivation and engagement. You would use it to apply the eight core human behaviour drives, to motivate a user to complete a task more efficiently through an interactive experience.

There are many other models. Octalysis is an analytical, measurement framework, applied by enterprise companies such as Ericsson, Cisco, Accenture, Lego, eBay and many more.

The eight Core Drives are charted on an Octagon shape not simply for aesthetic purposes, but because the placement determines the nature of the motivation: the basic balancing forces behind our actions.

The left side of the octagon gathers extrinsic motivational drives, while the right side are of an intrinsic nature. Those on top are the white hat motivators, those that leave us feel good but do not create urgency - the flip side is the black hat motivators, those actions we keep taking although they leave us feel bad.

In experience design, we would first set the key behaviors we want to see from our users. Then we analyze what sort of interactions happen among the users. We then pick the gamification elements that fit this behavior. Applying the octalysis at this stage would show us what type of system we have and help shifting it in other directions.

Techwriter practices for API documentation

Technical writing practices are rapidly changing, and the pattern of tech writers basically becoming technical information editors and documentation community managers is coming your way.

How can a technical documentation editorial process interface with the software code?

Designing developer journeys
Treating docs as if it was code
About the definition of done
Automation of your docs testing
Gamification

Docs as Code for API documentation

The interface is basically when you make a pull request. If you manage documentation in the code repository, then the editorial process parallels a code review.

This paves the road to documentation being part of the Continuous Integration and Continuous Deployment (CI/CD) processes and being part of your definition of done. Especially in the case of APIs, keeping docs and code in lock-step is crucial.

Advice on how to implement such change in your organization:

Focus on the engineer, adapt to their existing tools, avoid context switches.
Start small and be reliable, iterate often.
Impact the life of the users, solve their problems within their workflow.
Version control is teamsport. Embracing the software coding toolset and culture for documentation is not an easy shift from a legacy situation.
Make use of linters for automated checking of patterns of language use, spell check, and terminology.

What to watch out for:

Tech writers who used to be responsible for all deliverables now have to rely on other teams for API work.
Managers of higher-paid developers will want to make docs the problem of the lower-paid writers.
Getting resources for web development that is unrelated to product development is a problem.
Who gets to control what gets done when? The Dev team or the Docs team? Product or Support? You need a Content strategy overall across disparate sources.
Style guides, image standards, tool standards, basically, agreement is necessary.

API documentation development environment

How to implement a truly iterative documentation process into the code workflow.

In a docs as code workflow, updates and small changes to your documentation are almost instantaneous. However, to document new features or an even entirely new product, you have to adjust the scope. In the planning phase you have to account for the documentation types you need and you have to have a content model for each type that defines the parts, structure and organization for reference, concept, task or other types.

In the development phase, you are going to write, review and test content, just like you would do with software code – and you need a documentation development environment set up. You do need those content models and templates and you will also need to provide information on how to contribute: create guidelines and training materials.

In the research phase you will have to map the user stories into docs and see how these get implemented into product features by the developers. The time needs to be planned and the documentation has to go hand-in-hand with development: this way, coders can provide at least the first draft of the docs.

If you have a CI/CD environment set up, you can commit your first pass at the docs into the repo and immediately see the rendered content in a staging environment.

Even with clear principles and best practices to follow, we find there is a need for editorial guidance and review. The technical writer’s traditional function changes to that of a project manager for documentation, but writing well is still a learnt skill and editing is still partly a soft skill.

There are some gaps between how we can treat code and docs: code is extensively and automatically tested. Docs used to be tested mostly manually, mostly on language and syntax, as part of the review process. The code frameworks, however, can be leveraged to do testing of docs.

Separate the traditional full review process into:

automatable parts, such as linters for syntax and language and examples testing;
and into the only human-doable reviews, that are user stories and product implementation standards.

API documentation in Definition of Done?

Is it possible to not allow a feature merge until the docs are done?

Rob Woodgate has an exhaustive poston what to consider:

Is documentation part of your deliverables? YES if it’s an API.
Can other team members write the docs? YES if it’s minimal docs for an API.

The rest you need to carefully answer for yourself:

What proportion of the docs can be done quickly or be automated?
Do you get all the information in time to work with?
Do you have standards and templates?
Do enhancements happen at the beginning of the sprint, and bug-fixes at the end?

In an optimal model the developer engineer drafts the complex feature’s documentation. The technical writer team ensures compliance with the style guide and the format constraints. Technical review becomes a priority.

Adopting the engineering workflow and tools gives a much higher chance to put docs into the definition of done.

API documentation as a thinking tool

A little about the element of human empathy and about semantics.

“It would simplify our work greatly to automate the specification and concentrate our talents in developing tutorials. This decision may look obvious in hindsight, but we had to do some research before we came to that conclusion.” - Tony Sansone, MongoDB

To date, an API’s conceptual documentation cannot be automated, but we can do a lot to make it machine consumable. We need to establish API description formats as a middle ground, to write concepts alongside code and enable developers and tech writers to collaborate according to docs like code practices.

In Writing for thinking, Zdenek Nemec an API design consultant and co-creator of the API Blueprint standard, said the following:

“Documentations of today are designed to sort of funnel the users into your systems, to get something sold or to get signups. We are at an age where people are programming first before they start thinking. I propose that we should shift our minds and think of Documentation as a thinking tool, it helps people to think about how a system works. Documentation is not just a marketing tool.”

“The very important parts in API documentation are obfuscated and burdened under technical details. You know how to use it but you don’t know what it is.”

We can give all the technical details of a system, we can automate their description, but there is a human element of empathy within the reader’s specific context that is necessary to explain them what something really is.
As Z said, we need to think about what is it we are documenting, and what makes this something unique. In case of APIs, it is the data that matters. Tutorials are nice, but they are already narrowing down your thinking.

Define domain semantics first

First explain the specific terms in your business, that is your domain semantics: say what is the data exposed by the API, together with what you can do with that data and how.

You need to get the software developers, the business stakeholders and the technical writers to decide and define the domain semantics first. This results in systems that can last longer, that can evolve and scale better.

Ontology, domain knowledge

Controlled vocabularies can bring back the lost glue into your fragmented documentation. Emilie Boillat gave a talk about how you can start small with a glossary or a thesaurus and then evolve it into a taxonomy or better yet, a full-out ontology.
Ultimately, controlled vocabularies and such clear semantics are what can enable machines to consume APIs, because until then it requires humans as the ultimate semantics machines to connect the unsaid dots.
So you would want to bring in a data modeling expertise, to set up the ontology just right for computing systems to understand.

Ontology engineering aims to make explicit the knowledge contained in software applications, and organizational procedures for a particular domain.

Since domain ontologies are written by different people, they represent concepts in very specific and unique ways, and are often incompatible within the same project.
As systems that rely on domain ontologies expand, they often need to merge domain ontologies by hand-tuning each entity.
At present, merging ontologies that are not developed from a common upper ontology cannot be automated, is a largely manual process, it’s time-consuming and it’s expensive.

Linters for automated API documentation

Linters are already here to help you automate docs review and build tests. Linters are testing tools to improve software code. They flag mistakes, recommend alternatives, remind you of missing elements or outright stop you if so set them. Originally, you could lint for syntax only, but you can also check semantics and style. Linters are meant to be part of your automated technical toolchain, in your editor and your build system.

The principles of linters are available for creative writing, mostly for the English language. In technical documentation, you can mix code linting and language linting together and the options are getting more and more elaborate and sophisticated.

A linter assisting your reviews and your editorial team, in your automated or semi-automated docs build is a great step towards consistent, complete and correct API documentation. Making contributions easy is the goal.

The decisions on what to do with the linter’s warning messages are complicated. Avoiding false-positives is crucial for company-wide adoption, for instance. The settings are a balancing act between too precise or not precise enough. You will probably iterate a lot on which doctests must pass for build.

API documentation style guides

Style guides help to enforce the reference docs in a certain format. They require adherence to overall standards that will help to improve the quality of the deliverables that get published in the developer journey. There are tools that can automatically check your docs against a style guide.

For example the IBM Style guide linter was made by the Write The Docs community. You can also find testing tools from the Plone community.

Are you building a developer portal? Interested in a shortcut? As a devportal company, Pronovix has created developer portals for 70+ customers since 2016. Talk with us to learn how our Zero Gravity developer portal can accelerate and simplify your launch.

References

Updated June 2022. This article was originally published in October 2018 as an abridged transcript of the presentation given at IDEAS two-day industry conference organized by the Center for Information–Development Management. The theme of the conference was Judicious Editing, where we spoke about centralizing the editorial processes and distributing API ownership throughout the whole organization.

All Pronovix publications are the fruit of a team effort, enabled by the research and collective knowledge of the entire Pronovix team. Our ideas and experiences are greatly shaped by our clients and the communities we participate in.

Editorial Services: a Central Utility for API Teams