Melissa Mahoney, John Williams - Annotate, Automate, & Educate: Driving generated OpenAPI documentation to benefit everyone

What problems MongoDB were trying to solve?

• Out-of-date docs: Some of their API Reference docs were not always up-to-date. They also provide several API tools for their users, and problems with their documentation mean that there will be difficulties with the tools as well. It was necessary to find a solution to provide more accurate and up-to-date information for their users.
• Code is canon: They wanted their code to be canon, meaning that they are going to use code as their ultimate source of knowledge when it comes to writing API Reference documentations. Swagger core allows annotations to existing Java code and to generate Open API specification every time a code is built, allowing them to create more accurate documentation that is synced with these resources.
• Empower writers: They wanted their writers to create more API Reference documentations that require more engagement, more insight and more creativity from the person writing it.

When they switched to working with OpenAPI annotations, their API Reference documentations immediately became consistent with the API code, which was a much faster rate than before.

Getting buy-in and coordinating across teams

Once MongoDB decided that they are going to be working with OpenAPI annotations, they had to find a way to convince the Lead engineers and Stakeholders of their company that this is the right direction.

• They use processes that are familiar to their Stakeholders. Also, learning engineering processes along the way to make sure their writer team is reaching the correct audience in the format that was familiar for them. This way the writers ensured that they had approval from the executives and the engineering team.
• “Company initiative”: it is an umbrella term that covers several large EPICs towards a common goal. By the time MongoDB started this project, they already got a green light from the company to start working on creating new CLIs and setting technical design standards for the existing APIs. With this initiative, it was possible to tie the existing project to a company business goal that already had buy-in.
• After the work with the engineering team was completed, the writers would help out in the support phase of the projects.

Over the course of two years of this project, they've completed:

• 4 proofs of concept,
• 7 unique scopes and specs,
• received approval from over 35 stakeholders.
• Wiki pages and presentations for the engineering team were also created.

The implementation

• The first step was to implement a Scraper script. The spec they generated before had a minimum retained part of the handwritten docs that was created. Using the Python Library, they wrote a script to extract the handwritten Reference pages, parameters and descriptions to a CSV file.
• Next, the writer team started an editing phase and made sure they’re using up-to-date and consistent language throughout the Reference documentations, while also adhering to their style guide.
• The work continued on with an ETL phase (ETL: extract, transfer and load). The first step here was to find a way to generate annotation blocks, so that MongoDB’s writers could easily add annotations to the docs they are working on. After several trial and error, they were able to create blocks that can be inserted into documents.
• The writers pasted in the annotation blocks for each resource, while making sure they didn’t break the build. They ensured accuracy and consistency remained in their docs and sought approval from the engineering team.

During these phases, there was constant support and feedback from the engineering team. The last step on the writers' end was to publish their documentations with the new blocks.

Handing off responsibility for the API Docs to the engineering team

The API docs handed over were using OpenAPI annotations. The writer team’s responsibility from now on is to review copies and every single port request.

As a result of this project:

• their users have access to API Docs that are immediately consistent with the API code in production,
• their process now has fewer steps, so it is easier to iterate more quickly when they need to,
• their users feel more empowered to accomplish their goals using the Atlas REST API.

Lessons learned

The project almost took up 5 years. During those years the writer team learned:

• to raise awareness early and often. Sending weekly or monthly updates to the engineer team members would have been helpful to help them stay updated with their progress.
• To set milestones for themselves during projects of this length and depth. They deprioritized this project over more pressing issues, which made the whole project take longer to finish.
• It’s not necessary to automate everything. The automated annotation block creation was a great accomplishment, but some of them had many issues and required extra manual work on them.
• The API code might need to be refactored, especially if dealing with several years worth of code. Some of the APIs were not compatible with OpenAPI specifications. To solve this problem, they hand-wrote schemas for these APIs.

The engineering team is working to refactor the code to make it more compatible with OpenAPI. If they can remove the handwritten schemas, then they have successfully accomplished their goal of generating all of their Reference docs from code.

What’s next?

MongoDB has several new projects for the future to work on:

• Generating client SDKs in several languages of the OpenAPI specification.
• Writing more API tutorials.
• Add more annotations in other contexts where it is appropriate.