As part of our work to make Drupal 8 the leading CMS for developer portals, we are implementing a mechanism to import the OpenAPI (formerly known as Swagger) specification format. This is a crucial feature not only for dedicated developer portals, but also for all Drupal sites that are exposing an API. Now that it has become much easier to create a RESTful API service in Drupal 8, the next step is to make it more straightforward to create its API reference documentation. That is why we think our work will be useful for site builders, and not just for technical writers and API product owners.
Swagger is a REST API documentation tool, it provides a specification language for describing the APIs and also a set of support tools. One of those tools is Swagger UI, which generates an appealing and readable layout for API endpoints and methods. The Drupal community is considering using the Swagger specification to document Drupal 8 core web services, and Swagger tool adaptations can be found in several contributed modules. In this blogpost we will introduce some of these modules and explain how we want to go beyond the shallow integration that most of them have done, to take full advantage of Drupal’s architecture. But before diving into the technical details, we want to list the features that we seek in the ultimate API reference CMS.
6 features that make the difference between a good and a great API reference system
The following are 6 technical developer portal features that customers have requested from us in the past 2 years working with Apigee’s developer portal. They provide functionality that go beyond what most API management platforms provide.
For an overview of other Developer Portal components (besides the API reference), check out our blog post series on this topic.Read more
This feature list is based on our investigations of existing developer sites, our practical experience from creating developer portals and architecture workshops we’ve held.
1. Storing multiple API versions: versioning
As API services can have multiple supported versions (e.g. v1, v2) in parallel, a Developer Portal should provide a clear user experience that gives visitors the option to choose which version they would like to read about (but still default to the latest supported stable one).
2. Track changes in the documentation of each API version: revisioning
Most developer portal platforms rebuild the documentation as part of an automated build process, Drupal’s revisioning system allows editors and site maintainers to make and track changes in specific versions of the API documentation. While this is less important for the developers that use the documentation, it is an editorial feature that can be useful for technical writers and site owners.
3. Possibility to attach conceptual content to the API reference
API references are very technical and factual. Sometimes developers need more verbose documentation that provides a longer explanation of the context an API operates in. That is why, several of our customers have asked us to add conceptual documentation to their imported content - about domain language, underlying architecture, data models, or code samples that surround an API call.
4. Access control for individual API methods
In order to restrict the visibility of certain API methods (e.g. for partner APIs), a Developer Portal must allow site maintainers to set granular access permissions/restrictions for specific versions, endpoints or other parts of the documentation.
5. Trying out API calls on the Developer Portal’s UI
Integrating a system with an API service can be accelerated by a Try it out feature, that helps developers to decide which API endpoint with what parameters to use in order to get the expected result.
6. Importing reference documentation from a version control system
Recently technical writers have also started using the online collaboration and versioning tools that developers work with. Documentation is now often committed into code repositories, especially when developers contribute to the writing process. One key problem with this approach is that, apart from the API reference documentation where most teams use the Swagger specification, there is no obvious standard to store the content and layout of documentation. We’ve been working with markdown topics, and manifest files to allow technical writers to store conceptual documentation and their navigation structure (what we used to organize in a book hierarchy in Drupal) separate from the API specification. This way all the documentation can be stored in the version control system (e.g. a GitHub or GitLab repository).
Existing Swagger modules in Drupal
The Swagger API documentation toolset covers the entire publishing process: building (Swagger Codegen), documenting, and visualizing (Swagger Editor, Swagger UI). Existing Drupal modules typically focus on the building and visualization steps.
As usual Drupal.org has some modules that seem to be abandoned, but there are two Swagger docs related modules that have been maintained in the past year. One is the Swagger UI Field Formatter module, it renders fields with valid Swagger source file using the Swagger UI tool. The other is Swagger php module (sandbox only), it can generate JSON formatted Swagger code based on annotations and it can render that code using the Swagger UI.
Both of these modules use the Swagger UI project to generate a human readable output from the specification. Swagger UI only needs a valid source file to generate the output and the ‘Try it out’ section (for sending requests to the endpoints); it is useful if you only need to publish the content, but it has its limitations.
If you are interested in other free and open source API documentation generators, check out our blog post about other solutions.Read more
The problem we see with this solution and most other API documentation tools is that API providers usually need access control, search, and conceptual documentation for their API descriptions. These functions demand a different approach.
Don’t just show it, integrate it!
After careful evaluation, we came to the conclusion that the currently existing Swagger tools can’t support the 6 advanced API documentation features our customers request from us. To make the API documentations fieldable, revisionable and to be able to apply custom access control on all components of them in Drupal, a more robust API integration is needed. No open-source module is available for Drupal 8 that does this, so we decided to make it a key contribution we would work on with our team.
Since there are other specification languages (such as RAML or I/O Docs) that are widely used and that store similar information as the OpenAPI format, we take great care to make sure that our architecture would be extendable and reusable.
Mapping Swagger objects into Drupal data types
To get a flexible system that can be extended and altered with proper Drupal 8 solutions, we designed custom Drupal 8 entities and field types for every piece of a Swagger source file. The first step was to observe the individual Swagger specification elements and to decide the most suitable Drupal 8 data types for storing them.
We just finished the planning phase of the entity architecture, the overall structure won’t change much, but there might be some small changes during the implementation period.
The below image describes a small part of the planned entity architecture. We defined a vendor independent API documentation as a content entity (basically the root entity) which might have bundles, providing the ability to extend the base system with vendor specific formats other than Swagger (e.g. I/O Docs or Raml). Based on this concept, each specification language format makes a new bundle with vendor specific fields. By default the Swagger 2.0 specification format bundle is provided. Each piece of content in a bundle represents a different API version, so multiple versions (e.g. v1, v2) can be made available in parallel on the Developer Portal (feature 1).
All of the documentation components are tied to the properties or references of an API documentation entity. For example API endpoints form another content entity type, which can get referenced from the root (API documentation) entity. Moreover, as we are planning to use fieldable entities, any additional information can be attached to them easily (feature 3).
Thanks to the OOP nature of Drupal 8, reusable properties and methods can be attached to entity classes through traits. For example, base field definitions of the
consumes and the
produces specification properties can be defined in traits and used in multiple entities, as they can be attached to the API documentation entity or to an API method/operation (overriding the default global settings of these properties). The
consumes and the
produces properties in the Swagger source are technically MIME types (such as
application/json), so they can be collected into a vocabulary as taxonomy terms.
Thinking in traits will also enable us to extend the default API specification with custom properties (e.g. extend Swagger specification objects with ‘x-’ properties). Code snippets could for example be included for different programming languages (such as Java, PHP, Python), these might help the readers to understand the API reference.
With the above architecture we can map specification languages into a Drupal entity system where basic revisioning is supported by default (feature 2). Although custom access control can also be added to any type of entity and its fields (feature 4), it’s not as powerful as Drupal’s node access control system. There is already a Drupal core issue that tries to expand the node access grants system to a generic entity grants system, and we are trying to contribute to it while working on our Drupal Developer Portal.
Importing the data into the Drupal system
For the import process we leverage Drupal 8’s Migrate API to import any type of API specification formats to our custom entities and to store them in a unified way. Source files can be either uploaded in the UI or imported from a Github repository (feature 6) through a documentation importer that we are building to support editorial workflows that rely on code repositories and that automatically publish to Drupal as part of a continuous integration process.
If you are interested in our GitHub importer and Migrate processing solution, join our Developer Portal mailing list to receive notifications about blog posts on the subject.
We chose Drupal 8 as the framework for our Developer Portal, because it already had a large number of features that our customers need. With a 10 year long history in Drupal, we are obviously somewhat biased, but even if we disregard our prior expertise, we believe that Drupal is one of the best CMSs for building documentation and developer portals.
That is why we decided to extend the existing solutions, with a sufficiently complex system that would enable us to address all the needs our customers have. Some of our code is still in stealth mode, the developer portal market is a relatively small niche, and we need to make sure we can find a sustainable way to give back to the Drupal community. That is why in parallel to our development, we are working on a new business model for our distribution to make sure we will be able to continue sharing our work with the wider community. We are committed to the open source community and credo, but we want to prevent some of the failures we have seen with previous Drupal distributions, more about that in a later post...