Authors and main contributors
The foundation: cross-team collaboration
Structured content and well-written, rich data is more important than ever. However, while developer portals have come a long way, we still often see teams (like those of technical writers or UX specialists) working in silos behind the scenes. In our experience, these disconnects become even more pronounced when AI comes into the picture.
Design without content is hollow and content without structure is invisible. Both fall short unless usability and information architecture are treated as shared responsibilities. For example, challenges arise when UX designers create intuitive flows without accounting for content requirements or when technical writers try to produce clear, relevant documentation without insight into user personas and their journeys. Without close collaboration, even well-executed work can result in a fragmented developer experience.
When setting up the information architecture of a developer portal, our UX team aims to reflect the company’s unique context: its business goals, user expectations, and long-term strategy. Our technical writers play a critical role in making content well-written, accessible, and logically structured: foundations that are increasingly vital for enabling meaningful interaction.
As you will see soon in the example of a global financial services provider we worked for, missing metadata, a lack of clear structure, and inaccessible legacy documents can prevent AI tools from retrieving reliable, up-to-date information.
Collaboration between UX and Technical Writer specialists
What might a fruitful collaboration look like? At Pronovix, we focus on the following areas to ensure that both UX and technical writer specialists bring their expertise:
When our UX and techwriter teams join forces, we ensure better outcomes. Even if our teams are not in the same meetings, we still work from a shared roadmap. For UX designers, it is crucial to involve the technical writers early, or the other way round: when you are creating content, make sure you check the personas the UX specialists set up and which user requirements need to be fulfilled. In other words: collaboration from the start is key. Long-term success is impossible if specialist teams are stuck in silos.
AI as an intermediary user
Speaking of personas, AI brought changes in this practice too. We use personas to translate complex user behavior into clear, actionable insights that support more focused, user-centered planning. Traditionally, we focused on human users, but now generative AI is starting to participate in the user journey as a persona in its own right. It gathers and interprets content to generate answers, recommendations, or insights for others. In this way, AI becomes a kind of intermediary user: it consumes to provide.
This shift challenges us to rethink our approach. To serve this new user well, we must ensure that portals are structured, annotated, and interconnected in ways that AI can understand and use effectively. When we treat AI as a persona, we can list its characteristics, needs, typical behavior, and the way it will interact with the portal.
A persona card can help us understand if the decisions we make around organizing the sitemap and setting up content will benefit or hinder users. In this case, the user is AI:
Designing for generative AI
The output provided by any generative AI is only as good as the structure and content it relies on.
AI should be able to:
- consume through predictable and clear structure, and accurate information,
- recognize and find relationships, dependencies, and semantic markers, and
- follow an accessible layout and content (with the help of alt text or captions).
We cannot just add AI as something new to the portal and expect it to interpret data and documentation as a human would, we need to ensure that what we have is prepared for it. For this reason, AI becomes part of the whole user experience design: UX and technical writer teams need to align more than ever to build developer portals that are helpful to human users and interpretable by machines, laying the foundation for AI-driven services.
While the previous chapters focused on tool-agnostic principles to help you build AI-ready developer portals, we know that choosing the right platform is a critical part of the implementation process. Based on our experience, we can recommend Drupal as a reliable and future-proof publishing solution (CMS).
A CMS is the foundation of the digital content infrastructure: it enables teams to create, organize, and maintain content efficiently without needing to write code. In the context of AI, a CMS plays a strategic role as it allows the structuring and enrichment of content with metadata, making it more accessible to humans and machines. With a CMS you can allow a multitude of users to access the site via a log in, fill in forms, search, or participate in discussion forums. A CMS can reliably scale to your needs.
Drupal offers a flexible and robust foundation for integrating generative tools. With its structured content model, powerful taxonomy system, and open-source extensibility, Drupal is well-suited to support RAG and AI-powered search.
In this section, we highlight some of the capabilities that explains why Drupal is suitable for future-proof developer portals, how to configure and operate RAG Search in Drupal. Then you can learn more about how llms.txt works and why it matters. While there are many more capabilities worth exploring, this white paper focuses on those that offer long-term value for your developer portal and are already available for implementation, empowering you to take action independently or with our help.
Drupal RAG Search
As we explained in the Semantic Search and Retrieval-Augmented Generation (RAG) chapter, instead of just returning links, RAG retrieves relevant content chunks from a knowledge base (Drupal content indexed in a Vector Database) and uses an LLM to synthesize a direct answer based only on that retrieved information. This provides users with contextually accurate answers grounded in our specific content, complete with source references.
AI Module (Drupal) orchestrates interactions with LLMs and vector databases. Key submodules/features include:
- Search API AI plugin: Handles the embedding process during ingestion.
- AI Vector database provider: Facilitates connection to various Vector Databases.
- Moderation submodule: Filters user input for harmful content.
- AI Providers: Choose from the most popular AI providers or connect your own LLM.
- DeepChat integration: Provides a ready-to-use Chatbot UI.
- AI agents: It is the framework to manage and orchestrate the agents.
- AI Automators: Tools to generate content or automate AI-assisted tasks based on content input. By integrating RAG into Drupal, you can provide relevant, context-grounded answers sourced directly from your knowledge-base without leaving the site.
RAG workflow in practice
Based on the represented high level workflow diagram, the two main phases include the following steps:
Phase 1: Content Ingestion
- Chunking: Drupal content is split into semantically meaningful units via Search API processors. If we follow the diagram, the document is embedded into the vector space through a Search API AI Plugin, and it will transform the data into digestible chunks.
Tip: Smaller chunks improve the precision of semantic search.
- Embedding: Each chunk is processed by an LLM (via the AI Module's Search API AI plugin) to generate a vector embedding (a numerical representation of its meaning).
- Storing: Embeddings, along with metadata (node ID, title, URI), are saved to the vector database. You can choose from different vendors: open-source (like Milvus) to Cloud-based (such as Pinecone).
Phase 2: Query Processing
- User Query: A user submits a natural language question via chatbot or another interface.
- Moderation: The input is checked for harmful or inappropriate content.
- Embedding the Query: The query is embedded using the same LLM model as in ingestion.
- Semantic Search: The query vector is matched against stored content chunks in the vector DB.
- Context Augmentation: Retrieved chunks are compiled into a prompt.
- Answer Generation: The LLM generates a response using only the provided context, which is then presented to the user via UI.
- The process can be repeated as much as needed.
Drupal AI Module
The Drupal AI module provides a framework for easily integrating artificial intelligence on any Drupal site regardless of the vendors. The AI module aims to provide a suite of modules and an API foundation for generating text content, images, content analysis and more.
You can combine the best features and approaches from the AI Automator, Search API AI and other modules in a unified framework and solution for AI in Drupal. There is an abstraction layer enabling integrations with third party AI providers such as Google Gemini, OpenAI (ChatGPT, DALL-E), Anthropic (Claude), Fireworks, Mistral and more. You can even use open-source models on servers you host and control.
As we see, in most cases, a simpler solution is better for initial experimenting, and for highly accurate results, it is necessary to invest in more mature tools. You can start your journey based on the mentioned suggestions, but if you need help, we can guide you through the planning, implementation, and maintenance.
Considerations and best practices
- Chunking Strategy: Balance chunk size for speed vs. contextual richness.
- Model Selection (Vector Database Choice and LLM Choice): Match the same model family for embedding and generation when possible.
- Prompt Engineering: The quality of a prompt plays a critical role in shaping the effectiveness of AI outputs. In many cases, the success or failure of the interaction hinges on how well the prompt is constructed.
- Moderation: Ensure safeguards for responsible AI use.
- Performance Tuning: Expect to iterate especially for threshold values and embedding strategies.
The RAG Search module and the Drupal AI ecosystem are evolving rapidly. With the current version, changes to how AI agents and assistants interact will open up new use cases and more flexible architectures. For full technical docs and examples, see the official Drupal AI module documentation.
Are you interested in this solution and want to know more about dependencies, installing, and enabling the required modules? Contact us and we can discuss your case.
llms.txt Standard
The llms.txt file is an emerging standard designed to support responsible content discovery and usage by large language models. It acts as a centralized entry point, guiding LLMs, web crawlers, and AI agents to relevant, high-quality content on a website while helping them avoid less useful sections such as navigational menus or archives. Typically written in Markdown or plain text for simplicity and ease of parsing, llms.txt may include brief descriptions, context, and direct links to deeper documentation or structured content (often also in Markdown). This format enhances both human readability and machine processing, facilitating more efficient indexing and enabling LLMs to generate more accurate, contextualized answers from the content it references.
To support more formal mechanisms for controlling data usage, we have open-sourced a Drupal Recipe that provides full support for llms.txt. The promise of llms.txt, alongside providing content in machine-preferred formats as well, is not necessarily for the crawlers of today, but for the agents not yet built.
Furthermore, it is about extending your developer portal: enabling it to serve both humans and machines with content tailored to their needs. A future crawler, seeing a pointer to a simpler, lower-token version of the page, could switch tactics in real-time. For our clients’ portals, we recommend enabling this option by default to improve machine discoverability.
From our comparison tests you can find in the previous chapter, consuming a streamlined version of a page can reduce token count, translating directly into lower cloud processing costs and faster throughput for LLM-based systems. LLMs benefit from reduced noise, and your portal should offer a clear directive to machines: one that will probably differ from what is designed for human navigation.
How it works in practice
In the LLM Support Recipe, we combine three Drupal modules to achieve full llms.txt standard support:
- The /llms.txt module provides support for authoring a full featured /llms.txt file.
- The Markdownify module enables output of Drupal entities as Markdown-formatted text.
- The Markdownify File Attachment submodule enables output of API specification files as code blocks inside MarkDown to provide full context for LLMs when it comes to API-related content.
The Markdownify module provides further support to the proposed llms.txt standard by turning all the site’s content into machine-readable format at their original URL appended with a .md extension, so that you do not have to worry about large HTML contexts.
Pronovix’s /llms.txt module aims to provide an advanced solution to manage the content of the llms.txt file. What it does:
- Provides a configurable and custom entity type to manage the llms.txt file’s header and other sections in an editable format.
- Adds menu tokens that enable users to embed dedicated menus in a machine readable Markdown format.
Our module has advanced caching and content management features, like section reordering and editable view of each descriptive section.
Benefits
The Pronovix LLM Support Drupal Recipe addresses the crucial need for websites to be AI-accessible, particularly for the nascent category of AI agents. To summarize, the module’s main benefits are:
- Enable AI agents: Provides information to help LLMs use your developer portal at inference time.
- Reduce AI cost: Simplifies documentation output to reduce token use via the LLM API.
- Increase inference speed: The reduction in token count results in quicker processing of your content in Markdown format.
By creating the structured llms.txt file and providing additional context with links, Drupal site owners can make their websites AI-ready. Coupled with the Markdownify module, this solution reduces token usage, accelerates processing, and enhances the clarity of information provided to LLMs.
Do you need help with empowering your Drupal developer portal to speak fluently with AI? Contact us and let’s discuss your case.
