Amara Graham (camunda): The Good, Bad, and Ugly of AI + Docs

This talk was presented at the AI The Docs online conference on April 4, 2024. We are thrilled to share the recording and the summary with you.

Visit the talk summary page to see all of the presentations from the conference.

Amara Graham

Head of Developer Experience at Camunda

Amara's presentation

Amara shared insights on integrating AI with documentation. The presentation outlined the benefits, challenges, and future considerations of implementing an AI agent in their documentation system.

Key Takeaways

Positive Outcomes and Validation

• Successful Implementation: The AI agent has been well-received by both the developer experience team and other internal stakeholders since its launch earlier this year.
• Effective Responses: The AI agent accurately responds to user queries, indicating that the documentation is robust and effective.
• Iterative Process: The development of the AI agent was iterative, adapting as the team identified the specific ways they wanted to leverage AI.

Essential Features of the AI MVP

• Citing Sources: Ensuring the AI can cite sources to build trust and allow users to verify information.
• User Feedback: Allowing immediate user feedback for corrections to enhance the accuracy and reliability of responses.
• Low Effort Implementation: The solution needed to be easy to implement and maintain.
• Impressing Stakeholders: The AI had to meet high standards set by the team, including being low-risk and efficient.

Current Usage and Benefits

• CAP AI Integration: The AI agent uses CAP AI to ingest various knowledge bases including documentation, forums, and select company web pages.
• Continuous Improvement: Regularly measuring and monitoring engagement to improve the AI's performance.
• Conservative Responses: The AI is designed to provide accurate, conservative answers within its domain, and is capable of admitting when it does not know an answer.

Challenges Faced

• Vendor Selection: Early challenges included evaluating numerous AI vendors, many of which were immature or seemed unreliable.
• Community Forum Data: Forums provided chaotic and unreliable data sources, complicating the AI's ability to parse and respond accurately.
• Establishing Criteria: Defining success criteria and determining good source material were significant hurdles.
   

Addressing the Ugly

• Initial Skepticism: Internal surveys revealed that early AI implementations often produced unsatisfactory results.
• Pivoting as Needed: The team was prepared to switch vendors or solutions if the AI's responses were inadequate or incorrect.
• Improving Documentation: Recognizing the need to maintain and improve documentation to ensure it is parsable by both humans and machines.

Advice for Stakeholders and Implementers

• Enforcing Good Practices: AI can enforce good documentation practices, highlighting the need for clarity and maintenance.
• Leveraging User Questions: Using actual user questions to refine documentation and improve user experience.
• Reducing Support Load: An effective AI can reduce the volume of "how do I" questions, easing the support team's workload.
• Testing and Risks: Design a set of test questions and sources to validate the AI's responses, and be prepared for some trial and error.
   

Conclusion

Amara concluded by emphasizing the importance of continuous improvement and adaptability in the journey of integrating AI with documentation. This approach ensures that both the AI and the documentation remain effective and reliable tools for users.