Ellis Pratt (cherryleaf): Streamlining API Docs Updates: Leveraging GenAI for Enhanced Efficiency and Accuracy

This talk was presented at the AI The Docs online conference on April 4, 2024. We are thrilled to share the recording and the summary with you.

Visit the talk summary page to see all of the presentations from the conference.

Ellis Pratt

Director at Cherryleaf

Ellis' presentation

Ellis presented a comprehensive exploration into the use of GPT-based tools to automate the updating of documentation in response to changes in software repositories. The main focus was on building proof-of-concept models to test the feasibility and effectiveness of these tools. 

Key Takeaways

Problem Statement

• Complex Documentation Management: The challenge was to identify and flag documentation updates needed in response to frequent software releases.
• AI and Automation Potential: The goal was to explore AI tools capable of comparing repository commits with help center documentation to identify necessary updates.

Proof of Concepts Developed

• Understanding GPTs: Ellis explained the concept of GPTs (customized versions of ChatGPT) that can be tailored for specific tasks if one subscribes to ChatGPT Plus or Enterprise.
• Configuration Process: The process involves setting up prompts, canned responses, uploading knowledge files, and integrating external APIs.
• Tourist Guide Example: A simple GPT was created to update a tourist guide for London based on weather data from an API, demonstrating the basic functionality.
• Open Source Mailman Project: A more complex example used GitLab’s API to update the Mailman project documentation by comparing commits with existing documentation.
• Joplin Project: Tested handling multiple commits per day in a larger project, showing GPT's capability to suggest documentation updates based on recent commits.

Challenges and Limitations

• Tool Access: Limited access to authoring tools for direct integration into their system.
• Configuration Complexity: Ensuring correct API schema configuration for GitLab and GitHub proved challenging.
• Subscription Requirement: The need for a paid subscription to ChatGPT or similar services.
• Context Memory Limitations: Potential issues with handling large volumes of commits and data within GPT's memory constraints.
• Data Security Concerns: Risks associated with data security, particularly when making GPTs publicly accessible.
   

Potential Improvements

• Automated Retrieval: Developing a Python app using OpenAI’s API to automate the retrieval of the latest help files.
• Third-party Services: Exploring tools like Zapier Central for similar functionalities.
• Enhanced Capabilities: Investigating if GPTs can not only recommend but also implement changes directly, while managing context memory and content quality.

Future of AI in Technical Writing

• Efficiency Enhancement: AI can be a valuable tool for making technical writers more efficient and improving output quality.
• Current Usage: Early adopters are primarily using AI for idea generation, planning, and performing monotonous tasks.
• Long-term Role: AI will likely augment rather than replace human technical writers, with a continued need for human oversight to ensure accuracy and quality.
   

Questions and Insights

• AI Integration in Technical Writing: Ellis emphasized the role of AI in enhancing efficiency and output quality for technical writers.
• Adoption and Frontline Usage: He noted that while there is growing interest, most organizations are in the early stages of adopting AI tools for technical writing, focusing on backend tasks rather than customer-facing solutions.
• Accuracy and Reliability: The discussion included concerns about AI accuracy and the potential impact of errors on support lines and legal risks. Strategies for mitigating errors include prefiltering AI-generated answers to improve reliability.