A Scandinavian bank evaluated 23 developer portal solutions, and selected Pronovix. We had the chance to observe the Company’s mature approach to vendor assessment, and learn how they answered questions like: how can you convince management about your decision? What are the complex needs? What should the selection process be like?
Since this can serve as a best practice for others—and we’re honored to be ranked at the top—we sought approval to share the details behind the selection process.
Summary
One of the largest Scandinavian banks (referred to here as the Company) set out to review existing developer portal solutions on the market. Their goal was to enhance internal processes and prevent issues like API duplication caused by inadequate search functionality.
Using scientific methods and well-defined requirements, the buyer team identified 23 potential vendors. They then narrowed the list, ranked each option, weighed the pros and cons, and ultimately selected Pronovix.
‘What the big difference between you and number two was, was that your product is much more lightweight. (...) It has a very flexible meta model that can easily be set to structure APIs based on Products or to have a flat APIs-only structure (...) the navigation is very intuitive to work with, which makes it simple to get started as an end-user.’ - Chief Software Architect at the Company.
The buyer team used an innovative and informative method. It highlights the cornerstones of choosing from a remarkably large list of vendors and the key factors for making an informed decision. To make the ulitmate choice, it was essential to discuss internally:
- What are the requirements?
- Which ones are the must-haves?
- How to perform and evaluate the proof of concept and the demo?
- How do you justify your decision to management?
The Needs: Flexibility and Findability
The Company had specific requirements for an API developer portal, including security and GDPR compliance. They wanted a solution that was easy to manage and had good search capabilities so that developers could easily find the APIs they needed. Additionally, they wanted a lightweight product that was flexible and user-friendly.
A Scientific Method to Choose a Developer portal Solution
Bias, Notion, and Decision Hygiene
To set up a working process, one of the buyer team's key sources was Noise: A Flaw in Human Judgment (2021), a book by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein. As the authors state, we need to be aware of noise and bias when we make decisions.
- Noise: inconsistency in judgments. It happens when different people (or the same person at different times) make inconsistent decisions under similar conditions.
- Bias: systematic error, it occurs when judgments consistently deviate in the same way (e.g., toward a particular viewpoint).
To reduce these factors, the book's authors argue for a method called decision hygiene, which was an important base for the buyer team.
Decision hygiene is a systematic approach that helps bring structure to decision-making and prevents errors before they occur. It includes: you gather inputs from multiple people, evaluate smaller pieces of information, apply rules and algorithms when it is possible, and delay intuition.
As we learned from the Chief Software Architect at the Company, this approach guided them to make a confident and suitable decision from an extensive list of vendors.
The Process: RFI, Demo, POC, and the Concept of CMMI and DAR
The Research
At the outset, they conducted an analysis with input from a diverse group of end-users, including Business Analysts and Software Engineers. This helped them develop a fundamental understanding of their needs. Throughout the evaluation process, the team also had access to an additional Software Architect for consultation.
Using these insights, the buyer team began preliminary research to identify potential vendors. Once they had a shortlist, they investigated whether these vendors specialized in API developer portals and could meet the Company’s specific requirements. Their initial list included 23 potential vendors.
The Evaluation System
The company prepared a Request For Information (RFI) with questions based on the requirements, including security and General Data Protection Regulation (GDPR) compliance, and sent it to the vendors.
Next, they applied a scoring system to evaluate each vendor’s responses individually. Their process was inspired by Decision Analysis and Resolution (DAR) from the Capability Maturity Model Integration (CMMI) framework, as well as the Decision Hygiene method.
The DAR approach allows different weights to be assigned to various requirements before calculating overall scores, ensuring a structured evaluation. Meanwhile, the Decision Hygiene method helps eliminate bias and noise, leading to a more objective selection process.
In short, a weight was assigned to each question based on their importance. Team members scored separately and then collaboratively compiled pros and cons for the top ranked vendors for management to justify why one vendor is more suitable than the others.
This allowed the company to narrow down the list to the top five contenders for further evaluation.
The demo and the Proof of Concept
After assessing which vendors met their requirements on paper, the Company requested demos to evaluate whether the solutions addressed key use cases and to assess the overall user experience and design.
To ensure a fair comparison, all vendors followed the same agenda, use cases, and demo length, keeping the evaluations within a predefined scope. After another round of scoring, the Company invited the top two vendors to provide a proof of concept (POC).
At this stage, the Company defined two user personas for testing: the end-users who would use the platform daily and the platform owners responsible for maintaining it. They created new use cases, building on the previous ones, which focused more technically on certain aspects and targeted one of the two personas.
At Pronovix, we strongly advocate for working with user personas from the start of a project. With user personas in our hands, we can make sure and/or check if a developer portal will actually serve those who will use it and/or have expectations of it. Personas help to figure out actual or possible pain points and bump into opportunities.
After conducting the tests, the results were documented and presented to the team, who scored the outcomes. They gave additional weight to areas that had been thoroughly tested, along with an overall product score. This final score was averaged with the previous scores and combined with pricing, pros, and cons to recommend a vendor to upper management.
Before presenting the recommendation to upper management, the buyer team first shared the results, with Pronovix as the selected vendor, with the enterprise architecture decision board, which approved the choice. This approval was then included in the presentation to upper management.
The Buyer Team
The evaluation process involved a Product Owner, Software Engineers (both back-end and front-end), Business Analysts, and Software Architects from the Company. Their combined expertise ensured the team considered all relevant aspects of the decision.
Together, they worked to provide management with a well-reasoned, data-driven explanation for choosing Pronovix, including a detailed list of pros and cons to support their recommendation.
The Result
Management approved the selection process due to its thoroughness, objectivity, and professionalism. They recognized the value of having a centralized API developer portal to streamline internal processes, prevent API duplication, and help developers easily find the APIs they need.
After evaluating the requirements and conducting the assessment, the Company chose Pronovix. According to the buyer team, Pronovix’s solution is well-suited to achieving their goals.
The Chief Software Architect also emphasized that Pronovix’s solution is user-friendly and significantly enhances the efficiency with which developers can locate APIs.
Are you interested in our solution and want to discuss how we can help with your API/integration program? We can also help with standalone UX and Technical Writing audits.
