The new experimentation platform provides better insight into what the customer wants, enables the optimization of their experience, and ultimately increases revenue.
Experimentation would, in turn, allow bol.com’s teams to develop new ideas, products, and services in a more hypothesis-driven way; that is, by thinking about them as a series of experiments, with measurable evidence and learning as a key outcome.
The new platform also needed to work for both technical and non-technical users, while deeply integrating into bol.com’s existing technology and way of working. The company needed a custom solution, rather than an “on-the-shelf” platform to allow for flexibility and ongoing improvements in bol.com’s approach to experimentation. The platform also needed to integrate bol.com’s existing analytics pipeline and allow for the introduction of advanced features in the future.
Lack of resources in the teams to increase experimentations
Building an experimentation platform
Build and integrate an experimentation tool into existing infrastructure
Xebia was asked to help bol.com with building this experimentation platform. The bol.com teams that were already capable of performing A/B-tests had created different tools and systems to support their experiments. Meanwhile, business users were recording their ideas and tracking the progress of their experiments in a separate administration system. A third system was being used to capture the analytics data of end users, which was then copied to another tool to determine its significance. Finally, these results were copied and archived for future reference to the previously mentioned administration system.
In addition to the analytics solution used in this process, bol.com had built a custom solution to gather click-stream data, called “Measuring2 (M2).” Although M2 offered much richer data, none of the existing experimentation approaches used it.
Bol.com wanted a single tool they could use to input and administer an idea, initiate an experiment, analyze results, and record the findings. The company also wanted to maintain control of its data and statistical analysis. The tool needed to integrate with bol.com’s various channels and devices, which meant an on-the-shelf solution would not suffice. Bol.com needed to build a custom experimentation tool that would allow them to embed validated learning throughout the entire company and all internal APIs. In this way, they could connect partner platforms with consumer-facing applications, such as the webshop and app.
"Xebia has helped us get off to a flying start with the development of our experimentation tool. With this tool, all bol.com users (and in the future partners)can do experiments independently, so that you can determine the pace at which you learn. Through a smart way of registering the experiments and the associated data, the system can automatically determine the results and possible interaction effects."
We set out to consolidate the existing experimentation system into a single application that both business and technical users could navigate. The new system would facilitate the administration, execution, and outcome analysis of all experiments and use the M2 analytics data as its clickstream data source.
To ensure that the bol.com developers could maintain the application for the long term, we aligned it with their existing knowledge, technology, and way of working. Then we integrated the application with the bol.com infra structure and handed it over to its developers.
The setup includes a CI/CD pipeline integrated with Spinnaker and Kubernetes, which removes all manual steps and speeds up the release process. The technology leverages open source contributions, and aligns the knowledge of bol.com developers, increasing development speed. We used Spring-Boot and Kotlin to build the experimentation service and implemented the communication administration tool as a React application written in Typescript. In addition to some bol.com components, the set up uses Material-UI. We thoroughly tested the app and automatically deployed reviewed code to production through a continuous deployment setup.
The bol.com webshop performance was a critical concern, so we had to make sure that experimentation would have no impact on its load speed. We used an approach called “in-memory bucketing,” which allocates new users to experiments without service calls. The experimentation service exposes its API via an OpenAPI schema to generate TypeScript definitions which are used to guarantee end-to-end type safety. This type safety speeds up development because it helps to identify errors before code reaches production. We performed an initial sanity test to test the viability of the experiment. Capturing production data early on gave the team and other stakeholders confidence in the solution. This so-called A/A-test divided the users into two groups and sent the allocation to M2, which allowed the team to quickly evaluate if the division was even and determine if there were anomalies in the data. Fortunately, the team did not find any irregularities.
Experimenting provides insight that bol.com can now use to increase its revenue, enabling teams to learn more and faster from user behavior, and in turn, improve conversion and user experience. Business and technical users can both use the same tool to run experiments and make decisions based on data rather than on “gut feeling.” The experimentation tool successfully integrated into the webshop and the backend of the mobile app, which uses the rich clickstream data from M2.
Selective tagging makes measuring experiment effects more accurate. Measurements of visitor sessions that did not see the experiment can be excluded from the result set. Previously it was only possible to tag a visit. With the new tool, it’s possible to tag page views, component groups, and single components. By only looking at the relevant observations, the difference between experiment groups becomes more apparent, so a more confident conclusion can be drawn more quickly.
The migration of the data of the existing experiment administration system is in progress. Some of the data is already available in the new experimentation tool. It will replace the current system as soon as enough data is migrated. No historical data will be lost, and there will be a single system. Experiments for the webshop as well as the native app are being developed using the new experimentation tool. Building the experiment itself requires work from multiple teams. Once the first experiments are developed it will be easy to add new experiments. An additional benefit of the experimentation tool is that it gives a user-friendly insight into the clickstream data. The existing users of the data were mostly technical. The experiment results show how often various elements have been shown and what the click rate is on an element is, given that it was viewed.
Xebia enables you to build top quality software that truly has an impact. Great software doesn’t just meet, but exceeds expectations of your increasingly critical users. It’s scalable, easy to maintain and quick to adapt to leverage new opportunities. With over 15 years of experience and well over 100 of the best engineers and consultants in the field of cutting-edge software development, Xebia is your logical partner to achieve this.