Web application event streaming makes real-time marketing a reality. With it we can react to specific e-commerce consumer behaviours and conditions with instant marketing campaigns and sales opportunities.
Imagine offering promotions direct to specific customers, at the exact time and place of sale, from within your e-commerce platform’s user interface. That’s real-time marketing. It implies knowing what your customers are doing, not just recently, but right now, at this precise moment.
This is what one of our clients wants to do. BlueSoft built for them a proof-of-concept (PoC), quickly, simply, and cheaply, using Apache’s Kafka, Druid, and SuperSet, and some custom Node.js code. The PoC turns our client’s unformed idea into a working model. It helps them clarify their vision, understand their needs, and test their idea’s true potential.
This PoC’s combination of technology is not unique or special. BlueSoft’s achievement is in bringing an idea to life, in making it real.
Web application event streaming
Event streaming is an approach to application telemetry analytics. It is used not only in marketing but also in fields such as UI/UX testing, behavioural science, and supply chain management.
When our client asked us to develop a PoC of their outline specification, we did so in days. We leveraged the awesomeness of Apache Kafka, Druid, and SuperSet, and our in-house skills in rapid web application development.
Our client sells products via its own e-commerce platform, which its customers use for:
- inventory and stock ordering;
- invoicing and payments processing;
- tech specs and docs;
- live support.
They also use it for marketing, and it’s this aspect our client wants to explore.
Why not use Google analytics?
Before I show you the PoC system architecture, you might be asking, Why not just use Google analytics?
In fact, that is what our client uses now, but there are problems. The data is:
- slow—by the time it arrives, any opportunity of acting on it is lost;
- chunky—Google aggregates the raw data, and access to it is…
- expensive—you must pay to see the detailed data.
The custom part of the PoC uses web application event streaming to capture browser events. That means every move, click, hover, or key press, emitted by a web application when a user interacts with it. The data is streamed continuously, so there’s a lot of it. I can summarize the challenges as:
- Efficiency—not caching click data but registering and forwarding it immediately, without affecting the performance of the web application;
- Velocity and volume—storing vast amounts of data (possibly in the range of 100’s of billions of records per day), and doing it quickly (so-called fast ingest); and,
- Visualizing, or interpreting the data—to get an instant insight into it, one that leads to a clear course of action.
Web application event streaming architecture
The PoC solution architecture looks like this.
- The E-commerce platform user interacts with a demo Web Application that emulates the client’s e-commerce app.
- Our custom Node.js-based Click Collector streams every event to…
- an Apache Kafka queue, from where it is fed into…
- an Apache Druid database, which becomes the source for…
- Apache SuperSet, which visualizes the live data for the Analyst.
I’ve yet to see which direction the client wants to go with this, and besides, there will be more to think about:
- Privacy and security, remembering the goal is to help end users with marketing, not to monitor or track them, or allow anyone else to do so.
- Scalability, understanding that much of the unique benefit of this technique comes from the speed of collection and the volume of collected data.
- Insight, ensuring the right data is collected, that it yields a true picture of the end user’s market, and that the actions (the ‘real-time marketing’) really work.
About the author
Maciej Buczek is an IT Manager in BlueSoft’s Banking & Capital Markets & Automotive division.