Why a modern data processing platform must have Stream, AI, and Graph at the core

Modern use cases are demanding modern ways to deal with data. These use cases have a lot in common. For example, they all have fast-moving data at the core, mostly coming from devices. They all need modern processing data in a lot more contextual and predictive manner which will require data to be linked in graph structure coupled with many AI models working together. The era of assembling a few pieces from open source to build such systems is gone already, most of the tools and systems in the market were created decades ago. Traditional architecture is failing to cope with the requirements increasingly. This is a great time to come up with the architecture and system which is designed from the ground up to address such problems. BangDB is a converged data platform that natively integrates AI, stream, and Graph processing within the NoSQL database.

Please see this blog which covers some of these aspects as well in a nice manner

Let’s look at the simple fact. In just a few years, the majority of the data will be generated by the devices. In less than a decade, device data will grow from being less than 2% to over 50% of the global data sphere [ Ref: IDC]. This change has been extremely rapid compared to the natural growth of the other systems and tools/technologies in the market. This has created an impedance mismatch and a huge gap when it comes to tackling the emerging or even the traditional use cases in the modern context.

Data will mostly originate from devices in just a few years, which means device data will override the use cases

By nature, these data move fast and carry lots of valuable insights which should be mined as soon as possible for higher value extraction. Graph structure gives these data much-needed context which could be leveraged in an absolute or probabilistic manner. Native AI provides real-time prediction support for various streaming events. And, since data may have many different formats, therefore, we must have a mechanism in place to deal with unstructured data in a scalable manner.

Some of the use cases for modern data processing would be.

  1. Smart device data analysis for thousands to millions of such devices in a real-time and predictive manner for operational efficiency by harnessing local / edge data
  2. Vehicle sensor data analysis for finding or predicting anomalies, interesting patterns for safety, security, and maintenance of the vehicle
  3. Satellite image analysis for finding the signature of interest. UAV, SAR type image analysis for finding changes in topologies of any given area, from a security, agriculture, or climate analysis point of view
  4. Operational efficiency for Integrated large-scale system using various logs, sensors, devices, apps, systems, services, etc. data, all at the same time

All the above use cases involve a plethora of log/data streaming into a single system (for linking, correlation). They also require real-time stream and Graph processing and analysis. Building several machine learning models for online predictions and mechanisms to take action. On top of it, we must do all these with sub-second latency as these data have quickly diminishing intelligence value.

What are the problems with the existing systems/ architecture?

  1. There is a lack of convergence at the system level. If you see it from a high level, the problem is posing a challenge of convergence, by bringing too many disparate requirements to the same place. Therefore, from a solution perspective as well, we must converge to offset the problems and their challenges. Such as, we must natively integrate modern data procssing like Graph, Stream processing, AI, and unstructured data handling within the data platform. Whereas the reality is that we often deal with several silos instead of one unified platform
  2. Stitching too many tools or systems may not be effective. First, it may take several human resources for quarters to years to build such a system. Then enabling use cases on top of it may not be very effective. Further, such a system may not fulfill all the basic requirements. For example, copying data across multiple sub-systems, and the network overhead of bouncing packets from one place to another would simply make the latency unacceptable. Such systems would enforce many different constraints from different dimensions, for example, data could be kept in memory which will either make the proposition brittle and/or very costly
  3. True edge computing is required for most of these use cases. Which tells us that we must embed part of the system within the device itself. What it means is that we must have a scenario where part of the same system is deployed within the device (ARM processors) in an embedded manner where it can deal with hyper-local data analysis. The same system is deployed on the LAN or Cloud (or both) which is interconnected with all the embedded deployments within several devices. This forms a hybrid compute network which is cooperating, shares different responsibilities, and collaborates to achieve a single goal (or set of goals). Most of the existing systems or platforms in the area would not be able to do so, thereby defeating the basic need of the use cases

How does BangDB fit into such a scenario?

BangDB is a converged database that has natively implemented following

  • Modern Data Processing
  • Stream Processing
  • Graph Processing
  • Machine Learning
  • Multi-model support


Moving away from n-tier to space-based architecture is best suited for future anonymous and distributed computing

From an architectural point of view, BangDB follows the convergence model which means instead of splitting the system into multiple pieces and scaling different pieces separately, it follows space-based architecture where all necessary pieces are always in a single space, and we scale the spaces as we need. In other words, instead of 3-tier or n-tier architecture, in space-based architecture, we always deal with several units of computing (or machines) where each unit of computing contains all necessary components.

This avoids extra copy of data, network hops, distribution overhead for splitting requests, and combining responses which result in higher latency, computational overhead, and complex management apart from high cost and resource guzzling procedure. True convergence allows us to scale the system linearly without much overhead along with hyper-fast processing speed and ease of development and management.

BangDB is a full-fledged database as well, it implements some of the advanced features of a mature database like transaction, write-ahead log, buffer pool/page cache, indexes, persistence, rich query support, etc. BangDB is available free of cost, download it and start building modern apps with ease

Predictive real-time data platform to boost e-commerce sales

E-commerce business needs to collect data from various sources, analyze them in real-time and gain insights to understand the visitor’s behavior and patterns which will allow the company to serve the customers in contextual and better ways to improve the conversion rate. A real-time data platform is needed of the hour which can combine stream analytics with Graph and AI to enable predictive analytics for better personalization for the users which significantly improves the e-commerce sales by 2X or even more. Therefore predictive real-time data platform is needed to boost the e-commerce sales

A real-time and predictive data platform for boosting e-commerce sales by visitor analysis is a need of the hour

Read a good article on this here to get more info about it

Some of the general facts (statistical) which relate to e-commerce sales are following

  • Based on survey reports, 45% of online shoppers are more likely to shop on a site that offers personalized content/recommendations
  • According to a report by Gartner, personalization helps in increasing the profits by 15%
  • The majority of the data (more than 60%) is not even captured and analyzed for visitor or customer analytics
  • Less than 20% of data is captured in true real-time, which diminishes the potential of relevant and contextual personalized engagement with the visitors and hence leads to scoring as well

To boost sales, e-commerce is looking to answer some of the following questions in general

  • How to develop personalized real-time engagement and messages, content
  • How to engage with the visitors and customers on a 1-on-1 basis for higher CR
  • How to identify and leverage purchasing patterns
  • What the entire consumer cycle looks like
  • Ways to improve promotional initiatives
  • How to make the customer and the customer experience the focus of marketing strategies – better lead score and reasons for the score
  • How to identify your customers’ switches between each channel and connect their movements and interactions

The businesses typically seek to predict the following for predictive analysis

  • Personalized content in a 1-0n-1 manner for better next steps or conversion
  • Exactly which customers are likely to return, their LTVs
  • After what length of time, they are likely to make the purchase
  • What other products these customers may also buy at that time
  • How often they will make repeat purchases of those refills

What are some common challenges e-commerce businesses face?

  • Understanding who is “Visitor” and “Potential Buyer”
  • Relationships between different entities and the context
  • Nurturing the existing prospects
  • Personalization
  • Calculating the Lifetime Value
  • Understanding the buyers’ behavior
  • Cart Abandonment
  • Customer Churn

So, how can e-commerce businesses tackle the above challenges?

Predictive Analytics encloses a variety of techniques from data mining, predictive modeling, and machine learning to analyze current and historical data and make predictions about future events and boost thier e-commerce sales.

With Predictive analytics, e-commerce businesses can do the following

  • Improve Lead scoring
  • Increase e-commerce sales
  • Increase Customer retention rates
  • Provide personalized campaigns for each customer
  • Accurately predict and increase CLV
  • Utilize Behavioral Analytics to analyze buyers’ behavior
  • Reduce cart abandonment rates
  • Use Pattern recognition to take actions that prevent Customer Churn

Following is a brief list of use cases that can be enabled on BangDB

A. Real-time visitor scoring for personalization and lead generation for higher conversion

  1. Predictive real-time visitor behavior analysis and scoring for personalized offering/targeting for a much-improved conversion rate. The potential increase in CR or expected biz impact could be 2X or more if implemented and scaled well
  2. Faster, contextual, and more relevant lead generation for higher conversion
  3. Personalized content, offerings, pricings, for visitors 1 on 1 basis, leads to much deeper engagement and higher conversion
  4. Projecting much relevant and usable LTV for every single user/visitor could lead to better decision making for personalization or targeting or offering
  5. Inventory prediction for different product/versions/offerings for better operation optimization

B. Improve engagement

  1. Personalized interaction and engagement with the customers
  2. Shopper’s Next Best Action
  3. Recommendations about relevant products based on shopping and viewing behavior
  4. Tailored website experience

C. Better target promotions

Collate data from other sources (demographics, market size, response rates, geography, etc.) and past campaigns to assess the potential success of the next campaign. Throw the right campaign to the right users

D. Optimized pricing

Predictive pricing analytics looks a historical product pricing, customer interest, competitor pricing, inventory, and margin targets to deliver optimal prices in real-time that deliver maximum profits. In Amazon’s marketplace, for example, sellers who use algorithmic pricing benefit from better visibility, e-commerce sales, and customer feedback.

E. Predictive inventory management

Being overstocked and out of stock has forever been a problem for retailers but predictive analytics allows for smarter inventory management. Sophisticated solutions can take into account existing promotions, markdowns, and allocation between multiple stores to deliver accurate forecasts about demand and allow retailers to allocate the right products to the right place and allocate funds to the most desirable products with the greatest profit potential.

F. Prompt interactive shopping

Interactive shopping aims for customer loyalty. Integration of an online logistics platform helps maintain end-to-end visibility of purchases and orders, and business intelligence software helps process customer transaction data. It also enables retailers to offer multiple delivery options and can prompt customers for additional purchases based on their buying patterns. Consistent customer service, coupled with technology, can greatly increase customer reliability.

Data mining software enables businesses to be more proactive and make knowledge-driven decisions by harnessing automated and prospective data. It helps retailers understand the needs and interests of their customers in real-time. Further, it identifies customer keywords, which can be analyzed to identify potential areas of investment and cost-cutting.

Challenges and Gaps in the market


  1. Need to capture all kinds of data, across multiple channels, not just a limited set of data
  2. Need to capture all data truly in a seamless and real-time manner
  3. Store different entities and their relationships in a graph structure and allow rich queries
  4. Need to auto-refresh and retrain the scoring model for relevant and higher efficacy
  5. Need to scale for high speed, high volume of data across multiple levels/ channels
  6. Need to have full control over the deployment and data
  7. Need to have the ability to add and extend the solution easily and speedily in different contexts or domains as required

Gaps with the existing systems in the market

  1. The majority of systems (GA, Omniture, etc.) can ingest a limited set of data. It’s virtually impossible to ingest other related data into the system for a better scoring model. Also, with these systems, it’s difficult to extend the ingestion mechanism for a custom set of data, coming from totally different data sources than just the clickstream. Therefore, there is a need for a system that can ingest heterogenous custom data along with typical CS data for better results and higher efficiency
  2. Most of the systems ingest data with latency not acceptable from the solution perspective. Forex; GA allows a limited set of data ingestion in real-time, the majority of data come with high latency. Omniture also has latency which is not acceptable to certain scenarios for the use cases.  Therefore, there is a need for true real-time data ingestion and processing system/platform
  3. All the systems come with the pre-loaded model(s) which are trained outside the actual processing system. This is hugely limiting from the AutoML perspective where the models could be trained and improved as it ingests more and more data. Also, finding the efficacy of the model is limiting which may result in poor and non-relevant prediction. Therefore, there is a need to have an AI system natively integrated with the analytic scoring platform
  4. As we wish to deploy the system for various locales, different verticals, websites, or companies, the system must scale well. The speed and volume of data coupled with model preparation, training, deployment, etc. make it very difficult for such a system to scale well. It takes many weeks and months just to prepare and integrate the system with the new set of data sources. Software deployments, library configurations, infrastructure provisioning, training and testing of models, versioning of models, and other large files, all of these create a huge block in terms of scaling the system. Therefore, there is a need to have a platform that hides all these complexities and provides a simple mechanism to scale linearly for a higher volume of data, more num of websites, locales, or simply for a larger number of use cases as things move forward.
  5. Most of the system acts as a black box allowing lesser control on deployment and access to data in a larger sense. This results in brittle solutioning and faster development of use cases. Better access to
  6. Most of the systems in the market won’t have “stream”, “graph”, “ML” and “NoSQL” in the same place. Integration takes lots of time, resources and is sometimes not feasible at all
  7. Also, it provides huge restrictions in terms of dealing with ML since the models and their metadata are often abstracted. More often than not, we might need to upload pre-baked models or model creation logic or file to leverage existing code. Therefore, we need a system that allows us to have greater control of various processes along with the ability to reuse and extend already existing knowledge and artifacts

BangDB’s offering

BangDB platform is designed and developed to address the above gaps and challenges.

  1. Captures all kinds of data for visitors
  2. Clickstream, pixel data, tags, etc.
  3. Website specific data
  4. Any other data that may be useful/required
  5. Existing data
  6. Retailers’ data, external data
  7. Any other infrastructure or system data as required

Captures all data in real-time

Captures all data in real-time, as opposed to GS which captures only a small fraction of data in real-time. This limits the scoring efficacy as real-time data is the basis for proper analysis. Omniture captures most of the data, but they are available for analysis in a few minutes rather than in a few milliseconds. Proper personalization or any action is best taken as soon as possible, not after a few minutes or hours

Accelerated time to market

BangDB comes with a platform along with a ready solution that implements the use cases as needed and has the majority of the plumbing in place. Further, it has the built-in KPIs, models, actions, visualizations, etc. which are ready from day 1. We need to just configure the system, add more data points, fix the API hooks, etc., set the model training/retraining processes which are in contrast with many other systems where they may take several weeks or even months to just get started

Scales well across multiple dimensions, in a simple manner

Several IPs for high performance and cost-effective methods to deal with a high volume of fast-moving data. The platform has a built-in IO layer for improved, flexible, and faster data processing. Convergence allows the system to scale linearly as required in an uninterrupted manner

  • Integrated streaming system to ingest all kinds of data as required for analysis in real-time. Build your apps/solutions or extend the existing ones as needed by just using the UI and not doing coding etc.
  • Integrated machine learning system for faster, simpler, and automated model training, testing, versioning, and deployment

The platform comes with AI natively integrated, which allows us to get the models trained and retrained frequently as more and more data arrives. It starts producing output from the model within a week and as it moves forward it keeps improving the model and its efficacy. It also measures its efficacy and tunes/retunes as needed for higher performance.

Install BangDB today and get started

To check out BangDB, go ahead download it for free

Quickly get started here

Checkout the BangDB benchmark