BangDB Manual - version 1.5 (current)

Welcome to the BangDB Manual!

BangDB is a multi-flavored distributed key value store available under BSD license for free download and usage. Please see the license section for detailed information on free and licensed version of the database or contact us for details

The goals of the manual are following;

  • Introduce BangDB as a nosql database
  • Explain the concept behind the design and implementation
  • Showcase sample application code using BangDB for various reasons
  • Describe the API, query method and other operations using BangDB
  • Explain the various configuration parameters and tuning methods for best results
  • Discuss the administrative aspect of the database
  • Be the through reference guide for delevelopment using BangDB

BangDB 1.5.2 (embedded) and 0.9 (server)

The BangDB 1.5.2 and 0.9 is a feature rich release with better db efficiency from the usage perspective and also from the core db performance. It's a robust and stable version with greater query support using more control available to the user.The write ahead log, transaction, buffer pool, high concurrency, multi access/index support are some of the features that BangDB implements in order to provide traditional database robustness for a very high performance nosql db. Please see the overview section to get more details on the BangDB

Following are the highlights of the current release

  • Multi Table Types - Normal Table, Wide Table, Primitive Table
    • Normal Table - key value store with value as opaque entity. Key can be string or opaque and value may again be string or opaque. In earlier version only this type of table was supported.
    • Wide Table - Here value can be bit structured which means that db can do bit more than just storing the value. Currently data can be simple or json string. which means user would be able to create index. For string data, user should specify the substring on which index can be created and for json data user may create index on different fields (nested as well) and db would then accordingly store the data. This allows user to create various indexes for better and efficient data retrieval. In next release column family would also be supported here where user may create different columns and store them as value
    • Primitive Table - This is special table type mainly to address particular set of use case where value is of fixed size. Note that BangDB asks for key size upfront hence size of it is always fixed (or variable length but less than the supplied max key size). But value could be of variable length and hence different ways are required to store the value. But when user wishes to store int or long or datatime or string with constant size then it would be much efficient to treat these values differently for less resource consumption and better performance. Primitive table comes handy in these case as user can just create table suitable for the value data type
  • Multi Indexing
    • Index - BangDB allows user to create index in unstructured data (like opaque value or string) or structured data(json data). With unstructured data user needs to tell the index name and value for each put, whereas for structured data (json data) user can just create index on field and db implicitly will add index for every put
    • Multi Index - With BangDB, user may create as many indexes as required
    • Nested Index - For json data, user may create nested index
    • Retrieve data using index - With BangDB user can get data for a given index name and value. Get returns ResultSet for such get and user may get as many ResultSets as needed and then do various operations on the ResultSets (ex; add, append, intersect etc...)
    • The index can be created using Btree or ExtHash(primary only) methods depending upon various need
    • User can create index or drop index whenever they like using the simple API
  • json data support
    • BangDB 1.5 has native support for json data. It has parser and set of APIs to deal with json data. However in the current release the json data parser is not exposed hence user will need to use some other parser to deal with json data in thier app but db would use its own json parser for processing the data>
  • Composite key support, index on composite keys
    • BangDB allows user to create composite key using two or more keys(fields in json doc) and store data for the key. This helps in scenarios where we need to define the composite key as unique even though the individual keys are non unique. BangDB searched the documents when scanned according to the individual part of the composite key
  • Nested Index
    • For json data, user should be able to define nested index and add data accordingly. The key can be arbitrarily nested to any length as desired and BangDB will be able to create and add index along with the data as required. While scan, BangDB would do proper things to return the resultset based on the filter defined for the nested key
  • Native support for opaque, string, int, long data type (for better performance for native data types)
    • BangDB now can create tables suitable for int/long such that the overall effectiveness is optimized. Also it allows user to define key type separately even for non native tables (normal tables). For ex; we can tell BangDB that the table is going to habdle long key and not opaque or string key, this helps db in optimizing the resources, consuming less space and also performing better for put and get
  • Built in abstraction for RealTime data analysis
    • Generic Approach - With the support of json data and multi indexes with primitive table types, BangDB becomes attractive option for doing data analysis. With the given high performance and various configurable parameters, BangDB can be used totally as in memory data base or a persistent store backed by the disk/ssd. User will create the data model and then store the data accordingly and finally retrieve data with some post processing to get the insight
    • Built in abstraction for different kinds of query - With 1.5 release, BangDB provides built in abstraction for handling few important types of data analysis which can be simply used with the get/set kind of API. User will not have to bother about the details of the data storage, management and query etc.. but simply create the object of the apprpriate type and start using API for putting the data and getting the result of the query.
    • Since the abstraction is fully baked up for given analysis type, hence it stores data in such a fashion which doesn't require any post processing and returns the stored data directly to the query. This makes it very efficient and fast
    • Supported abstractions are for following types of analysis;
      • Counting - Absolute, Probablistic, Unique, Non-unique
      • topk - (query top k items group by 'field')
      • Sliding Window - spanned and slotted, handle huge amount of data in sliding window using humble commodity machine
  • BangDB now installs several signal handlers to handle the db/app/machine crash or any other shutdown gracefully
    • Various signal handlers are added to the db. This by default is not enabled but it can be enabled (or later disabled) by setting the config param BANGDB_SIGNAL_HANDLER_STATE in the bangdb.config file. When enabled, it ensures that various log files are flsuhed to disk in the case of process crash or ther events when program has to quit.
  • App Logger
    • App logger is a handy tool for user to log application related info using BangDB logger. The logger is supported natively by the BangDB which provides high performance and durability of the app log. BangDB also logs its own error messages using the logger. However, user can use the config param (BANGDB_APP_LOG) to set logging using the logger (BANGDB_APP_LOG = 2) or to syslog(for linux)(BANGDB_APP_LOG = 1) or simply flush to stdout/stderr (BANGDB_APP_LOG = 0)
  • Transaction changes
    • More efficient - Few changes and fixes have been done to make transaction lot more efficient and consumes less resources
    • For different log types - Transaction will now internally support operations involving multiple tables with different types of log(shared or private). User will not have to worry about these details anymore
  • Memory usage optimization
    • DB would now be able to use user defined pre allocated memory for various db operations. This allows db to operate with very little memory creation dynamically and for many cases it doesn't at all
    • Scan now use user allocated memory as cursor for returning results, making scan as highly optimized from memory overhead perspective
  • Re-designed duplicate key index
    • The duplicate key index has now new structure which allows user to gain very high performance while update/put and get in concurrent scenario. The redesigned sturtcure allows index keys space to be very very small compared top earlier design and at the same time makes much less access to file system for regular IO
    • The user will now be able to create unique or non-unique indexes (secondaries) quite efficiently and with lot more expected performance
  • Supported Platforms
    • BangDB Embedded 1.5.2
      • C++ on Linux and Windows
      • Java on Linux and Windows
      • C# on Windows
      • 32 and 64 bit
    • BangDB Server 0.9
      • Server in C++ on Linux
      • Client in C++ on Linux and Windows
      • 32 and 64 bits
  • Bug fixes

Additional Resources

Check out the resources details and download white papers on various topics

If you are keen on checking out BangDB now, go ahead to the section of getting started

This manual is work in progress and we are in process of adding much more resources here which will help in understanding BangDB and developing apps using it