## Introduction Nowadays, there are many databases that are universal, efficient and having very friendly user interface. The examples are: PostgreSQL, MySQL, MongoDB, ClickHouse, etc. Each one has its own unique features and advantages regarding the others that is done by the algorithms and the concept. There is no "the best" database because what has advantage in one tasks usually lacks in the other ones, so each database can be "the best" in the use cases it was developed for. In this article, we are researching the database `Lbasedb` specially designed for working with huge datasets. Most famous databases are general purpose, that means they are applicable in a set of standard data processing like adding new information, updating it or deleting it, performing an efficient search, supporting flexible data types and so on. Obviously, they have performance limits that can be a problem in some projects where we require a too fast data fetching or storing too many records on a small disk. What if we sacrifice some general purpose requirements in order to boost some special operations that are typical for huge datasets? This is where the motivation of `Lbasedb` comes from. Datasets usually require: 1. Small size in the storage 2. Fast adding new records 3. Fast extracting a subset in a known range and listed columns 4. Fast bulk update 5. Fast and convenient record iterating (one by one or in chunks) 6. Flexible working with adding, renaming and deleting columns 7. Wide compatibility with data processing tools (like Machine Learning tools in Python) Comparing to general purposes, datasets rarely require searching for special records, deleting the data. In other words, datasets do not have to support `CRUD` entirely. Inspired by these ideas, technically `Lbasedb` has following features: 1. Column-oriented architecture, so working with a column does not impact on the others. 2. Lack of indexes, so add operation has the complexity `O(1)` (adding is fast and does not depend on the whole database size). 3. Lack of deleting, that makes selecting and updating subsets easier and faster. Instead of deleting, truncate operation is supported that may be helpful in some cases. 4. Limited datatypes support: int32, int64, float32, float64, raw bytes (any size), nesting is not added so the record structure is always flat. Since these datatypes are universal it can be easily integrated with many other tools for dataset processing or visualizing. Under the hood, the core of `Lbasedb` is written in Rust ([https://github.com/fomalhaut88/lbasedb](https://github.com/fomalhaut88/lbasedb)) making the access to disk with asynchronous `tokio` framework, that brings extremely high performance and simple integration of the pieces into other Rust projects as a common dependency ([https://crates.io/crates/lbasedb](https://crates.io/crates/lbasedb)). Also `Lbasedb` has REST API interface in JSON format and can be used as a Docker container or built from source as a Rust executable project ([https://github.com/fomalhaut88/lbasedb-api](https://github.com/fomalhaut88/lbasedb-api)). ## Installation 1. Pull the docker image: ``` docker pull fomalhaut88/lbasedb-api ``` 2. Create and run a container: ``` docker run \ -p 8080:8080 \ --restart=always \ --ulimit nofile=1024:1024 \ --volume /tmp/lbasedb:/app/db \ --name lbasedb-api-app \ --env WORKERS=16 \ -d fomalhaut88/lbasedb-api ``` After that following [http://localhost:8080/version](http://localhost:8080/version) will show you the version. For the API and environment details: [https://hub.docker.com/r/fomalhaut88/lbasedb-api](https://hub.docker.com/r/fomalhaut88/lbasedb-api) ## How to start ### 1. Create a feed Feed is a table in Lbasedb, it contains records and columns. In order to create one called "xyz" just perform `POST http://localhost:8080/feed` with the content `{"name": "xyz"}`. After that `GET http://localhost:8080/feed` will list the available feeds. ### 2. Create columns Once feed is created, you may need to have columns. To create a 32-bit integer column, perform `POST http://localhost:8080/col?feed=xyz` with the content `{"name": "x", "datatype": "Int32"}`. We also as an example add the columns "y" and "z" of the same type. So now `GET http://localhost:8080/col?feed=xyz` shows the columns you just created. ### 3. Add data To fill the feed with a data, perform `POST http://localhost:8080/data?feed=xyz` with the content `{"x": [1, 2, 3, 4, 5], "y": [6, 7, 8, 9, 10], "z": [11, 12, 13, 14, 15]}` (if there were byte datatype, you would need to set bytes as Base64 strings). So after that `GET http://localhost:8080/size?feed=xyz` will return 5. As example, if you need a subset of 3 records starting with second one (index 1) and columns "y" and "z", do `GET http://localhost:8080/data?feed=xyz&ix=1&size=3&col[]=y&col[]=z`. ### 4. Update data Lbasedb supports bulk update that means we can update a whole subset in the stored dataset. For example, we would like to update "x" and "z" only starting with index 2 with the subset `{"x": [100, 100], "z": [300, 300]}`, so all we need is to perform `PATCH http://localhost:8080/data?feed=xyz&ix=2` passing the update subset in the body. "y" will not change, but if you use `PUT` method instead of `PATCH` the columns not listed in the subset will be zeroed. ### Extra We just considered very basic things to create a database structure and simple commands to work with datasets. Lbasedb also supports renaming, size change (it truncates the feed) and working with raw bytes instead of JSON that may be useful in some performance cases to avoid data converting on the DBMS side. See the details in the documentation: [https://hub.docker.com/r/fomalhaut88/lbasedb-api](https://hub.docker.com/r/fomalhaut88/lbasedb-api). ## Recommendations Mainly, Lbasedb is recommended to use if you need to save a data stream that will not have deleted records and is supposed to be analyzed or visualized by third-party tools. Lbasedb is also flexible to add, rename and remove columns. But since Lbasedb is not a general purpose database, it may be not suitable in some other cases. For example, Lbasedb does not support searching for records by a column value at all.