|
kvs is a key-value datastore based on bplus trees, on-disk, thread-safe datastore. Kvs is not ORM or structured around Sql but a direct access to the bplus tree and its mapped data on the file system. It supports basic INSERT, GET, DELETE, QUERY and more complex relations need to programmed. kvs.jar a single java library with no third party dependencies, it to your class path to use it. Disclaimer: create regular backups and use it with caution, any unusual use may result in errors or data loss.
Last updated: 22.11.2023
kvs-1.0.5.beta.r3.jar sha256: e097662988fe87c0d6a91c2bda05197bf3f34aabf1ae53193b11fb4dbae122a1
kvs datastore - ex_kvs.zip.
kvs datastore - ex_KeyValue.zip.
kvs datastore with Tomcat - ex_servlet_kvs_with_Tomcat.zip.
First released: 19.06.2021
Quick start notes:
Set datastore path.
Create datastore instance.
InstanceConfig cfg = new InstanceConfig(PATH, DB_INSTANCE, InstanceConfig.RW);
Datastore datastore = Datastore.instance(cfg);
for example:
InstanceConfig cfg = new InstanceConfig("f:/data/", 501, InstanceConfig.RWD);
Datastore datastore = Datastore.instance(cfg);
will create this path for this instance:
f:/data/datastore_501/
n is between 0 and 999, you can have up to 1000 instances, independent datastores, organized under the datastore path:
DATASTORE_PATH\datastore_000
DATASTORE_PATH\datastore_001
DATASTORE_PATH\datastore_002
...
DATASTORE_PATH\datastore_999
// see java read/write options "rw" vs "rwd"
// InstanceConfig.RW in case of power loss, is most likely to loose the commits within the log file(rvloga)
// to recover delete rvloga
// InstanceConfig.RWD - default, more secure and slower forces data to be written on the device
See examples how to get, insert, delete and create queries.
Entities
Key parent = KeyFactory.create_key(key_kind, name);
Entity entity = new Entity(e_kind, name_id, parent);
datastore.insert(entity);
Key k = KeyFactory.create_key(kind, name_id);
Entity e = datastore.get(k);
datastore.delete(entity);
name_id is one of the two type Long or String.
There is an implicit index "NAME_ID" which is always build for all kinds, ancestors, and kindless ancestors,
and is used to build other indexes.
Note: Following symbols are reserved and cannot be used in names for entity name_id, kinds, keys, properties names etc.
(:) is used as a path separator in Keys
(,) is used to declare composite indexes
(|) is used as a separator for composite indexes
There is no internal check if you use illegal symbols for names, and using them may result in errors.
Currently the names are not encoded, which may change in later versions.
Do not use "NAME_ID", "ROOT" or any of the above symbols when naming entities, kinds, keys etc.
"ROOT" is the default kind of an entity created without kind.
Multi-value properties are not supported for indexing.
Note: Properties values of the entities are red on demand, when you call e.get_property_value(),
and there is a StoreContext attached to each entity. This means you cannot insert an entity from one datastore instance
into another directly. You need to create a copy of the entity and then insert it.
Properties
Properties types must be one of the Java types:
TYPE_BYTE;
TYPE_SHORT;
TYPE_INT;
TYPE_LONG;
TYPE_FLOAT;
TYPE_DOUBLE;
TYPE_STRING;
TYPE_BYTE_ARRAY;
Queries
try
{
String property_name = "NAME_ID";
Query q = new Query(kind).addSort(property_name, SortDirection.DESCENDING);
PreparedQuery pq = datastore.prepare(q);
QueryResultIterator it = pq.asQueryResultIterator();
while(it.hasNext())
{
Entity entity = it.next();
}
} catch (DataStoreException e) {
e.printStackTrace();
}
Ranged Query. If the keys don't exists they are temporary inserted (in memory only) to mark the range.
Query q = new Query(kind).addSort(property_name, SortDirection.ASCENDING).addRange(from_key, to_key);
Declared indexes.
Declare the indexes within indexes.xml. You can change and rebuild indexes later. See Offline tools for rebuilding indexes.
Each index can be red in ASCENDING and DESCENDING order.
To declare composite index:
Create a Copy of the datastore;
Create copy instance of the datastore, recycles free space, removes entities changes history,
sorts btree blocks breadth-first in favor of queries.
Log file rvloga.
The changes after every operation (insert, delete) are committed in rvloga file.
Every commit represents a different snapshot of the datastore.
You can choose between auto flush, or manual flush, default is auto_flush=true with max_size=200MB.
The log will auto flush when exceeds the max_size;
If you choose manual flush you will have to lookup for the log file and flush it manually. Note: the log grows very fast.
In case of manual flush, you could backup the log files, before every flush, which will provide you with consistent restore points.
Note: auto flush may cause currently running queries to expire and transactions to fail. This is because ones the log is flushed,
the current snapshot within wich the query executes is lost.
Offline Tools
Offline is a set of unsynchronized methods that can be run only offline.
Running these methods while the datastore is in use may result in a data loss.
"D:\Program Files\Java\jdk1.8.0_40"\bin\java -classpath E:/g_server/kvstore/kvs-1.0.5.beta.jar com.aza.kvs.store.tools.Offline build-indexes f:/data/ 0
"D:\Program Files\Java\jdk1.8.0_40"\bin\java -classpath E:/g_server/kvstore/kvs-1.0.5.beta.jar com.aza.kvs.store.tools.Offline check-indexes f:/data/ 0
"D:\Program Files\Java\jdk1.8.0_40"\bin\java -classpath E:/g_server/kvstore/kvs-1.0.5.beta.jar com.aza.kvs.store.tools.Offline flush f:/data/ 0
"D:\Program Files\Java\jdk1.8.0_40"\bin\java -classpath E:/g_server/kvstore/kvs-1.0.5.beta.jar com.aza.kvs.store.tools.Offline datastore-copy f:/data/ 0 f:/data/ 501
KeyValue
Since 1.0.5 introduces new, fast KeyValue interface for mapping primitive types Long/Int to Long/Double/Int, etc,
as well as limited size Strings and ByteArrays. Indexes have ordered keys only. Nodes are limited node size to 4096,
see examples how to configure KeyValue index. For ordered values and keys, Entities should be used.
IdxConfig ib = new IdxConfig("idx_name", KvType.KEY_STRING, KvType.VALUE_STRING);
ib.set_factor(30).set_key_len(8).set_value_len(8);
boolean created = datastore.get_kv_controller().create_if_not_exist(ib);
KeyValue kv = datastore.get_kv_controller().get_index(index_name);
AKey k = kv.create_SKey("key1"); // max len 8 bytes
kv.insert_string(k, "value1");
kv.commit();
kvs stores Strings in UTF-8. UTF-8 uses 1-4 bytes per character. For ex.: for Cyrillic set length to (num_chars*2bytes).
All values take the whole length given in the IdxConfig. Thus, byte array values must be the same size as configured.
String also take the whole length, but are null-terminated.
KeyValue methods are synchronized, but indexes are cached and any thread may call get_index(index_name) and any
of its methods. In which case you may have to synchronize the access to concrete indexes.
KeyValue.clear() will clear uncommitted inserts/deletes. It may be good to call it before inserts, or in try finally
block.
Known bug: Query on empty index will throw RuntimeException.Fixed in r1;
Some orientational test results. Insert 100k in index declared as above:
- insert commit after all inserts: 0.2s
- update commit after all inserts: 1.1s
- insert/update commit after each insert: 3.2s
- query 0.06s
New in 1.0.5.r3
- KeyValue - max value length is 1200 bytes.
New in 1.0.5.r2
- added byte array key - KeyValue.create_BAKey() for UUID, see examples.
- added more controls over the types and lengths of keys and values.
New in 1.0.5.r1
- revision of KeyValue interface, changed the behaviour of reads get/query
see KeyValue.read_uncommited(), examples 19, 20.
- bug fix optimizations
New in 1.0.5
- new KeyValue interface
- bug fix optimizations
New in 1.0.4
- new copy method
- bug fix composite index interface
New in 1.0.3
- bug fix in query range.policy.inclusive/exclusive
New in 1.0.2
- optimized log file
- optimized data formats
- 1.0.2 will flush the log before start
|