Building & Embedding

Currently, libmdbx is only available in a source code form. Packages support for common Linux distributions is planned in the future, since release the version 1.0.

Source code embedding

libmdbx provides three official ways for integration in source code form:

Using an amalgamated source code which available in the releases section on GitFlic.

‍An amalgamated source code includes all files required to build and use libmdbx, but not for testing libmdbx itself. Beside the releases an amalgamated sources could be created any time from the original clone of git repository on Linux by executing make dist. As a result, the desired set of files will be formed in the dist subdirectory.
Using Conan Package Manager:
- optional: Setup your own conan-server;
- Create conan-package by conan create . inside the libmdbx' repo subdirectory;
- optional: Upload created recipe and/or package to the conan-server by ‘conan upload -r SERVER 'mdbx/*’`;
- Consume libmdbx-package from the local conan-cache or from conan-server in accordance with the Conan tutorial.
Adding the complete source code as a git submodule from the origin git repository on GitFlic.

‍This allows you to build as libmdbx and testing tool. On the other hand, this way requires you to pull git tags, and use C++11 compiler for test tool.

Please, avoid using any other techniques. Otherwise, at least don't ask for support and don't name such chimeras libmdbx.

Building and Testing

Both amalgamated and original source code provides build through the use CMake or GNU Make with bash.

All build ways are completely traditional and have minimal prerequirements like build-essential, i.e. the non-obsolete C/C++ compiler and a SDK for the target platform. Obviously you need building tools itself, i.e. git, cmake or GNU make with bash. For your convenience, make help and make options are also available for listing existing targets and build options respectively.

The only significant specificity is that git' tags are required to build from complete (not amalgamated) source codes. Executing git fetch --tags --force --prune is enough to get ones, and --unshallow or --update-shallow is required for shallow cloned case.

So just using CMake or GNU Make in your habitual manner and feel free to fill an issue or make pull request in the case something will be unexpected or broken down.

Testing

The amalgamated source code does not contain any tests for or several reasons. Please read the explanation and don't ask to alter this. So for testing libmdbx itself you need a full source code, i.e. the clone of a git repository, there is no option.

The full source code of libmdbx has a test subdirectory with minimalistic test "framework". Actually yonder is a source code of the mdbx_test – console utility which has a set of command-line options that allow construct and run a reasonable enough test scenarios. This test utility is intended for libmdbx's developers for testing library itself, but not for use by users. Therefore, only basic information is provided:

There are few CRUD-based test cases (hill, TTL, nested, append, jitter, etc), which can be combined to test the concurrent operations within shared database in a multi-processes environment. This is the basic test scenario.
The Makefile provide several self-described targets for testing: smoke, test, check, memcheck, test-valgrind, test-asan, test-leak, test-ubsan, cross-gcc, cross-qemu, gcc-analyzer, smoke-fault, smoke-singleprocess, test-singleprocess, long-test. Please run make --help if doubt.
In addition to the mdbx_test utility, there is the script stochastic.sh, which calls mdbx_test by going through set of modes and options, with gradually increasing the number of operations and the size of transactions. This script is used for mostly of all automatic testing, including Makefile targets and Continuous Integration.
Brief information of available command-line options is available by --help. However, you should dive into source code to get all, there is no option.

Anyway, no matter how thoroughly the libmdbx is tested, you should rely only on your own tests for a few reasons:

Mostly of all use cases are unique. So it is no warranty that your use case was properly tested, even the libmdbx's tests engages stochastic approach.
If there are problems, then your test on the one hand will help to verify whether you are using libmdbx correctly, on the other hand it will allow to reproduce the problem and insure against regression in a future.
Actually you should rely on than you checked by yourself or take a risk.

Common important details

Build reproducibility

By default libmdbx track build time via MDBX_BUILD_TIMESTAMP build option and macro. So for a reproducible builds you should predefine/override it to known fixed string value. For instance:

for reproducible build with make: make MDBX_BUILD_TIMESTAMP=unknown ...
or during configure by CMake: cmake -DMDBX_BUILD_TIMESTAMP:STRING=unknown ...

Of course, in addition to this, your toolchain must ensure the reproducibility of builds. For more information please refer to reproducible-builds.org.

Containers

There are no special traits nor quirks if you use libmdbx ONLY inside the single container. But in a cross-container cases or with a host-container(s) mix the two major things MUST be guaranteed:

Coherence of memory mapping content and unified page cache inside OS kernel for host and all container(s) operated with a DB. Basically this means must be only a single physical copy of each memory mapped DB' page in the system memory.
Uniqueness of PID values and/or a common space for ones:
- for POSIX systems: PID uniqueness for all processes operated with a DB. I.e. the --pid=host is required for run DB-aware processes inside Docker, either without host interaction a --pid=container:<name|id> with the same name/id.
- for non-POSIX (i.e. Windows) systems: inter-visibility of processes handles. I.e. the OpenProcess(SYNCHRONIZE, ..., PID) must return reasonable error, including ERROR_ACCESS_DENIED, but not the ERROR_INVALID_PARAMETER as for an invalid/non-existent PID.

DSO/DLL unloading and destructors of Thread-Local-Storage objects

When building libmdbx as a shared library or use static libmdbx as a part of another dynamic library, it is advisable to make sure that your system ensures the correctness of the call destructors of Thread-Local-Storage objects when unloading dynamic libraries.

If this is not the case, then unloading a dynamic-link library with libmdbx code inside, can result in either a resource leak or a crash due to calling destructors from an already unloaded DSO/DLL object. The problem can only manifest in a multithreaded application, which makes the unloading of shared dynamic libraries with libmdbx code inside, after using libmdbx. It is known that TLS-destructors are properly maintained in the following cases:

On all modern versions of Windows (Windows 7 and later).
On systems with the __cxa_thread_atexit_impl() function in the standard C library, including systems with GNU libc version 2.18 and later.
On systems with libpthread/ntpl from GNU libc with bug fixes #21031 and #21032, or where there are no similar bugs in the pthreads implementation.

Linux and other platforms with GNU Make

To build the library it is enough to execute make all in the directory of source code, and make check to execute the basic tests.

If the make installed on the system is not GNU Make, there will be a lot of errors from make when trying to build. In this case, perhaps you should use gmake instead of make, or even gnu-make, etc.

FreeBSD and related platforms

As a rule on BSD and it derivatives the default is to use Berkeley Make and Bash is not installed.

So you need to install the required components: GNU Make, Bash, C and C++ compilers compatible with GCC or CLANG. After that, to build the library, it is enough to execute gmake all (or make all) in the directory with source code, and gmake check (or make check) to run the basic tests.

Windows

For build libmdbx on Windows the original CMake and Microsoft Visual Studio 2019 are recommended. Please use the recent versions of CMake, Visual Studio and Windows SDK to avoid troubles with C11 support and alignas() feature.

For build by MinGW the 10.2 or recent version coupled with a modern CMake are required. So it is recommended to use chocolatey to install and/or update the ones.

Another ways to build is potentially possible but not supported and will not. The CMakeLists.txt or GNUMakefile scripts will probably need to be modified accordingly. Using other methods do not forget to add the ntdll.lib to linking.

It should be noted that in libmdbx was efforts to avoid runtime dependencies from CRT and other MSVC libraries. For this is enough to pass the -DMDBX_WITHOUT_MSVC_CRT:BOOL=ON option during configure by CMake.

To run the long stochastic test scenario, bash is required, and such testing is recommended with placing the test data on the RAM-disk.

Windows Subsystem for Linux

libmdbx could be used in WSL2 but NOT in WSL1 environment. This is a consequence of the fundamental shortcomings of WSL1 and cannot be fixed. To avoid data loss, libmdbx returns the ENOLCK (37, "No record locks available") error when opening the database in a WSL1 environment.

MacOS

Current native build tools for MacOS include GNU Make, CLANG and an outdated version of Bash. Therefore, to build the library, it is enough to run make all in the directory with source code, and run make check to execute the base tests. If something goes wrong, it is recommended to install Homebrew and try again.

To run the long stochastic test scenario, you will need to install the current (not outdated) version of Bash. To do this, I recommend that you install Homebrew and then execute brew install bash.

Android

I recommend using CMake to build libmdbx for Android. Please refer to the official guide.

iOS

To build libmdbx for iOS, I recommend using CMake with the "toolchain file" from the ios-cmake project.

Getting started

‍This section is based on Bert Hubert's intro "LMDB Semantics", with edits reflecting the improvements and enhancements were made in MDBX. See Bert Hubert's original.

Everything starts with an environment, created by mdbx_env_create(). Once created, this environment must also be opened with mdbx_env_open(), and after use be closed by mdbx_env_close(). At that a non-zero value of the last argument "mode" supposes MDBX will create database and directory if ones does not exist. In this case the non-zero "mode" argument specifies the file mode bits be applied when a new files are created by open() function.

Within that directory, a lock file (aka LCK-file) and a storage file (aka DXB-file) will be generated. If you don't want to use a directory, you can pass the MDBX_NOSUBDIR option, in which case the path you provided is used directly as the DXB-file, and another file with a "-lck" suffix added will be used for the LCK-file.

Once the environment is open, a transaction can be created within it using mdbx_txn_begin(). Transactions may be read-write or read-only, and read-write transactions may be nested. A transaction must only be used by one thread at a time. Transactions are always required, even for read-only access. The transaction provides a consistent view of the data.

Once a transaction has been created, a database (i.e. key-value space inside the environment) can be opened within it using mdbx_dbi_open(). If only one database will ever be used in the environment, a NULL can be passed as the database name. For named databases, the MDBX_CREATE flag must be used to create the database if it doesn't already exist. Also, mdbx_env_set_maxdbs() must be called after mdbx_env_create() and before mdbx_env_open() to set the maximum number of named databases you want to support.

Note: A single transaction can open multiple databases. Generally databases should only be opened once, by the first transaction in the process.

Within a transaction, mdbx_get() and mdbx_put() can store single key-value pairs if that is all you need to do (but see Cursors below if you want to do more).

A key-value pair is expressed as two MDBX_val structures. This struct that is exactly similar to POSIX's struct iovec and has two fields, iov_len and iov_base. The data is a void pointer to an array of iov_len bytes.

Note: The notable difference between MDBX and LMDB is that MDBX support zero length keys.

Because MDBX is very efficient (and usually zero-copy), the data returned in an MDBX_val structure may be memory-mapped straight from disk. In other words look but do not touch (or free() for that matter). Once a transaction is closed, the values can no longer be used, so make a copy if you need to keep them after that.

Cursors

To do more powerful things, we must use a cursor.

Within the transaction, a cursor can be created with mdbx_cursor_open(). With this cursor we can store/retrieve/delete (multiple) values using mdbx_cursor_get(), mdbx_cursor_put() and mdbx_cursor_del().

The mdbx_cursor_get() positions itself depending on the cursor operation requested, and for some operations, on the supplied key. For example, to list all key-value pairs in a database, use operation MDBX_FIRST for the first call to mdbx_cursor_get(), and MDBX_NEXT on subsequent calls, until the end is hit.

To retrieve all keys starting from a specified key value, use MDBX_SET. For more cursor operations, see the C API reference.

When using mdbx_cursor_put(), either the function will position the cursor for you based on the key, or you can use operation MDBX_CURRENT to use the current position of the cursor.

Note: Note that key must then match the current position's key.

Summarizing the opening

So we have a cursor in a transaction which opened a database in an environment which is opened from a filesystem after it was separately created.

Or, we create an environment, open it from a filesystem, create a transaction within it, open a database within that transaction, and create a cursor within all of the above.

Got it?

Threads and processes

Do not have open an database twice in the same process at the same time, MDBX will track and prevent this. Instead, share the MDBX environment that has opened the file across all threads. The reason for this is:

When the "Open file description" locks (aka OFD-locks) are not available, MDBX uses POSIX locks on files, and these locks have issues if one process opens a file multiple times.
If a single process opens the same environment multiple times, closing it once will remove all the locks held on it, and the other instances will be vulnerable to corruption from other processes.
For compatibility with LMDB which allows multi-opening, MDBX can be configured at runtime by mdbx_setup_debug() with MDBX_DBG_LEGACY_MULTIOPEN` option prior to calling other MDBX functions. In this way MDBX will track databases opening, detect multi-opening cases and then recover POSIX file locks as necessary. However, lock recovery can cause unexpected pauses, such as when another process opened the database in exclusive mode before the lock was restored - we have to wait until such a process releases the database, and so on.

Do not use opened MDBX environment(s) after fork() in a child process(es), MDBX will check and prevent this at critical points. Instead, ensure there is no open MDBX-instance(s) during fork(), or at least close it immediately after fork() in the child process and reopen if required - for instance by using pthread_atfork(). The reason for this is:

For competitive consistent reading, MDBX assigns a slot in the shared table for each process that interacts with the database. This slot is populated with process attributes, including the PID.
After fork(), in order to remain connected to a database, the child process must have its own such "slot", which can't be assigned in any simple and robust way another than the regular.
A write transaction from a parent process cannot continue in a child process for obvious reasons.
Moreover, in a multithreaded process at the fork() moment any number of threads could run in critical and/or intermediate sections of MDBX code with interaction and/or racing conditions with threads from other process(es). For instance: shrinking a database or copying it to a pipe, opening or closing environment, beginning or finishing a transaction, and so on. = Therefore, any solution other than simply close database (and reopen if necessary) in a child process would be both extreme complicated and so fragile.

Do not start more than one transaction for a one thread. If you think about this, it's really strange to do something with two data snapshots at once, which may be different. MDBX checks and preventing this by returning corresponding error code (MDBX_TXN_OVERLAPPING, MDBX_BAD_RSLOT, MDBX_BUSY) unless you using MDBX_NOSTICKYTHREADS option on the environment. Nonetheless, with the MDBX_NOSTICKYTHREADS option, you must know exactly what you are doing, otherwise you will get deadlocks or reading an alien data.

Also note that a transaction is tied to one thread by default using Thread Local Storage. If you want to pass transactions across threads, you can use the MDBX_NOSTICKYTHREADS option on the environment. Nevertheless, a write transaction must be committed or aborted in the same thread which it was started. MDBX checks this in a reasonable manner and return the MDBX_THREAD_MISMATCH error in rules violation.

Transactions, rollbacks etc

To actually get anything done, a transaction must be committed using mdbx_txn_commit(). Alternatively, all of a transaction's operations can be discarded using mdbx_txn_abort().

Attention: An important difference between MDBX and LMDB is that MDBX required that any opened cursors can be reused and must be freed explicitly, regardless ones was opened in a read-only or write transaction. The REASON for this is eliminates ambiguity which helps to avoid errors such as: use-after-free, double-free, i.e. memory corruption and segfaults.

For read-only transactions, obviously there is nothing to commit to storage.

Attention: An another notable difference between MDBX and LMDB is that MDBX make handles opened for existing databases immediately available for other transactions, regardless this transaction will be aborted or reset. The REASON for this is to avoiding the requirement for multiple opening a same handles in concurrent read transactions, and tracking of such open but hidden handles until the completion of read transactions which opened them.

In addition, as long as a transaction is open, a consistent view of the database is kept alive, which requires storage. A read-only transaction that no longer requires this consistent view should be terminated (committed or aborted) when the view is no longer needed (but see below for an optimization).

There can be multiple simultaneously active read-only transactions but only one that can write. Once a single read-write transaction is opened, all further attempts to begin one will block until the first one is committed or aborted. This has no effect on read-only transactions, however, and they may continue to be opened at any time.

Duplicate keys aka Multi-values

mdbx_get() and mdbx_put() respectively have no and only some support or multiple key-value pairs with identical keys. If there are multiple values for a key, mdbx_get() will only return the first value.

When multiple values for one key are required, pass the MDBX_DUPSORT flag to mdbx_dbi_open(). In an MDBX_DUPSORT database, by default mdbx_put() will not replace the value for a key if the key existed already. Instead it will add the new value to the key. In addition, mdbx_del() will pay attention to the value field too, allowing for specific values of a key to be deleted.

Finally, additional cursor operations become available for traversing through and retrieving duplicate values.

Some optimization

If you frequently begin and abort read-only transactions, as an optimization, it is possible to only reset and renew a transaction.

mdbx_txn_reset() releases any old copies of data kept around for a read-only transaction. To reuse this reset transaction, call mdbx_txn_renew() on it. Any cursors in this transaction can also be renewed using mdbx_cursor_renew() or freed by mdbx_cursor_close().

To permanently free a transaction, reset or not, use mdbx_txn_abort().

Cleaning up

Any created cursors must be closed using mdbx_cursor_close(). It is advisable to repeat:

Note: An important difference between MDBX and LMDB is that MDBX required that any opened cursors can be reused and must be freed explicitly, regardless ones was opened in a read-only or write transaction. The REASON for this is eliminates ambiguity which helps to avoid errors such as: use-after-free, double-free, i.e. memory corruption and segfaults.

It is very rarely necessary to close a database handle, and in general they should just be left open. When you close a handle, it immediately becomes unavailable for all transactions in the environment. Therefore, you should avoid closing the handle while at least one transaction is using it.

Now read up on the full API!

The full C API documentation lists further details below, like how to:

Configure database size and automatic size management: mdbx_env_set_geometry().
Drop and clean a database: mdbx_drop().
Detect and report errors: Error handling.
Optimize (bulk) loading speed: MDBX_MULTIPLE, MDBX_APPEND.
Reduce (temporarily) robustness to gain even more speed: SYNC MODES.
Gather statistics about the database: Statistics & Information.
Sstimate size of range query result: Range query estimation.
Double performance by LIFO reclaiming on storages with write-back: MDBX_LIFORECLAIM.
Use sequences and canary markers: mdbx_dbi_sequence(), MDBX_canary.
Use Handle-Slow-Readers callback to resolve a database full/overflow issues due to long-lived read transactions: mdbx_env_set_hsr().
Use exclusive mode: MDBX_EXCLUSIVE.
Define custom sort orders (but this is recommended to be avoided).

Bindings

Runtime	Repo	Author
Scala	mdbx4s	David Bouyssié
Haskell	libmdbx-hs	Francisco Vallarino
NodeJS, Deno	lmdbx-js	Kris Zyp
NodeJS	node-mdbx	Сергей Федотов
Ruby	ruby-mdbx	Mahlon E. Smith
Go	mdbx-go	Alex Sharov
Nim	NimDBX	Jens Alfke
Lua	lua-libmdbx	Masatoshi Fukunaga
Rust	libmdbx-rs	Artem Vorotnikov
Rust	mdbx	gcxfd
Java	mdbxjni	Castor Technologies
Python	PyPi/libmdbx	Lazymio
.NET (obsolete)	mdbx.NET	Jerry Wang