Kadalu Storage 1.1.0 released!

The previous version defines a pool as just a namespace, and the collection of storage from multiple nodes were called Volumes. This added a lot of confusion, especially when used with Kubernetes (Persistent Volumes(PVs) and Kadalu Volumes) and usability issues.

With this release, the collection of nodes is called Cluster and the collection of storage/directories from multiple servers is called Storage Pool. If you are familiar with GlusterFS, the Kadalu Storage pool is equivalent to a GlusterFS volume.

Kadalu Storage uses template-based Volfile generation. The previous version used YAML templates with Kadalu Storage Manager (non-Kubernetes) and Jinja templates when integrated with Kubernetes. We realized these templates are not flexible, and very difficult to add/customize the options. With this release, both Kubernetes integration and the Storage manager uses the new Volgen library. The new volgen library uses a hybrid approach that uses Jinja templates as a base and post-processes the templates to apply and customize the options.

Kadalu Storage uses only the core filesystem layer from Gluster. The Kadalu Storage manager can’t be used as the volfile server since it is ReST API based and the Storage clients (Mounts, gfapi clients) use different RPC. We enhanced the Storage unit processes (Brick processes in Gluster) to serve the volfiles instead of recreating the RPC server in the Storage manager. Serving volfiles from the Storage unit processes added a few more benefits, like high availability of the Volfile data (Backup volfile servers) or a selected number of nodes to update when a Pool configuration changes. For example, if a Cluster contains 100 nodes and the Pool consists of only three nodes, then update only those three nodes when a volfile changes instead of updating all nodes.

Kadalu Storage uses the GlusterFS core file system layer, and the applications using Gluster APIs(GFAPIs) work as is with Kadalu Storage. Kadalu Storage can adapt Gluster FSAL (NFS Ganesha Gluster plugin) with a few changes since it is GFAPI based. Volfile server port is hard coded as 24007 in Gluster FSAL since Glusterd in all nodes will use the same port. But in Kadalu Storage, Volfile serving is delegated to storage unit processes of respective storage pools. Volfile server port can’t be the same for all Storage pools. Hardcoded port value was a blocker to adding NFS access to Kadalu Storage pools. We enhanced Gluster FSAL to understand the Volfile server port and multiple volfile servers (Backup volfile servers). Install the NFS Ganesha Kadalu plugin by running the following command (Ubuntu 22.04).

A separate table in the DB stores the details of the storage-unit ports. Deleting a pool didn’t clear the ports used. Due to this, it was not possible to recreate a Pool using the same ports. With this release, Port reservations are cleaned up after the pool delete.

To increase the pool size, add more distribute groups. New storage units should be multiples of existing distribute group size. For example, if the existing Pool type is Mirror 3, then add 3, 6, 9… Storage units to increase the distribute groups.

After adding new distribute groups, run rebalance to re-distribute the files among the distribute groups (new and existing).

With this release, Kadalu Storage manager now handles pool expansion and rebalance.

The Storage manager starts and manages many other processes like fsd, shd etc. The goal is to make the filesystem layer available even when the Storage manager goes down or is upgraded. With this release, the Kadalu Storage manager creates the systemd unit files when it needs to start the fsd, shd or any other processes.

If the workload is to create small volumes from the Storage pool, creating a bigger pool may degrade the performance of small volumes. Each volume’s layout has to be maintained in all the distribute groups of the Pool and files are distributed between the distribute groups (from multiple nodes) even though all files of a Volume can be saved in a single distributed group. Alternatively, create multiple pools without enabling the distribution and assign the volume to a pool. For the home directory use case below,

Option 1: Create one big pool with 4 distribution groups

Option 2: Create four pools without distribution groups

Option 2 is the better solution for the use case mentioned above. All files belonging to a Volume (one user’s home directory) will be stored in one distribution group. Also, no rebalance is required when new users joined the company. Keep some buffer in the Pool while provisioning users or migrate selected Volumes (users) to the new pool if storage needs to be increased for existing users (Option 1 requires full Pool rebalance if more storage is added).

This release adds a warning when a user tries to create a distributed Pool. Use the --distribute flag or --mode=script to override this behaviour.

Example output:

Previously, the access logs were only printed in stdout. Now it is printed in log files when configured. Also, access logs now include additional details like the remote IP address.

Fixed the issues while generating the Volfiles of disperse pools.

Earlier PV create was failing because of the wrong calculation of available space. With this release, PV create will succeed if the space available in the respective storage pool.

With the new volgen library and volfile server, volfiles are not available in CSI provisioner pods. Heal info commands are now executed from server pods, since the client volfiles are available there.

Arbiter pool is a variant of the Replica/Mirror type. Two storage units of the Arbiter pool store the actual data and metadata, and the third storage unit only stores metadata and the layout (Empty files). The advantage of this type is less storage needed to get the same high availability of Replica/Mirror 3 type.

A huge thanks to all the awesome people who made this release possible. Kadalu Storage is 100% Open Source. To maintain and increase the development pace, sponsorships are essential. Github Sponsor and OpenCollective are available. Reach out to hello@kadalu.tech if you’d like to become a direct sponsor or find other ways to support Kadalu Storage. We thank you in advance!