Content based addressing for Kadalu Storage

Kadalu Storage volumes are accessible as Fuse mount, NFS(NFS Ganesha) or using GFAPIs (Since Kadalu Storage is based on GlusterFS, GFAPI works with Kadalu Storage without any changes).

Content-addressable storage (CAS), also referred to as content-addressed storage or fixed-content storage, is a way to store information so it can be retrieved based on its content, not its name or location. It has been used for high-speed storage and retrieval of fixed content, such as documents stored for compliance with government regulations. (Wikipedia)

Many attempts have been made to provide object storage using GlusterFS, but none of them are actively maintained. We have started a new project(github/kadalu/kadalu-cas) to provide a content addressable access and object storage for Kadalu Storage. This project can be used with Kadalu Storage/GlusterFS or any directory backend that supports hard links.

Similar to IPFS or S3, access the files from Kadalu Storage Volumes as objects. This is currently a beta feature, that allows to access the files using its content address (hash) similar to IPFS. Files are stored efficiently to handle de-duplication and provides ReST APIs to upload or download files. S3 API compatibility is not available with the current release, but it can be added in future on need basis.

Why another layer when Kadalu Storage can be mounted and used?

Kadalu Storage clients are not available for many devices, for example mobile devices, Windows apps or embedded devices.
Very easy to integrate with other web applications or CLI without needing any other dependencies.
Security - Instead of providing access to the servers from the external Gluster clients, host CAS in one of the managed node and provide API access on need basis.
Hybrid Volume access - Only a part of Kadalu Storage Volume can be exposed as Content addressable Storage.

Features

Bucket level authentication - Buckets are top level directories. Three types of roles are available - admin, writer, and reader.
Buckets access controls are maintained in config file itself. In the future, this will be moved to xattrs to avoid additional Storage to persist the meta details.
De-duplication - While creating the file, a hash is constructed based on the content of the file. If the hash already exists, then it will not create the file again.
Content based addressing - If you know the hash of the file, then access the file content using that hash without knowing the full path of the file (This again depends on the access available for the respective bucket).
Inbuilt web server to make any bucket to serve Static files or single page applications.
Version support (Future versions)
SDKs in Python, JS, Ruby (Future versions)

Many more features are planned with the upcoming releases. Please let us know if you need a specific feature to be implemented.

Install Kadalu Storage and Kadalu CAS packages

Follow this quick start guide to install Kadalu Storage.

Install Kadalu CAS server by running the following command

curl -fsSL https://github.com/kadalu/kadalu-cas/releases/latest/download/install.sh | sudo bash -x

Create and mount a Kadalu Storage Volume

kadalu volume create PROD/vol1   \
    mirror server1:/data/vol1/s1 \
           server2:/data/vol1/s2 \
           server3:/data/vol1/s3
mkdir /mnt/vol1
chattr +i /mnt/vol1
mount -t kadalu PROD/vol1 /mnt/vol1

Now create the configuration file based on your requirements. (Bucket configurations will be handled by APIs itself in the future, for now configure manually till the buckets APIs are available). All hash values used in the configuration file below are sha256 hash of secrets (Ex: echo -n "secret" | sha256sum).

port: 4001
global_admins:
  - e12e115acf4552b2568b55e93cbd39394c4ef81c82447fafc997882a02d23677
rootdir: "/mnt/vol1"
buckets:
  - name: app
    public: true
  - name: photos
    admins:
      - a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3
    readers:
      - b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0

Start the CAS server by running,
```
kadalu-cas kadalu.cas.vol1.yml
```

Upload file example

$ curl -i -XPOST -H "Authorization: Bearer 3dfdf4b9-3130-4264-ba2e-570afe74dd08" \
    http://localhost:4001/photos/hello.png -F "data=@/tmp/hello.png"
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json
Content-Length: 75

{"hash":"e67413efc0bb83f7f1e46179448d4f586567949d4d6496928950ded66a7d7f08"}

Download a file by hash

$ curl -i -XGET -H "Authorization: Bearer 3dfdf4b9-3130-4264-ba2e-570afe74dd08" \
    http://localhost:4001/photos/e67413efc0bb83f7f1e46179448d4f586567949d4d6496928950ded66a7d7f08
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/octet-stream
Accept-Ranges: bytes
X-Content-Type-Options: nosniff
Content-Length: 2550

Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

Download example: By file path

$ curl -i -XGET -H "Authorization: Bearer 3dfdf4b9-3130-4264-ba2e-570afe74dd08" \
    http://localhost:4001/photos/hello.png
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: image/png
Accept-Ranges: bytes
X-Content-Type-Options: nosniff
Content-Length: 2550

Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

Example of using an image from public bucket in HTML

<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8"/>
    </head>
    <body>
        <img src="http://localhost:4001/app/e67413efc0bb83f7f1e46179448d4f586567949d4d6496928950ded66a7d7f08"/>
    </body>
</html>

Please provide your valuable feedback and feel free to open issues with feature requests or bug reports (Github Issues)

Content based addressing for Kadalu Storage

Why another layer when Kadalu Storage can be mounted and used?

Features

Install Kadalu Storage and Kadalu CAS packages

Company

Resources

Community

Contact Us