8.1.1. Installation on Proxmox VE

Logical Volume and File System Setup

To start with GlusterFS, we need to install the server dependency

$ apt-get install glusterfs-server -y

After we’ve done that we need to create the Logical Volume where we’ll setup Gluster.

$ pvcreate /dev/sdb
$ vgcreate vg_glusterfs /dev/sdb
$ lvcreate -l100%FREE -n lv_gluster vg_glusterfs

Note

The device /dev/sdb will vary depending on your setup, be wary of not using the root OS disk in a production environment if possible.

Make a Filesystem for Gluster to sync.

$ mkfs.xfs /dev/vg_glusterfs/lv_gluster

Warning

Make sure the filesystem you use supports extended attributes, XFS and/or ZFS are recommended.

Now we can start the glusterd service.

$ systemctl enable glusterd
$ systemctl start glusterd

And create the data directory for glusterfs

$ mkdir -p /data/gluster

# Create an FSTAB entry for the volume:

$ echo /dev/vg_glusterfs/lv_gluster /data/gluster xfs defaults 0 0 >> /etc/fstab

$ mount /data/gluster
$ mkdir -p "/data/gluster/brick{1}"

Note

The number between braces will vary depending on what node of the cluster you’re in. E.g.: If you’re in pve02 the data set will be /data/gluster/brick2

Once we’ve verified that the local LV (for example with df -h) is correctly mounted, and we’ve done this on every node that will have GlusterFS we can start adding peers!

$ gluster peer probe pve01.local.home
$ gluster peer probe pve02.local.home
$ gluster peer probe arbiter-a.local.home

#Output should be the following:
$ gluster peer probe pve02
Probe successful
$ gluster peer probe arbiter-a
Probe successful

Gluster Peer Configuration (2 Storage Nodes and an Arbiter)

Warning

If you’re going to use more than 2 nodes without an Arbiter (since it would be unnecessary) skip ahead to Gluster Peer Configuration (3 Storage Nodes or More)

In this configuration scheme there will be 2 storage nodes and an arbiter with less space (to avoid split-brains).

Note

Arbiter Brick Sizing depends on your disk space, for that it’s recommended that you check the official gluster documentation.

Src: https://docs.gluster.org/en/latest/Administrator-Guide/arbiter-volumes-and-quorum/

After all nodes have been probed we can create the gluster volume

$ gluster volume create gluster_vol replica 3 arbiter 1 pve01:/data/gluster/brick1 pve02:/data/gluster/brick2 arbiter-a:/data/gluster/arbitera

Note

You can check the status by writing gluster peer status in the terminal/shell

Now we need to start the volume so that all data in the volume gets replicated to each storage node.

$ gluster volume start gluster_vol

Gluster Peer Configuration (3 Storage Nodes or More)

To set it up with more nodes and no arbiter, it’s basically the same process as above, but easier because you don’t have to calculate the size of the arbiter block device:

$ gluster volume create gluster_vol replica 4 pve01:/data/gluster/brick1 pve02:/data/gluster/brick2 pve03:/data/gluster/brick3 pve04:/data/gluster/brick4

And now you can start the volume.

$ gluster volume start gluster_vol

Done!