GlusterFS is a free and open source scalable filesystem it can be used for cloud storage or to store data in a local network. It can be used to set up an active-active filesystem cluster with failover and loadbalancing via DNS-round robin. Together with CTDB it is possible to build a fileserver for a network with the following advantages:
- Expandable without downtime
- Mount Gluster volumes via the network
- Posix ACL support
- Different configurations possible (depending of your needs)
- Support of snapshots if LVM2 thinly provisioned is used for the bricks
The different configurations available are:
- Replicated Volume
- Distributed Volume
- Striped Volume
- Replicated-Distributed Volume
- Dispersed Volume
To read more about the different configurations see:
|This article is part of CTDB-setup so it just shows how to setup a replicated volume to be used with CTDB. The setup will be a two node replicated volume with 2GB diskspace, so it will be easy to reproduce the setup.|
What you need
- Two hosts with two network cards
- An empty partition of 2GB to create the volume on each host
- Two IP addresses from your production network
- Two IP addresses for the heartbeat network
- The GlusterFS packages version 7.x
Hostnames and IPs
Here you see two tables with the used IP-addresses on both hosts.
If a client should connect to the Gluster-cluster an IP-address from the production network is used.
The heartbeat network is only for the communication between the Gluster-nodes
you need two mountpoints, one for the physical brick and one for the volume.
|Mountpoint||What to mount|
|/gluster||The brick on each node|
|/glusterfs||For the volume on each node|
Setting up the LVM-partition
|Be sure that you are working with the right partition, you will lose all data if you choose the wrong partition.|
The first step will be, setting up the replicated Gluster-Volume with two nodes. As an example a partition with 2GB is used.
root@cluster-01:~# fdisk /dev/sdc root@cluster-01:~# apt install lvm2 thin-provisioning-tools root@cluster-01:~# pvcreate /dev/sdc1 Physical volume "/dev/sdc1" successfully created. root@cluster-01:~# vgcreate glustergroup /dev/sdc1 Volume group "glustergroup" successfully created root@cluster-01:~# lvcreate -L 1950M -T glustergroup/glusterpool Using default stripesize 64,00 KiB Rounding up size to full physical extent 1,91 GiB Logical volume "glusterpool" created. root@cluster-01:~# lvcreate -V 1900M -T glustergroup/glusterpool -n glusterv1 Using default stripesize 64,00 KiB. Logical volume "glusterv1" created. root@cluster-01:~# mkfs.xfs /dev/glustergroup/glusterv1 root@cluster-01:~# mkdir /gluster root@cluster-01:~# mount /dev/glustergroup/glusterv1 /gluster root@cluster-01:~# echo /dev/glustergroup/glusterv1 /gluster xfs defaults 0 0 >> /etc/fstab root@cluster-01:~# mkdir /gluster/brick
Do all the steps on both nodes.
Creating the peer pool
|Make sure that you are using the the hostnames from the heartbeat network, to be sure that the communication between the nodes is using the heartbeat network.|
Before you can create the volume you have to set up a peer pool, by adding the two hosts as peer to the pool In next listing you will see the commands to add the second gluster-node to the pool. You have to do this on the first of the first gluster-host:
root@cluster-01:~# gluster peer probe c-02 peer probe: success.
If you try to add the peer and you get one of the following error messages:
root@cluster-01:~# gluster peer probe c-02 Connection failed. Please check if gluster daemon is operational. root@cluster-01:~# gluster peer probe c-02 peer probe: failed: Probe returned with Transport endpoint is not connected
The first error message will point you to a not running glusterd on the host you are trying to add the peer. Restart the the daemon
systemctl restart glusterd
The second error message will point to a not running daemon on the peer you are trying to add to the pool. Restart the glusterd on the other node.
If you could add the node c-02 on the node c-01, add the host c-01 to the trusted pool on node c-02
root@cluster-02:~# gluster peer probe c-01 peer probe: success.
Now you can check the status of each node and take a look at the list of all nodes with the gluster-command
root@cluster-01:~# gluster peer status Number of Peers: 1 Hostname: c-02 Uuid: aca7d361-51df-4d1f-9b0f-4cf494029f21 State: Peer in Cluster (Connected)
root@cluster-02:~# gluster peer status Number of Peers: 1 Other names: c-02 Hostname: c-01.heartbeat.net Uuid: adafbf93-e716-4d99-bf89-e8044d57e3aa State: Peer in Cluster (Connected) Other names: c-01
root@cluster-02:~# gluster pool list UUID Hostname State adafbf93-e716-4d99-bf89-e8044d57e3aa c-01.heartbeat.net Connected aca7d361-51df-4d1f-9b0f-4cf494029f21 localhost Connected
On each host you will find the information of the peer in /var/lib/glusterd/peers/<UUID>
Now you have all the peers added to the pool, you will need for the gluster-volume.