Setting up CTDB for Clustered NFS
Contents
Assumptions
This guide is aimed at the Linux kernel NFS daemon.
CTDB can be made to manage another NFS server by using the CTDB_NFS_CALLOUT
configuration variable to specify an NFS server-specific call-out.
Prerequisites
- Basic CTDB configuration
- Setting up a cluster filesystem
- Configuring the CTDB recovery lock (recommended)
- Adding public IP addresses (or some other failover/load balancing scheme)
NFS configuration
Exports
Requirements:
- NFS exports must be the same on all nodes
- For each export, the
fsid
option must be set to the same value on all nodes.
For the Linux kernel NFS server, this is usually in /etc/exports
.
Example:
/clusterfs0/data *(rw,fsid=1235) /clusterfs0/misc *(rw,fsid=1237)
Daemon configuration
Clustering NFS has some extra requirements compared to running a regular NFS server, so some extra configuration is needed.
- All NFS daemons should run on fixed ports, which should be the same on all cluster nodes. Some clients can become confused if ports change during fail-over.
- NFSv4 should be disabled.
statd
should be configured to use CTDB's high-availability call-out.statd
's hostname should be resolvable into the CTDB public IP addresses. It should be the same name used by Samba. This should use the value of NFS_HOSTNAME, since this is used by CTDB's high-availability call-out.
Red Hat Linux variants
The configuration file will be /etc/sysconfig/nfs
and it should look something like:
NFS_HOSTNAME="ctdb" RPCNFSDARGS="-N 4" RPCNFSDCOUNT=32 STATD_PORT=595 STATD_OUTGOING_PORT=596 MOUNTD_PORT=597 RQUOTAD_PORT=598 LOCKD_UDPPORT=599 LOCKD_TCPPORT=599 STATD_HOSTNAME="$NFS_HOSTNAME" STATD_HA_CALLOUT="/etc/ctdb/statd-callout"
Configure CTDB to manage NFS
In the CTDB configuration, tell CTDB that you want it to manage NFS:
CTDB_MANAGES_NFS=yes
CTDB will manage and start/stop/restart the NFS services, so the operating system should be configured so these are not started/stopped automatically.
Red Hat Linux variants
If using a Red Hat Linux variant, the NFS services are nfs
and nfslock
services. Starting them at boot time is not recommended and this can be disabled using chkconfig
.
chkconfig nfs off chkconfig nfslock off
The service names and mechanism for disabling them varies across operating systems.
Client configuration
IP addresses, rather than a DNS/host name, should be used when configuring client mounts. NFSv3 locking is heavily tied to IP addresses and can break if a client uses round-robin DNS. This means load balancing for NFS is achieved by hand-distributing public IP addresses across clients.
IMPORTANT
Never mount the same NFS share on a client from two different nodes in the cluster at the same time. The client-side caching in NFS is very fragile and assumes that an object can only be accessed through one single path at a time.
Event scripts
CTDB clustering for NFS relies on two event scripts 06.nfs
and 60.nfs
. These are provided as part of CTDB and do not usually need to be changed. The NFS server being managed can be changed by providing a call-out and setting the CTDB_NFS_CALLOUT
configuration variable to point to it.