Setting up CTDB for Clustered NFS
This guide is aimed at the Linux kernel NFS daemon.
CTDB can be made to manage another NFS server by using the
CTDB_NFS_CALLOUT configuration variable to specify an NFS server-specific call-out.
- Basic CTDB configuration
- Setting up a cluster filesystem
- Configuring the CTDB recovery lock (recommended)
- Adding public IP addresses (or some other failover/load balancing scheme)
- NFS exports must be the same on all nodes
- For each export, the
fsidoption must be set to the same value on all nodes.
For the Linux kernel NFS server, this is usually in
/clusterfs0/data *(rw,fsid=1235) /clusterfs0/misc *(rw,fsid=1237)
Clustering NFS has some extra requirements compared to running a regular NFS server, so some extra configuration is needed.
- All NFS daemons should run on fixed ports, which should be the same on all cluster nodes. Some clients can become confused if ports change during fail-over.
- NFSv4 should be disabled.
statdshould be configured to use CTDB's high-availability call-out.
statd's hostname should be resolvable into the CTDB public IP addresses. It should be the same name used by Samba. This should use the value of NFS_HOSTNAME, since this is used by CTDB's high-availability call-out.
Red Hat Linux variants
The configuration file will be
/etc/sysconfig/nfs and it should look something like:
NFS_HOSTNAME="ctdb" RPCNFSDARGS="-N 4" RPCNFSDCOUNT=32 STATD_PORT=595 STATD_OUTGOING_PORT=596 MOUNTD_PORT=597 RQUOTAD_PORT=598 LOCKD_UDPPORT=599 LOCKD_TCPPORT=599 STATD_HOSTNAME="$NFS_HOSTNAME" STATD_HA_CALLOUT="/etc/ctdb/statd-callout"
Configure CTDB to manage NFS
In the CTDB configuration, tell CTDB that you want it to manage NFS:
CTDB will manage and start/stop/restart the NFS services, so the operating system should be configured so these are not started/stopped automatically.
Red Hat Linux variants
If using a Red Hat Linux variant, the NFS services are
nfslock services. Starting them at boot time is not recommended and this can be disabled using
chkconfig nfs off chkconfig nfslock off
The service names and mechanism for disabling them varies across operating systems.
IP addresses, rather than a DNS/host name, should be used when configuring client mounts. NFSv3 locking is heavily tied to IP addresses and can break if a client uses round-robin DNS. This means load balancing for NFS is achieved by hand-distributing public IP addresses across clients.
Never mount the same NFS share on a client from two different nodes in the cluster at the same time. The client-side caching in NFS is very fragile and assumes that an object can only be accessed through one single path at a time.
CTDB clustering for NFS relies on two event scripts
60.nfs. These are provided as part of CTDB and do not usually need to be changed. The NFS server being managed can be changed by providing a call-out and setting the
CTDB_NFS_CALLOUT configuration variable to point to it.