Setting up CTDB for Clustered NFS
Assumptions
This guide is aimed at the Linux kernel NFS daemon.
CTDB can be made to manage another NFS server by using the CTDB_NFS_CALLOUT
configuration variable to specify an NFS server-specific call-out.
Prerequisites
- Basic CTDB configuration
- Setting up a cluster filesystem
- Configuring the CTDB recovery lock (recommended)
- Adding public IP addresses (or some other failover/load balancing scheme)
NFS configuration
Exports
Requirements:
- NFS exports must be the same on all nodes
- For each export, the
fsid
option must be set to the same value on all nodes.
For the Linux kernel NFS server, this is usually in /etc/exports
.
Example:
/clusterfs0/data *(rw,fsid=1235) /clusterfs0/misc *(rw,fsid=1237)
Daemon configuration
Clustering NFS has some extra requirements compared to running a regular NFS server, so some extra configuration is needed.
- All NFS daemons should run on fixed ports, which should be the same on all cluster nodes. Some clients can become confused if ports change during fail-over.
- NFSv4 should be disabled.
statd
should be configured to use CTDB's high-availability call-out.statd
's hostname should be resolvable into the CTDB public IP addresses. It should be the same name used by Samba. This should use the value of NFS_HOSTNAME, since this is used by CTDB's high-availability call-out.
Red Hat Linux variants
The configuration file will be /etc/sysconfig/nfs
and it should look something like:
NFS_HOSTNAME="ctdb" RPCNFSDARGS="-N 4" RPCNFSDCOUNT=32 STATD_PORT=595 STATD_OUTGOING_PORT=596 MOUNTD_PORT=597 RQUOTAD_PORT=598 LOCKD_UDPPORT=599 LOCKD_TCPPORT=599 STATD_HOSTNAME="$NFS_HOSTNAME" STATD_HA_CALLOUT="/etc/ctdb/statd-callout"
Configure CTDB to manage NFS
In the CTDB configuration, tell CTDB that you want it to manage NFS:
CTDB_MANAGES_NFS=yes
CTDB will manage and start/stop/restart the NFS services, so the operating system should be configured so these are not started/stopped automatically.
Red Hat Linux variants
If using a Red Hat Linux variant, the NFS services are nfs
and nfslock
services. Starting them at boot time is not recommended and this can be disabled using chkconfig
.
chkconfig nfs off chkconfig nfslock off
The service names and mechanism for disabling them varies across operating systems.
Client configuration
IP addresses, rather than a DNS/host name, should be used when configuring client mounts. NFSv3 locking is heavily tied to IP addresses and can break if a client uses round-robin DNS. This means load balancing for NFS is achieved by hand-distributing public IP addresses across clients.
IMPORTANT
Never mount the same NFS share on a client from two different nodes in the cluster at the same time. The client-side caching in NFS is very fragile and assumes that an object can only be accessed through one single path at a time.
Event scripts
CTDB clustering for NFS relies on two event scripts 06.nfs
and 60.nfs
. These are provided as part of CTDB and do not usually need to be changed. The NFS server being managed can be changed by providing a call-out and setting the CTDB_NFS_CALLOUT
configuration variable to point to it.