Setting up CTDB for Clustered NFS

Revision as of 06:01, 29 January 2017 by MartinSchwenke (talk | contribs) (→‎Prerequisites)

Assumptions

This guide is aimed at the Linux kernel NFS daemon.

CTDB can be made to manage another NFS server by using the CTDB_NFS_CALLOUT configuration variable to specify an NFS server-specific call-out.

Prerequisites

NFS configuration

Exports

Requirements:

  • NFS exports must be the same on all nodes
  • For each export, the fsid option must be set to the same value on all nodes.

For the Linux kernel NFS server, this is usually in /etc/exports.

Example: 

 /clusterfs0/data *(rw,fsid=1235)
 /clusterfs0/misc *(rw,fsid=1237)

Daemon configuration

Clustering NFS has some extra requirements compared to running a regular NFS server, so some extra configuration is needed.

  • All NFS daemons should run on fixed ports, which should be the same on all cluster nodes. Some clients can become confused if ports change during fail-over.
  • NFSv4 should be disabled.
  • statd should be configured to use CTDB's high-availability call-out.
  • statd's hostname should be resolvable into the CTDB public IP addresses. It should be the same name used by Samba. This should use the value of NFS_HOSTNAME, since this is used by CTDB's high-availability call-out.

Red Hat Linux variants

The configuration file will be /etc/sysconfig/nfs and it should look something like: 

 NFS_HOSTNAME="ctdb"
 RPCNFSDARGS="-N 4"
 RPCNFSDCOUNT=32
 STATD_PORT=595
 STATD_OUTGOING_PORT=596
 MOUNTD_PORT=597
 RQUOTAD_PORT=598
 LOCKD_UDPPORT=599
 LOCKD_TCPPORT=599
 STATD_HOSTNAME="$NFS_HOSTNAME"
 STATD_HA_CALLOUT="/etc/ctdb/statd-callout"

Configure CTDB to manage NFS

In the CTDB configuration, tell CTDB that you want it to manage NFS: 

 CTDB_MANAGES_NFS=yes

CTDB will manage and start/stop/restart the NFS services, so the operating system should be configured so these are not started/stopped automatically.

Red Hat Linux variants

If using a Red Hat Linux variant, the NFS services are nfs and nfslock services. Starting them at boot time is not recommended and this can be disabled using chkconfig. 

 chkconfig nfs off
 chkconfig nfslock off

The service names and mechanism for disabling them varies across operating systems.

Client configuration

IP addresses, rather than a DNS/host name, should be used when configuring client mounts. NFSv3 locking is heavily tied to IP addresses and can break if a client uses round-robin DNS. This means load balancing for NFS is achieved by hand-distributing public IP addresses across clients.

IMPORTANT

Never mount the same NFS share on a client from two different nodes in the cluster at the same time. The client-side caching in NFS is very fragile and assumes that an object can only be accessed through one single path at a time.

Event scripts

CTDB clustering for NFS relies on two event scripts 06.nfs and 60.nfs. These are provided as part of CTDB and do not usually need to be changed. The NFS server being managed can be changed by providing a call-out and setting the CTDB_NFS_CALLOUT configuration variable to point to it.

Retrieved from "https://wiki.samba.org/index.php?title=Setting_up_CTDB_for_Clustered_NFS&oldid=13031"