Difference between revisions of "Setting up CTDB for Clustered NFS"
m (MartinSchwenke moved page Setting Up CTDB For Clustered NFS to Setting up CTDB for Clustered NFS: Title looks better...) |
(Clean-ups and modernisation) |
||
Line 1: | Line 1: | ||
+ | = Assumptions = | ||
+ | |||
+ | This guide is aimed at the Linux kernel NFS daemon. | ||
+ | |||
+ | CTDB can be made to manage another NFS server by using the <code>CTDB_NFS_CALLOUT</code> configuration variable to specify an NFS server-specific call-out. | ||
+ | |||
= First steps = | = First steps = | ||
− | [CTDB_Setup|Configure CTDB] and set it up to use public | + | [[CTDB_Setup|Configure CTDB]] and set it up to use public IP addresses. Verify that the CTDB cluster works. |
+ | |||
+ | = NFS configuration = | ||
+ | |||
+ | == Exports == | ||
+ | |||
+ | Requirements: | ||
− | + | * NFS exports must be the same on all nodes | |
+ | * For each export, the <code>fsid</code> option must be set to the same value on all nodes. | ||
− | + | For the Linux kernel NFS server, this is usually in <code>/etc/exports</code>. | |
− | |||
− | + | Example: | |
− | = / | + | /clusterfs0/data *(rw,fsid=1235) |
+ | /clusterfs0/misc *(rw,fsid=1237) | ||
− | + | == Daemon configuration == | |
− | |||
− | |||
− | + | Clustering NFS has some extra requirements compared to running a regular NFS server, so some extra configuration is needed. | |
− | + | * All NFS daemons should run on fixed ports, which should be the same on all cluster nodes. Some clients can become confused if ports change during fail-over. | |
− | + | * NFSv4 should be disabled. | |
+ | * <code>statd</code> should be configured to use CTDB's high-availability call-out. | ||
+ | * <code>statd</code>'s hostname should be resolvable into the CTDB public IP addresses. It should be the same name used by Samba. This should use the value of NFS_HOSTNAME, since this is used by CTDB's high-availability call-out. | ||
+ | |||
+ | === Red Hat Linux variants === | ||
+ | |||
+ | The configuration file will be <code>/etc/sysconfig/nfs</code> and it should look something like: | ||
+ | |||
+ | NFS_HOSTNAME="ctdb" | ||
+ | RPCNFSDARGS="-N 4" | ||
+ | RPCNFSDCOUNT=32 | ||
STATD_PORT=595 | STATD_PORT=595 | ||
STATD_OUTGOING_PORT=596 | STATD_OUTGOING_PORT=596 | ||
Line 26: | Line 47: | ||
LOCKD_UDPPORT=599 | LOCKD_UDPPORT=599 | ||
LOCKD_TCPPORT=599 | LOCKD_TCPPORT=599 | ||
− | + | STATD_HOSTNAME="$NFS_HOSTNAME" | |
− | + | STATD_HA_CALLOUT="/etc/ctdb/statd-callout" | |
− | |||
− | |||
− | + | = Configure CTDB to manage NFS = | |
− | |||
− | + | In the CTDB configuration, tell CTDB that you want it to manage NFS: | |
− | + | CTDB_MANAGES_NFS=yes | |
− | |||
− | |||
− | |||
− | + | CTDB will manage and start/stop/restart the NFS services, so the operating system should be configured so these are not started/stopped automatically. | |
− | |||
− | + | == Red Hat variants == | |
− | |||
− | + | If using a Red Hat variant, the NFS services are <code>nfs</code> and <code>nfslock</code> services. Starting them at boot time is not recommended and this can be disabled using <code>chkconfig</code>. | |
− | |||
− | |||
− | |||
− | |||
chkconfig nfs off | chkconfig nfs off | ||
chkconfig nfslock off | chkconfig nfslock off | ||
− | + | The service names and mechanism for disabling them varies across operating systems. | |
− | + | = Client configuration = | |
− | |||
− | |||
− | + | IP addresses, rather than a DNS/host name, should be used when configuring client mounts. NFSv3 locking is heavily tied to IP addresses and can break if a client uses round-robin DNS. This means load balancing for NFS is achieved by hand-distributing public IP addresses across clients. | |
− | = | + | == IMPORTANT == |
− | + | Never ever mount the same NFS share on a client from two different nodes in the cluster at the same time. The client-side caching in NFS is very fragile and assumes that an object can only be accessed through one single path at a time. | |
− | |||
− | = | + | = Event scripts = |
− | + | CTDB clustering for NFS relies on two event scripts <code>06.nfs</code> and <code>60.nfs</code>. These are provided as part of CTDB and do not usually need to be changed. The NFS server being managed can be changed by providing a call-out and setting the <code>CTDB_NFS_CALLOUT</code> configuration variable to point to it. | |
− | The |
Revision as of 05:29, 21 October 2016
Contents
Assumptions
This guide is aimed at the Linux kernel NFS daemon.
CTDB can be made to manage another NFS server by using the CTDB_NFS_CALLOUT
configuration variable to specify an NFS server-specific call-out.
First steps
Configure CTDB and set it up to use public IP addresses. Verify that the CTDB cluster works.
NFS configuration
Exports
Requirements:
- NFS exports must be the same on all nodes
- For each export, the
fsid
option must be set to the same value on all nodes.
For the Linux kernel NFS server, this is usually in /etc/exports
.
Example:
/clusterfs0/data *(rw,fsid=1235) /clusterfs0/misc *(rw,fsid=1237)
Daemon configuration
Clustering NFS has some extra requirements compared to running a regular NFS server, so some extra configuration is needed.
- All NFS daemons should run on fixed ports, which should be the same on all cluster nodes. Some clients can become confused if ports change during fail-over.
- NFSv4 should be disabled.
statd
should be configured to use CTDB's high-availability call-out.statd
's hostname should be resolvable into the CTDB public IP addresses. It should be the same name used by Samba. This should use the value of NFS_HOSTNAME, since this is used by CTDB's high-availability call-out.
Red Hat Linux variants
The configuration file will be /etc/sysconfig/nfs
and it should look something like:
NFS_HOSTNAME="ctdb" RPCNFSDARGS="-N 4" RPCNFSDCOUNT=32 STATD_PORT=595 STATD_OUTGOING_PORT=596 MOUNTD_PORT=597 RQUOTAD_PORT=598 LOCKD_UDPPORT=599 LOCKD_TCPPORT=599 STATD_HOSTNAME="$NFS_HOSTNAME" STATD_HA_CALLOUT="/etc/ctdb/statd-callout"
Configure CTDB to manage NFS
In the CTDB configuration, tell CTDB that you want it to manage NFS:
CTDB_MANAGES_NFS=yes
CTDB will manage and start/stop/restart the NFS services, so the operating system should be configured so these are not started/stopped automatically.
Red Hat variants
If using a Red Hat variant, the NFS services are nfs
and nfslock
services. Starting them at boot time is not recommended and this can be disabled using chkconfig
.
chkconfig nfs off chkconfig nfslock off
The service names and mechanism for disabling them varies across operating systems.
Client configuration
IP addresses, rather than a DNS/host name, should be used when configuring client mounts. NFSv3 locking is heavily tied to IP addresses and can break if a client uses round-robin DNS. This means load balancing for NFS is achieved by hand-distributing public IP addresses across clients.
IMPORTANT
Never ever mount the same NFS share on a client from two different nodes in the cluster at the same time. The client-side caching in NFS is very fragile and assumes that an object can only be accessed through one single path at a time.
Event scripts
CTDB clustering for NFS relies on two event scripts 06.nfs
and 60.nfs
. These are provided as part of CTDB and do not usually need to be changed. The NFS server being managed can be changed by providing a call-out and setting the CTDB_NFS_CALLOUT
configuration variable to point to it.