Downgrading an Active Directory DC

From SambaWiki

Overview

In general, there are two ways to upgrade or downgrade a Samba Active Directory (AD) Domain Controller (DC):

  • In-place upgrades. This is the simplest approach, as you only need to install the new Samba packages. The same database will get used by the new Samba release. Occasionally this approach may require additional manual steps in order to operate correctly (for example, in v4.8 the location of the bind-dns directory changed).
The 'in-place' method is described in more detail here.
  • DC rejoin. As well as upgrading the Samba packages, the domain controller is rejoined to the domain. The rejoined DC is effectively treated as a new machine, with a new ID (i.e. its InvocationId). The rejoined DC's database gets completely rebuilt from scratch.
The 'DC rejoin' method is described in more detail here.


Database compatibility over time

As the Samba code changes over time, it can affect the database in several ways:

  1. Database format. How the database records are stored on disk can change, without altering the actual contents of the database. Changes to the database format were introduced in v4.8 and v4.11.
  2. Database semantics. New database functionality and improvements can optionally be unlocked. For example, v4.7 started storing linked attributes in sorted order, which significantly sped up many AD operations.

Newer Samba releases maintain database backwards-compatibility with older Samba releases. So upgrading generally should be seamless - the database automatically gets rewritten in the new format. Usually this happens when you first start the upgraded samba executable.

However, downgrading presents a problem. The older Samba code simply does not understand the new database format, so the samba executable won't run properly. Therefore when downgrading Samba 'in-place', you must downgrade the database format manually. Note that you must do this before you replace the Samba packages, as the older packages won't know how to downgrade the database correctly.

In-place upgrades vs DC rejoins

Each of the upgrade approaches have some benefits and some downsides.

  • Database semantics. Using the DC rejoin approach means that you can utilize the new and improved database functionality in the latest Samba release. Whereas in-place upgrades will continue to carry-over some legacy database functionality from earlier releases.
  • Consistency and reliability. DC rejoins should be more reliable overall. This is because the database is re-created over replication, so any non-replicated local database inconsistencies should be eliminated. Test coverage of older database semantics will also gradually erode over time, so using the latest database functionality should be more robust.
  • Automation and orchestration. DC rejoins are harder to script/automate, so if you have a large network of DCs it does take more effort to upgrade them. As you are essentially replacing every DC in the network, it may take some time for DRS replication across the network to stabilize.
  • Risk and recovery. Joining a DC is not a trivial process, so you may also encounter separate problems related to this. If the rejoin goes badly, you will have overwritten that DC's database - this is a more difficult situation to recover from. Whereas in-place upgrades are fairly simple, as it only involves replacing the packages and stopping and restarting Samba. Advanced Samba users, who join DCs regularly, will have more confidence in using the rejoin approach successfully, and should know how to recover the DC if something goes wrong.
  • Downgrades. Using DC rejoins means that you can always safely rollback to an older Samba release, as the database gets completely recreated using the older database format. Whereas an in-place downgrade requires you to downgrade the database format manually.


Database compatibility when downgrading

When downgrading between major Samba releases, you must use one of the following approaches:

  1. Rejoin the DCs one by one on the older Samba release.
  2. For in-place downgrades, you must downgrade the database manually before rolling back your Samba packages.
  3. Or if nothing has changed since you upgraded, you could restore the backup-file you made prior to upgrading. Note that you can only restore a single DC - the remaining DCs will then have to rejoin the restored DC.

The rest of this section covers the in-place manual downgrade in more detail.


Manual downgrade tool

The tool to manually downgrade your database has changed its name over time. It was originally called 'sambaundoguididx', but then in v4.11 was given the slightly more generic name 'samba_downgrade_db'. The table below shows you the correct name of tool to run, as it depends on which Samba version you're downgrading from.

Going to Going from
4.11 4.10 4.9 4.8 4.7
4.11 - - - - -
4.10 samba_downgrade_db - - - -
4.9 samba_downgrade_db OK - - -
4.8 samba_downgrade_db OK OK - -
4.7 samba_downgrade_db sambaundoguididx sambaundoguididx sambaundoguididx -

Downgrading between some major versions is OK (for example going from 4.10 to 4.9). However, note that running the downgrade tool should always be safe. If in doubt, just run the tool before downgrading.

Make sure you stop samba before running the downgrade tool.

How it works

The Samba AD database format was changed in v4.8, and again in v4.11. When Samba encounters an older database format, it repacks and reconfigures the database to use the latest format. This generally happens when the samba executable first starts.

The Samba v4.7 database format is used as the 'gold standard' to rollback the database to, as any prior release can understand that format. The sambaundoguididx and samba_downgrade_db tools both achieve the same thing, by rolling the database back to the v4.7 format. If necessary, any later version of Samba can handle automatically upgrading a v4.7-format database safely.

For example, to downgrade from 4.11 to 4.10 you should first stop samba, run the samba_downgrade_db tool, then downgrade the samba packages. The AD database is now in v4.7 format, so can any Samba release can safely load it. When you start the v4.10 samba executable, it will automatically upgrade the database from v4.7 format to v4.10 format (changing to use GUID-based database indexes). While this database reindex occurs, the domain controller will be running but out of operation for a brief period of time (auto-upgrade should take a few minutes at most for even very large instances).

Note that you should avoid making any other modifications to the database between running 'samba_downgrade_db' and starting the downgraded samba executable.

Downgrading multiple DCs

If you're downgrading multiple DCs, it may be simplest to run samba_downgrade_db on the primary DC and then rejoin the remaining DCs. Rejoining causes the local database to be re-created (via replication) in the older database format.

If your network has FSMO roles split throughout different DCs, you would need to move all the FSMO roles to a single DC so only that DC needs a database format downgrade. The other DCs can simply be demoted and rejoined after the downgrade.

Errors

Incorrect binary packing format

The following error occurs when starting Samba with the wrong database formats. The error message which is currently present is:

ldb: Invalid data for index  DN=@BASEINFO

===============================================================
INTERNAL ERROR: Signal 11 in pid XXXX (4.9.XXX)
Please read the Trouble-Shooting section of the Samba HOWTO
===============================================================

Make sure to run the samba_downgrade_db script before starting Samba with this version. In future, patched versions of older major releases may have slightly differing error messages.

Python 2 vs Python 3

Traceback (most recent call last):
  File "source4/scripting/bin/samba_downgrade_db", line 10, in <module>
    import samba
  File "bin/python/samba/__init__.py", line 28, in <module>
    import ldb
ImportError: dynamic module does not define module export function (PyInit_ldb)

If the following error occurs, try using a different version of Python than the one set by default.

Samba failed to prime database, error code 22

If this happens after downgrading Samba, it means your database was either provisioned with the version of Samba you downgraded from, or the database was upgraded by that version (or a higher one). Hopefully you made a backup of your database before upgrading, in which case you restore it. Otherwise, you can return to the version of Samba you downgraded from and run the samba_downgrade_db script. If you're using the installed version (/usr/local/samba/sbin/samba_downgrade_db) it should just work. If you're in the source tree, you can provide it with either:

1. Path to target database, eg: "-H /usr/local/samba/private/sam.ldb" (might be different path)

or

2. Your smb.conf file, eg: "-s /usr/local/etc/smb.conf" (might be different path)

The script won't work on the version of Samba experiencing the error because it doesn't know how to read the new database format. You have to upgrade Samba, downgrade the database, then downgrade Samba. In the future, backup your database before upgrading to avoid this hassle.

Older database schemas

One final point is that compatibility problems can occur at a higher level, with the database contents themselves. Specifically, at the AD schema and preparation level for older databases. This will only really be a problem if you migrated from an older Windows domain (2008 functional level or earlier) to Samba.

The bulk of Samba AD development is tested against a Windows 2008R2 schema (v4.11 now uses 2012R2 as the default for new domains). Newer Samba development may sometimes make an assumption that a modern schema is used (i.e. 2008R2 or later). As the older schemas are not tested as thoroughly, you may experience a compatibility problem at some point, especially at upgrade time.

In the long-run, it's worthwhile to upgrade your schema to something more modern. Unfortunately, the Samba code can't update pre-2008R2 schemas. See Windows 2012 Server compatibility for more details on this.