The customdc is a special testenv that's only used for manual testing, rather than the automated tests most testenvs are primarily used for. Customdc allows you to specify any backup-file to load into the new testenv.
This feature is primarily aimed at developers. It allows you to save particular Samba configurations or databases, and easily reload them later.
To run the testenv, you need to specify a 'BACKUP_FILE' shell variable, e.g.
BACKUP_FILE=/tmp/samba-backup-50k-dc-0-mdb-50k-offline.tar.bz2 SELFTEST_TESTENV=customdc make testenv
The main use-case for the customdc is testing changes against a large database. Adding users is very time-consuming, so it's much quicker to populate a domain with users once, take a backup, and then you can spin up a testenv based on the backup multiple times.
Another use-case is that if you get a database that's corrupted or in a bad state, then you could save a backup and be able to easily get the database back into the bad state. This allows you to try different commands to diagnose/fix the issue, without fear of never seeing the problem again.
You could even spin up a 'lab DC' inside a testenv, by taking a backup of a real network DC.