Name the custom ancestry the name of the NPC. For Example, in Age of Ashes I created a Charu-ka Dragon Priest with a custom ancestry called Charu-ka Dragon Priest. When you create a NPC from the vault you'll see that the Ancestry is the name of the creature.
Regarding class, you won't need to select one because you'll be using the Permanent Adjustments and Permanent Added Abilities to set values for almost everything that isn't covered by the custom ancestry settings. The NPC will not be valid, meaning you'll have the red exclamation on the top right of the screen.
Here are
some screenshots that might be helpful.