Here's a general breakdown:
All subclasses (archetypes, domains, schools, etc.) are created on the Custom Ability Tab.
Most of the subclass features will be created using the Class Special tab.
The following class/subclass features are created on the Custom Ability tab:
- Barbarian Totems, Aspects and Totemic Attunements
- Fighter Battle Master Maneuvers and Arcane Shots
- Monk Elemental Traditions
- Ranger Favored Enemy & Terrain - ALL Hunter subclass features (Prey & Tactics)
- Sorcerer Metamagic
- Warlock Eldritch Invocation and Pact Boons
It's hard to explain why some of the features are considered Custom Abilities, and some are considered Class Specials aside from if the class ability has a multiple choice it's a Custom Ability and if it's a fairly static entry, it uses Class Special.
Hope that helps.