Find the "Custom Abilities" section on the class tab
Custom Ability Count: row 0 is level 1, row 1 is level 2, etc., and the values are the number of abilities available at that level, and you can leave rows blank if they're the same as the row above them, so fill in 1, 2, 3, all the way to 10 in rows 1, 3, 5, etc., up to 10 in row 19.
Custom Ability Name: Rogue Talent
Allow Custom Abilities: check "Rogue" (there are two with this name - choose the second - the one that will say "cHelpRog" if you mouse over the ?, plus "Rogue (Unchained)" and "Ninja" (assuming your class gets the same ninja tricks and the talents that are only available to the unchained rogue, not the base rogue).
Where to learn this:
In the editor, choose Help, and then "Help on using the editor", and then "Adding a Complex Class (Tutorial 4)" or "Adding the Alchemist Class (Tutorial 7)"