Not Sure where to start
 

I want to make a OGL d20 system variant rules set that would change several of the core mechanics heavily. I'm unsure where to begin coding. the changes are big enough that I'm pretty sure I can't just use the inbuilt editor.

I've managed to follow most of the guide but it doesn't help explain things like using check boxes for options, or making systems like 3e's spell casting system. I'm really not sure where in this forum to go to for help but if I can make progress I was hoping to share my work.

It all depends on how much of a change you need to make. If its just a campaign setting then you might be able to use the Editor to create what you need. If its a huge system change it might require a huge effort and then could require the Authoring Kit.

The big difference between the two products is that the Authoring Kit is used to create brand new systems and the Editor is to make changes to existing systems.

For instance I used the Editor for my Dragonstar system, but there are a few features that I have had to fudge to get to work. I could have used the Authoring kit but then I would have had to code the entire system from the group up.