Import of PRD spells, feats, items, monsters
 

Hello folks,

Just like EightBitz, I've been working with the CSV files from pathfindercommunity.net. (My approach is to use XSLT to do the transformation rather than PowerShell.)

I have all four databases translated into

After a final bug is fixed in the import scripts in theory rwexport files can be created for all the information. However the realm would become stupidly large if you imported all the data, since you would end up with:

2,766 feats (5.8 MB import file)
5,271 monsters (47.4MB import file)
2,713 spells (11MB import file)
3,901 items (22MB import file)

On a test import, it took several hours to import all the data, and the master.realm increased in size by 5 or more GB!

The most obvious filter to start with is to make all the mythic feats a separate import, but how would all the other mass of topics be broken down to something more manageable.

The spells could possibly be reduced to maybe three blocks based on spell level. So you could import only the blocks required for your campaign.

A lot of the monsters appear to be named creatures, rather than generic monsters, so removing uniquely named monsters might reduce the database massively.

The intention would be to release the RWEXPORT files directly, since they contain no modifications to the information other than the format in which they are stored.

Thinks it's time for me to have a good hard look at the D&D 3-3.5 SRDs...

This sounds fantastic! good job!

I'm not familiar with the data or the work involved, so please excuse me, if I propose things that aren't possible or a huge amount of work:

- My first idea would be to break things up by books. Many GMs limit the number of books used in their campaign, so this could be an approach
- Does the data differentiate between paizo and 3rd party material? That could be sensible devision of the data then
- Throwing out mythic feats and names monsters sounds like a good idea.

By book sounds like a good idea :-)

I don't think any of the information is 3rd party, but certainly separating out 3rd party would be good.

Why not let the individual user run excel filters to do his own customization and then let him run the transform on that?

Are you sure the data is only the PRD? Those numbers seem quite large.

After looking at the Feats CSV it is clear it is not just the PRD. I didn't find any clearly 3rd party feats but I did find a lot of feats from various companions line products which are not in the PRD.

For a first pass I'd trim everything down to the material just in the actual PRD if that is feasible.

@Farling and @Eightbitz, please contact me via email about your efforts here. I'd like to coordinate things with you on this undertaking. I'm thrilled that you're doing this, and I want to figure out how to get everything to users in a manner that will work best for everyone.

Please note that I do NOT have PMs enabled. So you'll need to email me. It's my forum name at wolflair.com.

Oh, and I probably won't be able to get back to you until the weekend, so please give me a couple days. :)

Thanks!

Uh oh. Principal's office. ;-)

I think the creator/maintainer for the databases is a user who goes bv "Chopswil" on the Paizo forums.

Separating by source is reasonable.

I would not want to exclude the Player Companion- or Campaign Setting- sourced items unless necessary. I believe all of the rules content is Open. Paizo generally only posts the content from the "Pathfinder Role Playing Game" book line on the PRD (the world-neutral main rule books line); with the exception of the Technology Guide, nothing from the Player Companion or Campaign Setting line has been posted to the PRD ... but that does not make the content Closed. These amount to a world-specific set of items that some GMs running that world would very much want.. while others may not. So separating by source would help them to make those decisions.

Yes, the CSV files appear to be more of the "community use" availability rather than "open gaming license".

They are also missing the core bestiary monsters, so I may look for a better source anyway :-)

But it is time to visit the principle's office - fortunately I'm busy all weekend so don't need to worry about not working on it over the weekend.