Import of PRD spells, feats, items, monsters

[Farling]

Hello folks,

Just like EightBitz, I've been working with the CSV files from pathfindercommunity.net. (My approach is to use XSLT to do the transformation rather than PowerShell.)

I have all four databases translated into

After a final bug is fixed in the import scripts in theory rwexport files can be created for all the information. However the realm would become stupidly large if you imported all the data, since you would end up with:

2,766 feats (5.8 MB import file)
5,271 monsters (47.4MB import file)
2,713 spells (11MB import file)
3,901 items (22MB import file)

On a test import, it took several hours to import all the data, and the master.realm increased in size by 5 or more GB!

The most obvious filter to start with is to make all the mythic feats a separate import, but how would all the other mass of topics be broken down to something more manageable.

The spells could possibly be reduced to maybe three blocks based on spell level. So you could import only the blocks required for your campaign.

A lot of the monsters appear to be named creatures, rather than generic monsters, so removing uniquely named monsters might reduce the database massively.

The intention would be to release the RWEXPORT files directly, since they contain no modifications to the information other than the format in which they are stored.

[daplunk]

Thinks it's time for me to have a good hard look at the D&D 3-3.5 SRDs...

[Merion]

This sounds fantastic! good job!

I'm not familiar with the data or the work involved, so please excuse me, if I propose things that aren't possible or a huge amount of work:

- My first idea would be to break things up by books. Many GMs limit the number of books used in their campaign, so this could be an approach
- Does the data differentiate between paizo and 3rd party material? That could be sensible devision of the data then
- Throwing out mythic feats and names monsters sounds like a good idea.

[Farling]

Originally Posted by Merion

This sounds fantastic! good job!

I'm not familiar with the data or the work involved, so please excuse me, if I propose things that aren't possible or a huge amount of work:

- My first idea would be to break things up by books. Many GMs limit the number of books used in their campaign, so this could be an approach
- Does the data differentiate between paizo and 3rd party material? That could be sensible devision of the data then
- Throwing out mythic feats and names monsters sounds like a good idea.

By book sounds like a good idea :-)

I don't think any of the information is 3rd party, but certainly separating out 3rd party would be good.

[Bidmaron]

Why not let the individual user run excel filters to do his own customization and then let him run the transform on that?

[kbs666]

Are you sure the data is only the PRD? Those numbers seem quite large.

After looking at the Feats CSV it is clear it is not just the PRD. I didn't find any clearly 3rd party feats but I did find a lot of feats from various companions line products which are not in the PRD.

For a first pass I'd trim everything down to the material just in the actual PRD if that is feasible.

[rob]

@Farling and @Eightbitz, please contact me via email about your efforts here. I'd like to coordinate things with you on this undertaking. I'm thrilled that you're doing this, and I want to figure out how to get everything to users in a manner that will work best for everyone.

Please note that I do NOT have PMs enabled. So you'll need to email me. It's my forum name at wolflair.com.

Oh, and I probably won't be able to get back to you until the weekend, so please give me a couple days.

Thanks!

[EightBitz]

Uh oh. Principal's office. ;-)

[rob]

Originally Posted by EightBitz

Uh oh. Principal's office. ;-)

Er, that's not even remotely the situation here. I'm glad you used the smiley.

As you've already begun to tease apart, there are a bunch of complicating factors involved with this project and how you're going about it, many of which could undermine the success of your efforts in a diverse range of ways. I want to work with you guys to navigate those complications and come up with an approach that will work well for everyone involved: you guys, Lone Wolf, and RW users who want to use what you guys are putting together.

So keep working on your respective projects. My desire is to figure out how to make them a success.

[EightBitz]

Originally Posted by rob

Er, that's not even remotely the situation here. I'm glad you used the smiley.

As you've already begun to tease apart, there are a bunch of complicating factors involved with this project and how you're going about it, many of which could undermine the success of your efforts in a diverse range of ways. I want to work with you guys to navigate those complications and come up with an approach that will work well for everyone involved: you guys, Lone Wolf, and RW users who want to use what you guys are putting together.

So keep working on your respective projects. My desire is to figure out how to make them a success.

Ah, Rob, that was the whole point of the principal's office metaphor. You can be summoned to be congratulated for making the honor roll or
to be told you'll be serving detention for the rest of the week. It can have either connotation, and you never know which one until you walk into the office and sit down.

But I think we now know what kind of a student you were. ;-)