• Please note: In an effort to ensure that all of our users feel welcome on our forums, we’ve updated our forum rules. You can review the updated rules here: http://forums.wolflair.com/showthread.php?t=5528.

    If a fellow Community member is not following the forum rules, please report the post by clicking the Report button (the red yield sign on the left) located on every post. This will notify the moderators directly. If you have any questions about these new rules, please contact support@wolflair.com.

    - The Lone Wolf Development Team

Import of PRD spells, feats, items, monsters

F

Farling

Well-known member
Hello folks,

Just like EightBitz, I've been working with the CSV files from pathfindercommunity.net. (My approach is to use XSLT to do the transformation rather than PowerShell.)

I have all four databases translated into

After a final bug is fixed in the import scripts in theory rwexport files can be created for all the information. However the realm would become stupidly large if you imported all the data, since you would end up with:

2,766 feats (5.8 MB import file)
5,271 monsters (47.4MB import file)
2,713 spells (11MB import file)
3,901 items (22MB import file)

On a test import, it took several hours to import all the data, and the master.realm increased in size by 5 or more GB!

The most obvious filter to start with is to make all the mythic feats a separate import, but how would all the other mass of topics be broken down to something more manageable.

The spells could possibly be reduced to maybe three blocks based on spell level. So you could import only the blocks required for your campaign.

A lot of the monsters appear to be named creatures, rather than generic monsters, so removing uniquely named monsters might reduce the database massively.

The intention would be to release the RWEXPORT files directly, since they contain no modifications to the information other than the format in which they are stored.
 
This sounds fantastic! good job!

I'm not familiar with the data or the work involved, so please excuse me, if I propose things that aren't possible or a huge amount of work:

- My first idea would be to break things up by books. Many GMs limit the number of books used in their campaign, so this could be an approach
- Does the data differentiate between paizo and 3rd party material? That could be sensible devision of the data then
- Throwing out mythic feats and names monsters sounds like a good idea.
 
Merion said:
This sounds fantastic! good job!

I'm not familiar with the data or the work involved, so please excuse me, if I propose things that aren't possible or a huge amount of work:

- My first idea would be to break things up by books. Many GMs limit the number of books used in their campaign, so this could be an approach
- Does the data differentiate between paizo and 3rd party material? That could be sensible devision of the data then
- Throwing out mythic feats and names monsters sounds like a good idea.
Click to expand...

By book sounds like a good idea :-)

I don't think any of the information is 3rd party, but certainly separating out 3rd party would be good.
 
Last edited:
Why not let the individual user run excel filters to do his own customization and then let him run the transform on that?
 
Are you sure the data is only the PRD? Those numbers seem quite large.

After looking at the Feats CSV it is clear it is not just the PRD. I didn't find any clearly 3rd party feats but I did find a lot of feats from various companions line products which are not in the PRD.

For a first pass I'd trim everything down to the material just in the actual PRD if that is feasible.
 
@Farling and @Eightbitz, please contact me via email about your efforts here. I'd like to coordinate things with you on this undertaking. I'm thrilled that you're doing this, and I want to figure out how to get everything to users in a manner that will work best for everyone.

Please note that I do NOT have PMs enabled. So you'll need to email me. It's my forum name at wolflair.com.

Oh, and I probably won't be able to get back to you until the weekend, so please give me a couple days. :)

Thanks!
 
I think the creator/maintainer for the databases is a user who goes bv "Chopswil" on the Paizo forums.

Separating by source is reasonable.

I would not want to exclude the Player Companion- or Campaign Setting- sourced items unless necessary. I believe all of the rules content is Open. Paizo generally only posts the content from the "Pathfinder Role Playing Game" book line on the PRD (the world-neutral main rule books line); with the exception of the Technology Guide, nothing from the Player Companion or Campaign Setting line has been posted to the PRD ... but that does not make the content Closed. These amount to a world-specific set of items that some GMs running that world would very much want.. while others may not. So separating by source would help them to make those decisions.
 
Yes, the CSV files appear to be more of the "community use" availability rather than "open gaming license".

They are also missing the core bestiary monsters, so I may look for a better source anyway :-)

But it is time to visit the principle's office - fortunately I'm busy all weekend so don't need to worry about not working on it over the weekend.
 
Community Use Policy vs. Open Gaming License.

The rules content needs to be OGL to be used in the databases. The CUP affects how the names appear. PathfinderCommunity.net appears to be affiliated with d20pfsrd, which lost the rights to use the CUP when it became a commercial site. The rules content remains OGL regardless, but the use of Golarion-specific names in rules elements (such as named of countries, organizations, or NPCs, etc. in the names and requirements of Feats, Traits, Prestige Classes, etc.) is allowed under the CUP but not under OGL. So d20pfsrd has to re-name content that uses the protected names, and drop requirements that use the protected names.

This leads to occasional frustration for HeroLab users relying on d20pfsrd... they don't realize that the names were altered, and so can't find the d20pfsrd names in HeroLab (because they are present under their 'proper' names). I don't know which names are in the pathfindercommunity.net files.
 
Silveras said:
I think the creator/maintainer for the databases is a user who goes bv "Chopswil" on the Paizo forums.

Separating by source is reasonable.

I would not want to exclude the Player Companion- or Campaign Setting- sourced items unless necessary. I believe all of the rules content is Open. Paizo generally only posts the content from the "Pathfinder Role Playing Game" book line on the PRD (the world-neutral main rule books line); with the exception of the Technology Guide, nothing from the Player Companion or Campaign Setting line has been posted to the PRD ... but that does not make the content Closed. These amount to a world-specific set of items that some GMs running that world would very much want.. while others may not. So separating by source would help them to make those decisions.
Click to expand...
The PRD material should be redistributable without issue as long as no one charges for it. The player companion and campaign setting stuff is not. At the very least it should be broken out into a second package so people who only want the "core" material don't get it.
 
Bastards of Golarion said:
Product Identity: The following items are hereby identified as Product Identity, as defined in the Open Game License version 1.0a, Section 1(e), and are not Open Content: All trademarks, registered trademarks, proper names (characters, deities, etc.), dialogue, plots, storylines, locations, characters, artwork, and trade dress. (Elements that have previously been designated as Open Game Content or are in the public domain are not included in this declaration.)

Open Content: Except for material designated as Product Identity (see above), the game mechanics of this Paizo Publishing game product are Open Game Content, as defined in the Open Gaming License version 1.0a Section 1(d). No portion of this work other than the material designated as Open Game Content may be reproduced in any form without written permission.
Click to expand...

This is what I mean. The rules content is Open Content, and freely distributable. Every Player Companion and Campaign Setting book has such a declaration, and they are pretty much all the same.

Just because Paizo does not put it in the PRD does not mean it is Protected content. However, the restrictions on using the names may be an issue that forces renaming some items. The Community Use Policy allows the use of Protected content (such as names, but also art) within limits and according to the terms of the policy. The wrinkle here may be the requirement that any distribution of the OGL content requires that a human-readable copy of the OGL itself with its "Section 15" (copyright listing) be included with the distribution. That's more likely to be an issue than the actual content.

I certainly agree that separation by source for those who do not want to include world-specific content is advisable.
 
Look at it this way, it isn't going to be feasible for most people to import all that stuff to anything other than a Golarian realm and even then I think they're going to have to pick and choose. 5 GB of mechanics data? That's a lot of overhead for very little gain.
 
kbs666 said:
Look at it this way, it isn't going to be feasible for most people to import all that stuff to anything other than a Golarian realm and even then I think they're going to have to pick and choose. 5 GB of mechanics data? That's a lot of overhead for very little gain.
Click to expand...

I don't think it is 5 Gb of data. The growth of the MASTER.realm file is problematic. I noticed, and reported, excessive growth. I noticed it while doing some exports and imports of my own.

Exporting and importing something between 20% and 33% (at most, probably much closer to the 20% figure) of my 1.3 GB data caused the file to grow from 1.3 GB to 7.5 GB, Prior to "shrinking" the db down to 1.3 GB, it had been as large as 16.8 GB ... and of that, 15.3 GB was empty/wasted space, probably from other export/import testing. That 15.3 GB was "recovered" by deleting my local MASTER.realm and re-syncing the data from the cloud to local.
 
Silveras said:
Exporting and importing something between 20% and 33% (at most, probably much closer to the 20% figure) of my 1.3 GB data caused the file to grow from 1.3 GB to 7.5 GB, Prior to "shrinking" the db down to 1.3 GB, it had been as large as 16.8 GB ... and of that, 15.3 GB was empty/wasted space, probably from other export/import testing. That 15.3 GB was "recovered" by deleting my local MASTER.realm and re-syncing the data from the cloud to local.
Click to expand...

My test was on a single import of the converted PRD stuff, which could have been ~5,000 topics, which seemed to increase the master.realm up by more than 5GB from an empty master.realm before starting.
 
EightBitz said:
Uh oh. Principal's office. ;-)
Click to expand...

Er, that's not even remotely the situation here. I'm glad you used the smiley. :)

As you've already begun to tease apart, there are a bunch of complicating factors involved with this project and how you're going about it, many of which could undermine the success of your efforts in a diverse range of ways. I want to work with you guys to navigate those complications and come up with an approach that will work well for everyone involved: you guys, Lone Wolf, and RW users who want to use what you guys are putting together. :)

So keep working on your respective projects. My desire is to figure out how to make them a success.
 
rob said:
Er, that's not even remotely the situation here. I'm glad you used the smiley. :)

As you've already begun to tease apart, there are a bunch of complicating factors involved with this project and how you're going about it, many of which could undermine the success of your efforts in a diverse range of ways. I want to work with you guys to navigate those complications and come up with an approach that will work well for everyone involved: you guys, Lone Wolf, and RW users who want to use what you guys are putting together. :)

So keep working on your respective projects. My desire is to figure out how to make them a success.
Click to expand...

Ah, Rob, that was the whole point of the principal's office metaphor. You can be summoned to be congratulated for making the honor roll or
to be told you'll be serving detention for the rest of the week. It can have either connotation, and you never know which one until you walk into the office and sit down.

But I think we now know what kind of a student you were. ;-)
 
You must log in or register to reply here.
Back
Top