Lone Wolf Development Forums

Lone Wolf Development Forums (http://forums.wolflair.com/index.php)
-   Realm Works Discussion (http://forums.wolflair.com/forumdisplay.php?f=67)
-   -   Foundry loose integration (http://forums.wolflair.com/showthread.php?t=65924)

Farling May 17th, 2021 08:46 AM

Foundry loose integration
Hello all,

I have started development on a Foundry VTT module which will allow you to import some material from Realm Works into Foundry.

The module requires that you generate an RWEXPORT file from Realm Works (not rwoutput).
The contents of this file are then processed by my module to generate entries in your world, stored in a named folder.

It creates Journal Entries from topics, Scenes from smart images (including map pins), Actors from HL portfolios and statblocks, Playlists from audio snippets, and Roll Tables from tables embedded in topics (where the table "looks like" a rollable table).

The module can be found at https://github.com/farling42/fvtt-import-realmworks

Note that due to the restrictions of what Realm Works will allow to be stored in the RWoutput file, it is not possible for you to transfer your RW-purchased modules using this method.

Donations for efforts can be made via any of Ko-Fi or Paypal



- read RWEXPORT from a file (or copy/paste directly into a text box)
- read a Hero Lab portfolio file directly (for PF1, DND5E and Savage Worlds)
- Each topic imported into its own Journal entry, grouped by topic category.
- most normal snippet types
- simple pictures
- statblocks used to create Actor (statblock stored in notes/description panel)
- HL portfolios used to created populated Actors for some game systems, otherwise as per statblock.
- links between RW articles/FVTT journal entries.
- table of contents for each parent showing all descendents (optionally only down to a specific level).
- (optional) inbound/outbound links listed at the end of a topic's entry.
- create Scenes from smart images
- snippet styles and GM-direction formatting transferred to journal entries.
- display annotations and gm_directions in a distinct way
- examine tables in snippets and if appropriate create a Roll Table from that table.


- support more snippet types
- display more game systems for populated Actors (now only done on request)

GM-Directions on map pins are transferred to map Notes.
Create an Actor for EACH character in a HL portfolio file, not just the first one.
For PF1 characters, create sub-skills (e.g. Artistry, Perform, Profession).
Include prefix and suffix (if any) from the RW topic in title of journal entries.
Create thumbnails for each scene.

RW terminology used in standard section headings:
"Child Topics" changed to "Governed Content";
"Connections" changed to "Relationships";
"Links To/From Other Topics" changed to "Content Links: In/Out".
Format of relationships changed slightly, and includes any entered annotations.
README.md file updated with current information.
GM-Directions in topics converted to "secret" sections in journal entries.
Snippet styles change the background colour as per RW.

Snippet style is now displayed for all snippet types; and GM-directions are boxed with their other content.

This version works on both 0.7.9 and 0.8.6. It also reads weapons/ammo/armor/shields from embedded HL files for PF1.

I've added the ability to load just a .por file, so that it create Actors directly from that file.
It also decodes spells for PF1, and the correct number of levels of each class.

Hopefully improved performance for larger files.
Better decoding of Actors for PF1 (Skill Focus modifies the relevant skill; Correct number of monster levels)

Playlists are created from audio snippets. One playlist per topic, one sound within the playlist for each audio snippet within the topic.

Better decoding of Pathfinder 1st edition characters from Realms and HL portfolio files: unknown spells are added, class abilities are added.

Remember the settings selected in the dialogue window.

Fix a problem with not creating some links. I don't know if this is a new bug introduced by RW version 257 (I was searching for only <span> to contain links, but 257 sometimes uses <span class="RWSnippet">. This means searching will be slower since the longer span is used for all forms of text, not just links.

Fixed a problem for new FVTT worlds where the default location for asset files started with world instead of worlds.
ChangeLog and compatibleCoreVersion also updated.

Fixes compatibility with Foundry version 0.7.x.
In 0.7.x all playlists are put at the top level since folders for playlists is a new feature in Foundry 0.8.x

Converts BMP and TIF/TIFF files to PNG format before uploading to Foundry VTT.

A small improvement to place Actors into a sub-folder named after the containing topic. (But with the proviso that if the topic has only one actor AND the name of the actor is the same as the name of the topic, then no sub-folder is created.)

The module tries to store the statblock somewhere in the Actor sheet for ALL game systems.
For PF1, many of the other fields in the Actor sheet are extracted (this has been available for a long while).
For all other game systems, the HTML of the statblock will simply be copied into an appropriate Notes or Biography section of the Actor sheet.

Calculation of HP for imported PF1 actors should be better.
An internal change should also make processing a little faster when there are a LOT of actors in your realm.
Importing a HL .por file directly should work properly again (there was a bug which caused most of the actor data to disappear during creation).

Support the Cypher game system in Foundry VTT by creating "PC" Actors with the statblock put into the Notes section.
When importing a standalone Hero Lab .por file, if full Actor creation isn't supported then just store the HTML statblock(s) in the created Actor(s).

Add module configuration setting to set the preferred Actor type when generating Actors from statblocks.
When decoding PF1 Actors, check for presence of "challengerating" element to choose 'npc' Actor type instead of 'character'.
Better fix for HP calculation of PF1 Actors for Actors with multiple types of hit dice.
Create link at start of each Journal Entry to the parent topic. The label for this link is a configurable parameter.

The "Governed Content" will now contain the full hierarchy of descendents rather than only the direct children.
Provide link from Scene Notes journal entry back to the Scene.

Adds a configuration parameter in the module settings panel so that you can specify the maximum depth of the governed content added to journal entries.

Allow the asset files to be located in S3 storage (courtesy of the DirectoryPicker class from ddb-importer).

Adds decoding of Actor information from D&D 5E portfolio files.
Removes support for 0.7.9

Fixes a bug where the module would fail to start if the D&D 5E game system was not installed on your Foundry server (even though the game system you're importing to isn't D&D 5E).

The disposition on tokens for created Actors of NPCs is set to the HL allegiance when decoding portfolio files.
Decoding of basic "Savage World" HL portfolio files is now available - it supports both FVTT Savage World game system, and will use the "Savage Worlds Core Ruleset" if available.

Snippets are now placed immediately after the section header when the section has sub-sections. (Previously they were put AFTER all sub-sections, and so appeared to be part of the last sub-section.)

Ensure that ALL portfolios + statblocks in a topic are converted into Actors (instead of just the first one).
Prevent error when filename is missing for a portfolio or statblock.
Improve error logging to the console, particularly during actor creation.
Do NOT create actors in a topic when the actors are actually stored in a child topic (they are created for the child topic only).
The name of Actors created from Statblocks will now use the name from the snippet, and appended with the annotation if present (preceded by ":").
Fix an issue with processing a .por directly would fail if a character in the file had minions.

Links to the parent topic, and links to child topics, will now only contain the public name of the topic and will NOT include all the aliases.
Folders will be created with sorting set to MANUAL rather than AUTOMATIC, so that items in the folder retain the same order as in RW.
Speed up creation of Scenes by creating all Notes in a single call to the server.

Updates to PF1 Actor creation:
Attempt to automatically populate the Combat tab of Actors from all the weapons in their inventory.
Detect "masterwork" and "+X" weapons and armour (doesn't work if the item name has other item properties in it).

PF1 Actor decoding has more information in COMBAT tab:
Non-weapon attacks now added to COMBAT tab in the "Natural Attacks" section.
Special Attacks (probably from classes & feats) are added to the MISC section of the COMBAT tab.

Adds support for decoding Call of Cthulhu portfolios into Actors.
It works much better if the Foundry compendium and/or Items Directory contain definitions for weapons, skills, books and spells.

GIF images are converted to PNG during import, primarily so that they can be used for scene backgrounds.
Governed Content and Governing Content links now use the FULL topic name (prefix + title + suffix).
Re-ordered code so that scene Notes are always saved in the world database.

Fixes a problem where the governing/governed content links would not get created for any more children of a topic after a child with an & in the title was encountered.

RollTables are created from Tables within your RW snippets.
BUG FIX for Actors not being created from Statblock snippets.

Formatting is retained where appropriate.
Links and in-line dice rolls are created within the RollTable result details where applicable.
The first row of the table is assumed to be titles, and will not be added to the RollTable.
The first column of the table must contain the rolled number(s), either as a single number, two numbers separated by a dash, or two numbers separated by a comma.
The second column of the table is used to populate the RollTable (although if the second column contains all blank entries then the third column will be used instead).
If the title of the first column is a valid dice roll syntax then it will be used as the formula for the table, otherwise the formula will be calculated by examining the lowest and highest numbers in the table and creating a single dice roll with a fixed addition.
A link to the RollTable is added to the end of the converted section within the corresponding Journal Entry.

Fixes an issue where links would fail to a topic with an ampersand (&) in the title.

In transitioning to rwexport files, links will always be created properly.
True Names will be identified in the list of aliases for each topic.
The Category of each topic is added at the top of each journal entry.
(More changes will appear in the future based on information only available in rwexport files.)

Fixes a bug where links were mis-aligned with the text when a table is present in a snippet.

Governed Content and Content Links are sorted alphabetically (with numbers sorted properly).
It also provides a better warning message when a link is to a topic not found in the rwexport file.

Fix the labels on map pins.
Don't include aliases in the name of RollTables.
Remove old code that supported RWoutput files.

Improvement the layout of Connections.
They are sorted alphabetically using the name of the linked topic.
All components of the description of the connection are presented as a prefix to the topic link.

Two new processing modes:
Only import NEW topics - only NEW topics in the file will be used to create things in your Foundry world.
Overwrite previously imported things - this will keep the same IDs (except for Actors, which will be deleted and new ones created); this allows links that you've created yourself with Foundry to remain valid. This option will NOT delete any world things that you might have deleted from within your Realm.
Also: d% should be converted to d100 in roll tables.

Fixes an issue where some relationships were labelled as "undefined", and fixes the text for some other relationships to match what is seen in RW.

FIX: Custom section names should now be decoded properly.
IMPROVEMENT: bold and italic text now encoded as strong and em text (to match how Foundry VTT does it).
FEATURE: Add option to include numbering on section headers, emulating the numbering in Realm Works.
CHANGE: Connections and Content Links now use same topic name format as in Realm Works "topicname ( suffix - prefix )", rather than the format used in topic titles and governed content "prefix - topicname (suffix)".
FEATURE: Add option to handle REVEALED state of things. If a topic is revealed, then its contents will be only the snippets that are revealed, and actors/tables/playlists will only be created from that topic if the containing snippets are revealed, and the topic will have its permissions set for players to be OBSERVERS. For topics which are not revealed, the full contents will be created (and the observer will NOT be set).

When importing using the REVEALED state of topics, the journal entry will contain only the revealed snippets (links to unrevealed topics will appear as normal text, not links).
The FULL text of imported topics is available by using the "GM Notes" module for Foundry VTT (available from the normal "add-on modules" window in the Foundry VTT Configuration & Setup window).

Configuration options are now available for scene padding and grid size.

FIX: In revealed topics, the content links should only show links to the topics where the snippets/map_pins containing the link have been revealed.
FEATURE: Initial attempt at decoding some journal entries as Items instead of Journal Entries (currently only for PF1 and English RW category names).

No longer requires "GM Notes" module for handling revealed topics.
Revealed snippets are identified to the GM in displayed documents by a green bar on the left side.
Revealed smart images can have their corresponding scene automatically made navigable (and their vision/fog disabled).
For GMs, Relationships, Content Links and Governed Content will show REVEALED parts separately from the full list.
Lots of CSS have been added to more closely match the Snippet Style in RW.
Created Scenes, Actors and Tables are only made OBSERVABLE to players if the topic AND the snippet are both revealed.
Snippets with both GM notes and non-GM notes will be grouped together in a box for GMs (in a similar presentation to RW).
Known Bugs
Revealed state of map pins is ignored - only pins connected to non-revealed topics will be hidden on each scene.
All links are displayed in revealed snippets of revealed topics - even if the linked document is not revealed.

Fixed a bug that prevented actors being created from HL portfolio files

Only revealed map pins will be visible to players in scenes. For GMs, not-revealed pins will be displayed with a "blind" icon.
When displaying document text in journal entries and actors, any links whose destination is NOT visible to a player will have the link replaced by just the normal text - so a player will NOT see any link there.

Don't include GM Directions in Notes that are displayed to players.
Account for snippet annotations that might have links in them.
Veracity (Truth) of snippets is displayed to GMs (dashed border for Partial Truth, solid border for Lie)

Removes a lot of the colourful styling that was added in 1.10.
Adds block styles to the text editor for RW Veracity (Lie/Partial Truth) and Style (Callout, Handout, Flavor, Read-Aloud) and GM Directions.

Provides compatibility with Monk's Enhanced Journal to allow GM-only formatting to be seen (Veracity and GM-directions).

Remove occurrences of secret sections having double the darkness applied to its background.
Fix an issue where section headings were set/not set with 'secret' which did not match that state of its contents.
Put GM Directions & Contents above GM Directions in the text editor menu, and remove Block: from those entry names.
Further reduce the amount of HTML generated for each journal entry (faster world loading!)
Ensure new text block names are less likely to conflict with other modules (internal to the code).
Add a new configuration Use 'secret' in Unrevealed Topics, which when ticked will do the normal topic generation, but if unticked then topics which are NOT revealed in RW will NOT have any secret sections in their corresponding Journal Entries. This is so that the journal entry doesn't end up with large portions of it using the secret (darker) background.

The RW-specific styles are now immediately displayed in the text editor (TinyMCE).
You no longer have to save the changes before seeing the results.

Separate out the Player visible styles and the GM-only styles in the Text Editor.
Fix issue where you needed to add "GM Directions" and secret separately, and they were not getting merged into a single section.

The secret style is always added to GM Directions during import.
If the secret style is manually removed from a GM Directions paragraph then an inline warning will be displayed to the GM indicating that the GM directions might be visible to players.
To restore the situation, the user can manually deselect the "GM Directions (not secret)" on the paragraph(s) and then reselect "GM Directions (secret)" for them.

The first image in a topic will be set as the image for the created journal entry.
The module will now also work in Foundry V9 prototype 2.
(Note that Foundry will generate a warning about version compatibility, but it does work)

Changed the permission required for players to see links to LIMITED (previously it was OBSERVER).
Updated the compatibleCoreVersion to 9 so that no warnings are generated on Foundry V9.

Updates for scene Notes:
- The title of a map pin is now displayed within ">> title <<" instead of the previous "\*\*title\*\*".
- The text is broken across multiple lines (default max line length set to 60 in module settings) - this does not affect the note title/linked journal name.
- Multiple occurrences of "
" in the map pin descriptions are replaced by just a newline.
- The "GMDIR:" indicator has been put on it's own line with down arrows to indicate which text is the GM directions.
- The font size is now configurable in the module settings.
- The display of GM notes has been changed so that less of the core foundry code has been copied into this module.

Change the presentation of annotations for relationships since they can be multi-line (whereas annotations on other snippets are single line only). The annotation of a relationship now appears in emphasis (italics) immediately below the relationship details (still part of the same bullet).
Add a soft (optional) dependency on libWrapper, just in case any other modules want to interact with notes and/or journal links.
Only enable the relevant notes/links/gmtext sub-modules depending on whether logged in as GM or player.
Revealed map pins will always have the scene Notes be displayed to players even if the linked journal entry is not visible to the player. If the linked journal entry is NOT visible to the user then the Note will be displayed to the non-GM player as if it was a not-connected note. The tint colour of the icon will be set correctly based on whether or not the player can see the linked journal entry.
For users hosting on Forge, additional metadata has been added to the module information.
Use correct constants for token disposition.

Remove debug statement that was being sent to your browser console as a warning.

Update Note Config window to provide multi-line text entry areas for Note Label and GM Notes.
Update Note Config window to allow revealed state of scene Notes to be changed.
Add two new icon types (Boxed Circle and Crossed Eye) to the list of available icons for scene Notes (since these are used to identify imported map pins).
Add module options to set the icon tint colours to be used when displaying notes to players to indicate if they are linked to an accessible journal entry.

1.19 (30/Nov/2021)
The non-core import functions have been moved into separate modules, so you don't need the import module active while running your game.
You will be prompted to install these additional modules when you next update.
- GM-only text for Notes
- Disguise unreachable links
- Optional "reveal" state for Notes
These modules will work stand-alone, as well as the import module setting up the imported information to work with these modules.

1.20 (23/Jan/2022
Adds a simple option to convert all images to WEBP format.
The following image file formats will be converted: gif, png, jpg, jpeg, bmp, tif, tiff

1.21 (25/Jan/2022)
Ensure description is set properly when creating Items.
Add image to Items if available in the realm works input file.
Table ranges which end with just zeros (like 56-00) will have the upper limit calculated as it is is preceded with a 1, (e.g. 56-100).

1.22 (27/Jan/2022)
PF-Traits and PF-Racial-Traits are now processed for Traits.
PF-Feats are now also checked for Feats.
PF-Magic Items are now processed properly again for magic items.
"Spells Memorized" will be added if there are no "Spells Known" in the Portfolio file.
Try to support trait names in the "Pathfinder 1e content" where the trait name has the book source appended to it.
Remove racial attribute bonuses from base attribute values (if race is found in a compendium).
KNOWN BUG: weapons and armour with special qualities are not detected (they are listed in the "Miscellaneous" section of the Inventory).
KNOWN BUG: Natural Armor AC is always copied into the "Natural Armor AC" of the Combat tab, even if it comes from a magic item.

nightpanda2810 May 17th, 2021 11:23 AM

I'm sure this is totally surprising and unexpected, but an 800MB realm is a no go with this.

A small fraction of the contents of this file crashes the browser (chrome) itself just trying to paste. Not surprising in the least of course. A 16MB file worked perfectly.

Where do images get imported to? I host on AWS so want to make sure I keep an eye on storage.

I look forward to seeing this progress!

Farling May 17th, 2021 12:56 PM

At the moment the images which are within a topic are stored inside the compendium as part of the journal entry - they are inline in the HTML.

There isn't any facility (currently) to extract the images and store them elsewhere.

One of my more urgent tasks is to enable selecting a file, rather than expecting copy-paste to work. In fact, with Firefox and the 4 MB test file it takes longer to copy-paste into the browser than it takes to do the conversion.

My rwoutput file is 8,012 KB according to Windows Explorer. The realm.db compendium pack file into which it was imported is 6,362 KB in size. But this could change as more features are successfully transferred across.

Farling May 18th, 2021 04:11 AM


Originally Posted by nightpanda2810 (Post 295053)
I'm sure this is totally surprising and unexpected, but an 800MB realm is a no go with this.

A small fraction of the contents of this file crashes the browser (chrome) itself just trying to paste. Not surprising in the least of course. A 16MB file worked perfectly.

Where do images get imported to? I host on AWS so want to make sure I keep an eye on storage.

I look forward to seeing this progress!

Version 0.1.1 allows a file to be selected so that there is no need to use copy/paste.

I tested this with a 426,163 KB file (426 MB) which contains 4901 topics.

The created compendium db file in Foundry has a size of 330,950 KB (331 MB).

nightpanda2810 May 18th, 2021 09:03 AM


Originally Posted by Farling (Post 295072)
Version 0.1.1 allows a file to be selected so that there is no need to use copy/paste.

I tested this with a 426,163 KB file (426 MB) which contains 4901 topics.

The created compendium db file in Foundry has a size of 330,950 KB (331 MB).

Awesome! Updated and attempted to import my file, 773MB. Progress says "loading" for a few seconds then changes to "please enter some data". Is this just a matter of giving it some time?

Farling May 18th, 2021 02:01 PM


Originally Posted by nightpanda2810 (Post 295077)
Awesome! Updated and attempted to import my file, 773MB. Progress says "loading" for a few seconds then changes to "please enter some data". Is this just a matter of giving it some time?

I haven't tested it for maximum file size. It is down to the browser (I think) as to how big a file it can handle.

I know that when I tried a 900 MB file it failed to load.

All my browser testing has been using the built-in browser that comes with the Windows desktop version of Foundry. I haven't tested it with Firefox or Chrome as standalone browsers.

Happydevil43 May 18th, 2021 02:05 PM

Hi Farling,

This is really good, as an aside, it might be worth reaching out to the guys at Arkenforge, their new encyclopedia system might work well benefiting from some sort of importer, and they have been thinking of doing a RW importer type tool for some time.

Their new encyclopedia is a vast step over the first iteration.


Farling May 18th, 2021 02:07 PM


Originally Posted by Happydevil43 (Post 295082)
Hi Farling,

This is really good, as an aside, it might be worth reaching out to the guys at Arkenforge, their new encyclopedia system might work well benefiting from some sort of importer, and they have been thinking of doing a RW importer type tool for some time.

Their new encyclopedia is a vast step over the first iteration.


I'm not trying to do LWD out of business by writing a tool to import data into a competing world building tool. Foundry is primarily a VTT and doesn't really have the same capabilities as a world building tool.

Farling May 18th, 2021 02:13 PM

Using a simple tool at https://stackoverflow.com/questions/...iable/33768639, the maximum capacities for a string for each browser are:

Firefox = 1,073,741,822 (1 GB)
Chrome = 536,870,888 (500 MB)

The entire file is read into a single string, so these limits will apply to the maximum allowed file size.


I just ran a quick test on Firefox. It loaded a 900 MB file, but failed to load a javascript library which I use for reading portfolio files. So more testing is required...

With a fix for that problem, I see another error when trying to process a large XML file...

"Error: Cannot create a string longer than 0x1fffffe8 characters"

That hex number is 536,870,888 bytes, which is 24 bytes less than 512 MB.

Farling May 18th, 2021 03:12 PM


Originally Posted by nightpanda2810 (Post 295077)
Awesome! Updated and attempted to import my file, 773MB. Progress says "loading" for a few seconds then changes to "please enter some data". Is this just a matter of giving it some time?

I need to change the message here. This is an indication that the file is too large.

All times are GMT -8. The time now is 03:46 PM.

Powered by vBulletin® - Copyright ©2000 - 2022, vBulletin Solutions, Inc.
wolflair.com copyright 1998-2016 Lone Wolf Development, Inc. View our Privacy Policy here.