Lone Wolf Development Forums  

Go Back   Lone Wolf Development Forums > Realm Works Forums > Realm Works Discussion

Notices

Reply
 
Thread Tools Display Modes
Farling
Senior Member
 
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,486

Old May 17th, 2021, 08:46 AM
Hello all,

I have started development on a Foundry VTT module which will allow you to import some material from Realm Works into Foundry.

The module requires that you generate an RWoutput file from Realm Works (not RWexport, but RWoutput).

The contents of this file are then processed by my module to generate entries in your world, stored in a named folder.

It creates Journal Entries from topics, Scenes from smart images (including map pins), Actors from HL portfolios and statblocks, and Playlists from audio snippets.

The module can be found at https://github.com/farling42/fvtt-import-realmworks

Note that due to the restrictions of what Realm Works will allow to be stored in the RWoutput file, it is not possible for you to transfer your RW-purchased modules using this method.

Donations for efforts can be made via any of Ko-Fi or Paypal

----

Working:

- read RWoutput from a file (or copy/paste directly into a text box)
- read a Hero Lab portfolio file directly (for PF1, DND5E and Savage Worlds)
- Each topic imported into its own Journal entry, grouped by topic category.
- most normal snippet types
- simple pictures
- statblocks used to create Actor (statblock stored in notes/description panel)
- HL portfolios used to created populated Actors for some game systems, otherwise as per statblock.
- links between RW articles/FVTT journal entries.
- table of contents for each parent showing all descendents (optionally only down to a specific level).
- (optional) inbound/outbound links listed at the end of a topic's entry.
- create Scenes from smart images
- snippet styles and GM-direction formatting transferred to journal entries.
- display annotations and gm_directions in a distinct way

TBD:

- support more snippet types
- display more game systems for populated Actors (now only done on request)

0.2.4
GM-Directions on map pins are transferred to map Notes.
Create an Actor for EACH character in a HL portfolio file, not just the first one.
For PF1 characters, create sub-skills (e.g. Artistry, Perform, Profession).
Include prefix and suffix (if any) from the RW topic in title of journal entries.
Create thumbnails for each scene.

0.2.5
RW terminology used in standard section headings:
"Child Topics" changed to "Governed Content";
"Connections" changed to "Relationships";
"Links To/From Other Topics" changed to "Content Links: In/Out".
Format of relationships changed slightly, and includes any entered annotations.
README.md file updated with current information.
GM-Directions in topics converted to "secret" sections in journal entries.
Snippet styles change the background colour as per RW.

0.2.6
Snippet style is now displayed for all snippet types; and GM-directions are boxed with their other content.

0.3.3
This version works on both 0.7.9 and 0.8.6. It also reads weapons/ammo/armor/shields from embedded HL files for PF1.

0.3.4
I've added the ability to load just a .por file, so that it create Actors directly from that file.
It also decodes spells for PF1, and the correct number of levels of each class.

0.3.5
Hopefully improved performance for larger files.
Better decoding of Actors for PF1 (Skill Focus modifies the relevant skill; Correct number of monster levels)

0.3.6
Playlists are created from audio snippets. One playlist per topic, one sound within the playlist for each audio snippet within the topic.

0.3.7
Better decoding of Pathfinder 1st edition characters from Realms and HL portfolio files: unknown spells are added, class abilities are added.

0.3.8
Remember the settings selected in the dialogue window.

0.3.9
Fix a problem with not creating some links. I don't know if this is a new bug introduced by RW version 257 (I was searching for only <span> to contain links, but 257 sometimes uses <span class="RWSnippet">. This means searching will be slower since the longer span is used for all forms of text, not just links.

0.3.11
Fixed a problem for new FVTT worlds where the default location for asset files started with world instead of worlds.
ChangeLog and compatibleCoreVersion also updated.

0.3.12
Fixes compatibility with Foundry version 0.7.x.
In 0.7.x all playlists are put at the top level since folders for playlists is a new feature in Foundry 0.8.x

0.3.13
Converts BMP and TIF/TIFF files to PNG format before uploading to Foundry VTT.

0.3.15
A small improvement to place Actors into a sub-folder named after the containing topic. (But with the proviso that if the topic has only one actor AND the name of the actor is the same as the name of the topic, then no sub-folder is created.)

0.3.16
The module tries to store the statblock somewhere in the Actor sheet for ALL game systems.
For PF1, many of the other fields in the Actor sheet are extracted (this has been available for a long while).
For all other game systems, the HTML of the statblock will simply be copied into an appropriate Notes or Biography section of the Actor sheet.

0.3.17
Calculation of HP for imported PF1 actors should be better.
An internal change should also make processing a little faster when there are a LOT of actors in your realm.
Importing a HL .por file directly should work properly again (there was a bug which caused most of the actor data to disappear during creation).

0.3.18
Support the Cypher game system in Foundry VTT by creating "PC" Actors with the statblock put into the Notes section.
When importing a standalone Hero Lab .por file, if full Actor creation isn't supported then just store the HTML statblock(s) in the created Actor(s).

0.3.19
Add module configuration setting to set the preferred Actor type when generating Actors from statblocks.
When decoding PF1 Actors, check for presence of "challengerating" element to choose 'npc' Actor type instead of 'character'.
Better fix for HP calculation of PF1 Actors for Actors with multiple types of hit dice.
Create link at start of each Journal Entry to the parent topic. The label for this link is a configurable parameter.

0.3.20
The "Governed Content" will now contain the full hierarchy of descendents rather than only the direct children.
Provide link from Scene Notes journal entry back to the Scene.

0.3.21
Adds a configuration parameter in the module settings panel so that you can specify the maximum depth of the governed content added to journal entries.

0.3.22
Allow the asset files to be located in S3 storage (courtesy of the DirectoryPicker class from ddb-importer).

0.4.0
Adds decoding of Actor information from D&D 5E portfolio files.
Removes support for 0.7.9

0.4.1
Fixes a bug where the module would fail to start if the D&D 5E game system was not installed on your Foundry server (even though the game system you're importing to isn't D&D 5E).

0.5.0
The disposition on tokens for created Actors of NPCs is set to the HL allegiance when decoding portfolio files.
Decoding of basic "Savage World" HL portfolio files is now available - it supports both FVTT Savage World game system, and will use the "Savage Worlds Core Ruleset" if available.

0.5.1
Snippets are now placed immediately after the section header when the section has sub-sections. (Previously they were put AFTER all sub-sections, and so appeared to be part of the last sub-section.)

0.5.2
Ensure that ALL portfolios + statblocks in a topic are converted into Actors (instead of just the first one).
Prevent error when filename is missing for a portfolio or statblock.
Improve error logging to the console, particularly during actor creation.
Do NOT create actors in a topic when the actors are actually stored in a child topic (they are created for the child topic only).
The name of Actors created from Statblocks will now use the name from the snippet, and appended with the annotation if present (preceded by ":").
Fix an issue with processing a .por directly would fail if a character in the file had minions.

0.5.5
Links to the parent topic, and links to child topics, will now only contain the public name of the topic and will NOT include all the aliases.
Folders will be created with sorting set to MANUAL rather than AUTOMATIC, so that items in the folder retain the same order as in RW.
Speed up creation of Scenes by creating all Notes in a single call to the server.

0.6.0
Updates to PF1 Actor creation:
Attempt to automatically populate the Combat tab of Actors from all the weapons in their inventory.
Detect "masterwork" and "+X" weapons and armour (doesn't work if the item name has other item properties in it).

0.6.1
PF1 Actor decoding has more information in COMBAT tab:
Non-weapon attacks now added to COMBAT tab in the "Natural Attacks" section.
Special Attacks (probably from classes & feats) are added to the MISC section of the COMBAT tab.

0.7.0
Adds support for decoding Call of Cthulhu portfolios into Actors.
It works much better if the Foundry compendium and/or Items Directory contain definitions for weapons, skills, books and spells.

Farling

Author of the Realm Works Import tool, Realm Works Output tool and Realm Works to Foundry module

Donations gratefully received via Patreon, Ko-Fi or Paypal

Last edited by Farling; September 16th, 2021 at 07:04 AM.
Farling is offline   #1 Reply With Quote
nightpanda2810
Senior Member
 
Join Date: Jun 2016
Posts: 204

Old May 17th, 2021, 11:23 AM
I'm sure this is totally surprising and unexpected, but an 800MB realm is a no go with this.
/s

A small fraction of the contents of this file crashes the browser (chrome) itself just trying to paste. Not surprising in the least of course. A 16MB file worked perfectly.

Where do images get imported to? I host on AWS so want to make sure I keep an eye on storage.

I look forward to seeing this progress!

Realmworks Template Management
nightpanda2810 is offline   #2 Reply With Quote
Farling
Senior Member
 
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,486

Old May 17th, 2021, 12:56 PM
At the moment the images which are within a topic are stored inside the compendium as part of the journal entry - they are inline in the HTML.

There isn't any facility (currently) to extract the images and store them elsewhere.

One of my more urgent tasks is to enable selecting a file, rather than expecting copy-paste to work. In fact, with Firefox and the 4 MB test file it takes longer to copy-paste into the browser than it takes to do the conversion.

My rwoutput file is 8,012 KB according to Windows Explorer. The realm.db compendium pack file into which it was imported is 6,362 KB in size. But this could change as more features are successfully transferred across.

Farling

Author of the Realm Works Import tool, Realm Works Output tool and Realm Works to Foundry module

Donations gratefully received via Patreon, Ko-Fi or Paypal

Last edited by Farling; May 17th, 2021 at 01:03 PM.
Farling is offline   #3 Reply With Quote
Farling
Senior Member
 
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,486

Old May 18th, 2021, 04:11 AM
Quote:
Originally Posted by nightpanda2810 View Post
I'm sure this is totally surprising and unexpected, but an 800MB realm is a no go with this.
/s

A small fraction of the contents of this file crashes the browser (chrome) itself just trying to paste. Not surprising in the least of course. A 16MB file worked perfectly.

Where do images get imported to? I host on AWS so want to make sure I keep an eye on storage.

I look forward to seeing this progress!
Version 0.1.1 allows a file to be selected so that there is no need to use copy/paste.

I tested this with a 426,163 KB file (426 MB) which contains 4901 topics.

The created compendium db file in Foundry has a size of 330,950 KB (331 MB).

Farling

Author of the Realm Works Import tool, Realm Works Output tool and Realm Works to Foundry module

Donations gratefully received via Patreon, Ko-Fi or Paypal
Farling is offline   #4 Reply With Quote
nightpanda2810
Senior Member
 
Join Date: Jun 2016
Posts: 204

Old May 18th, 2021, 09:03 AM
Quote:
Originally Posted by Farling View Post
Version 0.1.1 allows a file to be selected so that there is no need to use copy/paste.

I tested this with a 426,163 KB file (426 MB) which contains 4901 topics.

The created compendium db file in Foundry has a size of 330,950 KB (331 MB).
Awesome! Updated and attempted to import my file, 773MB. Progress says "loading" for a few seconds then changes to "please enter some data". Is this just a matter of giving it some time?

Realmworks Template Management
nightpanda2810 is offline   #5 Reply With Quote
Farling
Senior Member
 
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,486

Old May 18th, 2021, 02:01 PM
Quote:
Originally Posted by nightpanda2810 View Post
Awesome! Updated and attempted to import my file, 773MB. Progress says "loading" for a few seconds then changes to "please enter some data". Is this just a matter of giving it some time?
I haven't tested it for maximum file size. It is down to the browser (I think) as to how big a file it can handle.

I know that when I tried a 900 MB file it failed to load.

All my browser testing has been using the built-in browser that comes with the Windows desktop version of Foundry. I haven't tested it with Firefox or Chrome as standalone browsers.

Farling

Author of the Realm Works Import tool, Realm Works Output tool and Realm Works to Foundry module

Donations gratefully received via Patreon, Ko-Fi or Paypal

Last edited by Farling; May 18th, 2021 at 02:08 PM.
Farling is offline   #6 Reply With Quote
Happydevil43
Senior Member
 
Join Date: Apr 2014
Location: Perth, Australia
Posts: 160

Old May 18th, 2021, 02:05 PM
Hi Farling,

This is really good, as an aside, it might be worth reaching out to the guys at Arkenforge, their new encyclopedia system might work well benefiting from some sort of importer, and they have been thinking of doing a RW importer type tool for some time.

Their new encyclopedia is a vast step over the first iteration.

Cheers
Happydevil43 is offline   #7 Reply With Quote
Farling
Senior Member
 
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,486

Old May 18th, 2021, 02:07 PM
Quote:
Originally Posted by Happydevil43 View Post
Hi Farling,

This is really good, as an aside, it might be worth reaching out to the guys at Arkenforge, their new encyclopedia system might work well benefiting from some sort of importer, and they have been thinking of doing a RW importer type tool for some time.

Their new encyclopedia is a vast step over the first iteration.

Cheers
I'm not trying to do LWD out of business by writing a tool to import data into a competing world building tool. Foundry is primarily a VTT and doesn't really have the same capabilities as a world building tool.

Farling

Author of the Realm Works Import tool, Realm Works Output tool and Realm Works to Foundry module

Donations gratefully received via Patreon, Ko-Fi or Paypal
Farling is offline   #8 Reply With Quote
Farling
Senior Member
 
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,486

Old May 18th, 2021, 02:13 PM
Using a simple tool at https://stackoverflow.com/questions/...iable/33768639, the maximum capacities for a string for each browser are:

Firefox = 1,073,741,822 (1 GB)
Chrome = 536,870,888 (500 MB)

The entire file is read into a single string, so these limits will apply to the maximum allowed file size.

---

I just ran a quick test on Firefox. It loaded a 900 MB file, but failed to load a javascript library which I use for reading portfolio files. So more testing is required...

With a fix for that problem, I see another error when trying to process a large XML file...

"Error: Cannot create a string longer than 0x1fffffe8 characters"

That hex number is 536,870,888 bytes, which is 24 bytes less than 512 MB.

Farling

Author of the Realm Works Import tool, Realm Works Output tool and Realm Works to Foundry module

Donations gratefully received via Patreon, Ko-Fi or Paypal

Last edited by Farling; May 18th, 2021 at 03:10 PM.
Farling is offline   #9 Reply With Quote
Farling
Senior Member
 
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,486

Old May 18th, 2021, 03:12 PM
Quote:
Originally Posted by nightpanda2810 View Post
Awesome! Updated and attempted to import my file, 773MB. Progress says "loading" for a few seconds then changes to "please enter some data". Is this just a matter of giving it some time?
I need to change the message here. This is an indication that the file is too large.

Farling

Author of the Realm Works Import tool, Realm Works Output tool and Realm Works to Foundry module

Donations gratefully received via Patreon, Ko-Fi or Paypal
Farling is offline   #10 Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 02:18 PM.


Powered by vBulletin® - Copyright ©2000 - 2021, vBulletin Solutions, Inc.
wolflair.com copyright ©1998-2016 Lone Wolf Development, Inc. View our Privacy Policy here.