Senior Member
Join Date: Jul 2017
Posts: 347
|
In my recent endeavor to optimize my monolithic realms I consider to move the published adventures in their own realms and create temporary ones for specific adventure on demand. Enter Manage Exports and Imports
While I was reading the documentation that came with RW, I found that it is incomplete when it comes to advanced usage of the tool. The sections related to export and data manipulation are TBD, empty or lacking. So, I figured out the following:
Questions:
The past is a rudder to guide us, not an anchor to hold us back. Last edited by Valyar; March 19th, 2018 at 05:23 AM. |
#1 |
Senior Member
Join Date: Oct 2014
Location: Chicago, IL
Posts: 1,690
|
You have a lot wrong.
A structure only export does not rely on tags. Tagging is very smart. It uses an existing feature to do something rather than adding an entirely new one. If you designate an export as full it will be full and the export tag will be applied everywhere. Partial exports do work the same way. The easiest way to apply the export tag is to create the export, but not do the export then create a view of the material you want in that export and use the bulk tagging feature to apply that tag to the entire view. the bulk tagging feature was added for expressly this usage. Structure exports are not required prior to exporting data. Any export of a realm will also include the structure of the realm. To the best of my knowledge no one has produced a DTD or XSD of the export file XML. According to what we've been told compact is for consumption by programs not RW while full is for RW's use. my Realm Works videos https://www.youtube.com/channel/UCZU...4DwXXkvmBXQ9Yw |
#2 |
Senior Member
Join Date: Jul 2017
Posts: 347
|
How I got "a lot wrong" when all only points 4 and 6 are off the target? The Quick Help here was of good use.
I wonder if DTD or XSD will be ever provided to facilitate integration of RW with other tools (I currently think for Fantasy Grounds, as we play there from time to time). Anyway, the tag for full export is not applied everywhere automatically, I had 20% of the articles without the full realm export tag and this includes manually created articles (I initially assumed it is due to bulk uploaded content). I haven't had time to check if the un-tagged content will be exported without the tag, but not that important as with bulk tag I can easily add whatever is missing. Custom Exports are clear. The past is a rudder to guide us, not an anchor to hold us back. Last edited by Valyar; March 20th, 2018 at 01:28 AM. |
#3 |
Senior Member
Join Date: Oct 2014
Location: Chicago, IL
Posts: 1,690
|
Did the full realm export? was it all in the new realm when you imported? Did it all have the tag there?
my Realm Works videos https://www.youtube.com/channel/UCZU...4DwXXkvmBXQ9Yw |
#4 |
Senior Member
Join Date: Jul 2017
Posts: 347
|
Finally I found time to fool around with the Full/Partial Export/Import features of Realm Works. The realm I exported has ~2200 topics/articles and the whole database with all realms is 533MB. The export/import operation was between different disks/arrays and from disk to NAS over 1GB network.
So, some observations Full Realm Export (Full output)
Full Realm Import
Partial Export/Imports
I don't know why those full export/import operations are slow and what is happening behind the scenes. I don't dare to think what will be when I reach GBs of content someday... The past is a rudder to guide us, not an anchor to hold us back. |
#5 |
Senior Member
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,623
|
The increase in size of various fields over the base database is due to all binary data being stored in the XML file in base64 format (so each 3 bytes of image/file takes up 4 bytes in the XML file).
In a realm with a lot of files or images, there will be a lot of binary to base64 encoding going on. I think removing the import tag will cause confusion when a later import is created from the same realm, since RW will probably create duplicate topics. Farling Author of the Realm Works Import tool, Realm Works Output tool and Realm Works to Foundry module Donations gratefully received via Patreon, Ko-Fi or Paypal |
#6 |
Senior Member
Join Date: Jul 2017
Posts: 347
|
Thanks for the clarification. Binary to base64 makes sense, I have all tables as images taken from the books.
The past is a rudder to guide us, not an anchor to hold us back. |
#7 |
Senior Member
Join Date: Oct 2014
Location: Chicago, IL
Posts: 1,690
|
One of the reasons for the lack of performance on modern workstations that should do task like RW very well is that RW is a 32 bit application and simply cannot take advantage of most of those system resources even when they are available. The program also doesn't appear to be multi threaded at all. Things like import or export don't create spikes in the usage of any cores/threads not already in use.
my Realm Works videos https://www.youtube.com/channel/UCZU...4DwXXkvmBXQ9Yw |
#8 |
Senior Member
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,623
|
Quote:
The infrequent nature of creating XML exports wouldn't make it a high priority for optimisation. Farling Author of the Realm Works Import tool, Realm Works Output tool and Realm Works to Foundry module Donations gratefully received via Patreon, Ko-Fi or Paypal |
|
#9 |
Senior Member
Join Date: Oct 2014
Location: Chicago, IL
Posts: 1,690
|
Quote:
<warning for those not interested in such technical matters> Certain parts of the XML can be generated in any order, such as the bulk of the topic and articles. Threads could be spun off to process individual topics with the master thread putting the results of each processing thread into the output file. This should take advantage of all the processing power available while having minimal impact on lower core count systems. Of course depending on how much overhead there is in creating and destroying a thread this might be slower than doing it single threaded. While I've done a lot of work in C# relatively little has been in any sort of heavily optimized processing. When I've known the program was going to have a heavy workload I've gone to a better solution. my Realm Works videos https://www.youtube.com/channel/UCZU...4DwXXkvmBXQ9Yw |
|
#10 |
|
|