Lone Wolf Development Forums

Lone Wolf Development Forums (http://forums.wolflair.com/index.php)
-   Realm Works Discussion (http://forums.wolflair.com/forumdisplay.php?f=67)
-   -   RealmWorks CSV Import Tool (http://forums.wolflair.com/showthread.php?t=58156)

Janrith March 17th, 2019 08:38 AM

Running version 2.3.0.0, when I open a project file, the Importer becomes unresponsive. I've left it running for up so 20 minutes and finally had to kill the process.

Is this a known issue?

Farling March 17th, 2019 03:21 PM

Version 2.4 is available

This version contains two improvements:

1 - you can now specify an annotation for an asset snippet without having to specify the actual file containing the asset (e.g. annotations for pictures and smart images).

2 - all warning and critical messages are now displayed in an error window. One example of this would be when a source data value doesn't match one of the known values for a tag.

This should fix Acenoid's problem reported in post #476 and the second issue reported in post #477.

Acenoid March 17th, 2019 05:35 PM

Imported the data now - I noticed that even after ensuring the XLSX and the RW information is the same the "Transporter, Mittelschwer" entry has not been inserted after all. I manually edited now as it only were 4 entries. However it made me believe that maybe the "," (comma) might have been responsible for the missing entry?

I will try to test it again tonight.

Oh - almost forgot - thanks for the updated version. Will give download it tonight, since i have to import a few more lists and want to test the issue above.

Quick question regarding the images -- Since the error was File / URL not found , does it really mean I could link in my source file any URL? Cool!

Farling March 17th, 2019 11:06 PM

Quote:

Originally Posted by Acenoid (Post 276785)
Quick question regarding the images -- Since the error was File / URL not found , does it really mean I could link in my source file any URL?

See release notes...

v1.16
When loading an image (see V1.11) it is now possible to specify a web URL instead of just a local file name. Note that the URL must point directly to the image, not to a web page which contains an image somewhere inside it.

v1.17
When reading images from a web URL, follow web redirects which are issued by the web server.

Acenoid March 18th, 2019 01:32 PM

Really nice !

Janrith March 18th, 2019 03:43 PM

Quote:

Originally Posted by Janrith (Post 276744)
I am also getting the C++ Exception: Not in a span at line 1633 in file utility.cpp error.

I am fairly confident it is a data issue; I have no HTML in my xlsx file, but I do have a Components field that does have gt characters occasionally (e.g. powdered diamond duct >= 500gpv).

I imported a single row that didn't have any notes (by removing all <topic/> elements from the rwexport file rather than killing my excel file).

I'm trying an import definition that leaves off that column and will report back.

Confirmed the > is the problem prematurely closing the xml elements inside the rwexport file. Converting these from >= to "at least" allowed for successful imports.

Thanks for this tool! 750 spells imported in 20 minutes.

Janrith March 18th, 2019 03:50 PM

Hi again Farling,

Two more questions:

1. Have you given any thought to allowing the user to select which worksheet they want to import from inside the workbook? It's be nice to have import data stored in a single workbook, delineated by worksheet. YMMV.

2. Any recommendations on how to include statblock imports inside the spreadsheet? I have an HTML template prepped that works when I manually insert it into a topic, but I'd like to be able to include it in the spreadsheet, use a concatenate() function to insert the appropriate row data into the statblock, and then use the importer to send it to Realm Works.

Aside from the issues having child <html></html> blocks may cause, the concat function is limited to 8192 characters in it's parameter set, so it seems like I may be dead in the water here.

At any rate, thanks again for a great tool!

-=Janrith

Farling March 18th, 2019 04:14 PM

Quote:

Originally Posted by Janrith (Post 276834)
Any recommendations on how to include statblock imports inside the spreadsheet? I have an HTML template prepped that works when I manually insert it into a topic, but I'd like to be able to include it in the spreadsheet, use a concatenate() function to insert the appropriate row data into the statblock, and then use the importer to send it to Realm Works.

Aside from the issues having child <html></html> blocks may cause, the concat function is limited to 8192 characters in it's parameter set, so it seems like I may be dead in the water here.

The tool allows you to specify an external file from which to import a stat block. The cell contains the name of the file (or URL) for what will become the stat block. The cell does not contain the actual contents of the stat block.

Perhaps you could write a script (perl, powershell?) which will generate a lot of files from your HTML template; and then reference the filenames of these templates in your XLSX file?

I haven't done much testing with importing stat blocks. I have a comment that it might only work with .rtf file; but I'd be interested to know what success you might have with other file types. (Maybe do Word mail merge to create a lot of .rtf files?)

Farling March 18th, 2019 04:15 PM

Quote:

Originally Posted by Janrith (Post 276834)
Have you given any thought to allowing the user to select which worksheet they want to import from inside the workbook? It's be nice to have import data stored in a single workbook, delineated by worksheet.

I've added this to the list of feature requests.

Janrith March 18th, 2019 08:42 PM

Quote:

Originally Posted by Farling (Post 276839)
The tool allows you to specify an external file from which to import a stat block. The cell contains the name of the file (or URL) for what will become the stat block. The cell does not contain the actual contents of the stat block.

Perhaps you could write a script (perl, powershell?) which will generate a lot of files from your HTML template; and then reference the filenames of these templates in your XLSX file?

I haven't done much testing with importing stat blocks. I have a comment that it might only work with .rtf file; but I'd be interested to know what success you might have with other file types. (Maybe do Word mail merge to create a lot of .rtf files?)

It would be bit of overkill, but I could write an SSIS package to read the excel file and generate the stat blocks. I was hoping for something a little more lightweight though :)

Maybe I'll whip up a configurable console app or something...

Thanks.

Farling March 18th, 2019 11:11 PM

Quote:

Originally Posted by Janrith (Post 276854)
It would be bit of overkill, but I could write an SSIS package to read the excel file and generate the stat blocks. I was hoping for something a little more lightweight though :)

Maybe I'll whip up a configurable console app or something...

Thanks.

TBH, I think that LWD aren't using Stat Block snippets any more because they aren't as accessible as was first hoped. Instead they put the formatted character stat block into a text snippet.

Janrith March 19th, 2019 12:07 AM

Quote:

Originally Posted by Janrith (Post 276774)
Running version 2.3.0.0, when I open a project file, the Importer becomes unresponsive. I've left it running for up so 20 minutes and finally had to kill the process.

Is this a known issue?

Has anyone else experienced this issue? My workaround is to just re-create the definition whenever I open the importer, but it would be nice to be able to load a project file.

I did notice there were no obvious memory, resource or handle leaks while it is in this state, although the I/O read count gets astronomical (in the order of 100s of millions), while the I/O bytes read never moves.

Farling, does QT generate PDB symbols? If so and you'd be willing to share, I could run this under NTSD and see if I can tell what it's spinning on; it seems like a resource wait of some kind...

Thanks.

-=Janrith

Farling March 19th, 2019 03:44 PM

Quote:

Originally Posted by Janrith (Post 276859)
Has anyone else experienced this issue? My workaround is to just re-create the definition whenever I open the importer, but it would be nice to be able to load a project file.

I will have a look this weekend. I admit I haven't used the save project option recently.

Acenoid March 19th, 2019 04:30 PM

Hi there! Wanted to retry an import - same for me 2.3.0 open project seems to freeze the program - at least for the moment :D

Farling March 19th, 2019 04:35 PM

Quote:

Originally Posted by Acenoid (Post 276893)
Hi there! Wanted to retry an import - same for me 2.3.0 open project seems to freeze the program - at least for the moment :D

Was it working in 2.2 ?
Or what was the last version in which it was working :)

Also, did you save the project in 2.3, or save in an earlier version and try to open in 2.3 ?

Acenoid March 19th, 2019 04:58 PM

I *think* it was working previously.
A separate file saved with 2.3 also does not seem to load.
The file is not readably with notepad++ for exmaple, can I recognize the file version of it somehow?

Farling March 20th, 2019 03:49 PM

Version 2.4.1 is out

This is a bug-fix release which fixes the problem with Save / Load Project.

The saved project information was not sufficient to restore the data properly. In particular, if a section had the same name as a snippet (e.g. "class skills"). I guess that I was lucky in picking the "Class" topic type to do the testing, otherwise it would have taken a lot longer to find the problem.

(My apologies for fixing it earlier than I originally planned.)

Janrith March 20th, 2019 11:18 PM

Thanks Farling.

I installed 2.4.1 and opened my latest project, but received the following error message:

RWTopic>> failed to find RWContentsItem for ":"

Additional issues:

1. My Notes column is being incorrectly attributed to the Spell's Name; none of the other column mappings are present.
2. The reveal settings for each snippet are lost.
3. The Parent article(s) as created in the project are missing.

Sorry to be the bearer of bad news, but it seems something is still wonky...

-=Janrith

Farling March 20th, 2019 11:48 PM

Quote:

Originally Posted by Janrith (Post 276964)
Thanks Farling.

I installed 2.4.1 and opened my latest project, but received the following error message:

RWTopic>> failed to find RWContentsItem for ":"

Additional issues:

1. My Notes column is being incorrectly attributed to the Spell's Name; none of the other column mappings are present.
2. The reveal settings for each snippet are lost.
3. The Parent article(s) as created in the project are missing.

Sorry to be the bearer of bad news, but it seems something is still wonky...

-=Janrith

Darn, more testing required.

Are the issues you reported related to what the tool is like after loading a project, or even immediately after launching the tool?

Also, was the "RWTopic>>" message from loading a saved project from an earlier version of the tool? Earlier saves aren't compatible with the format used by 2.4.1

Acenoid March 21st, 2019 02:45 PM

Hi Farling!

Is it possible somehow to get the rw importer to update my topics in rw? At the moment it will always create a duplicate topic tree correct? (its no problem - just wanted to know if iam missing something) :)

Farling March 21st, 2019 02:51 PM

Quote:

Originally Posted by Acenoid (Post 277009)
Hi Farling!

Is it possible somehow to get the rw importer to update my topics in rw? At the moment it will always create a duplicate topic tree correct? (its no problem - just wanted to know if iam missing something) :)

Hi,
Unfortunately, the tool isn't able to update existing topics.

Realm Works creates a unique id for each topic, and the tool would have to read a full realm export instead of only a structure file in order to get the ids (and then it would have to do some checking to see if you wanted to update an existing topic or create a new topic for each row in the CSV/XLSX).

Janrith March 21st, 2019 04:34 PM

Quote:

Originally Posted by Farling (Post 276965)
Darn, more testing required.

Are the issues you reported related to what the tool is like after loading a project, or even immediately after launching the tool?

After loading a project. Starting the process has no issues.

Quote:

Also, was the "RWTopic>>" message from loading a saved project from an earlier version of the tool? Earlier saves aren't compatible with the format used by 2.4.1
Tested loading both a 2.3.0.0 and a 2.4.1.0 project file. Same result opening both.

Thanks.

-=Janrith

Janrith March 21st, 2019 04:49 PM

Three quick 2.4.1.0 tests: Each is a new 2.4.1.0 project file.

1. XLSX Name column mapped to Article Name of Spell (AD&D) Article type.
Load: OK
Mapping: OK
2. XLSX Name column mapped to Article Name of Spell (AD&D) Article type. Class mapped to Parent General Abilities Article Name.
Load: OK
Mapping: Missing parent
3. XLSX Name column mapped to Article Name of Spell (AD&D) Article type. Class mapped to Parent General Abilities Article Name, "AD&D" literal text added to Parent Name Suffix.
Load: OK
Mapping: Missing parent

I'm going to go through one at a time with my column mappings and see what triggers the RWTopic>> failed to find RWContentsItem for ":" error.

I'll let you know what I find out.

Thanks.

-=Janrith

Janrith March 21st, 2019 04:52 PM

Feature request:

"New Project" option on the File Menu...

Would like to clean the slate w/o needing to exit and reload the tool.

Thank.

-=Janrith

Janrith March 21st, 2019 07:28 PM

OK,

After a bunch of tests, it appears the importer doesn't like having column N populated. When Casting Time (column N) is not mapped in the project file, but Saving Throw, Material Components, Notes, Author, Filtered Name are (columns O, P, R, S and T are mapped), the project load succeeds without incident.

For some reason, when column N is mapped to a structure field in the project file, loading the project prevents any column past N from mapping successfully.

(*EDIT: It appears the Article Name is unaffected, as that is currently being populated by column T (Filtered Name); but the problem does affect subsequent sections.)

NOTE: Creating the definition and then generating an rwexport file in the same session works fine. It seems to only be a problem with loading the project file.

DETAILS:

XLSX Column Definitions:

A: Class
B: Source Book
C: Edition
D: Frequency
E: Level
F: Name
G: Reversed Spell
H: Reversible
I: School of Magic
J: Range
K: Duration
L: Area of Effect
M: Components
N: Casting Time
O: Saving Throw
P: Material Component
Q: Description
R: Notes
S: Author (formula)
T: Filtered Name (formula)

Here are the last few results:

Test results:

20:
Type: XLSX
Structure: v.03
Target Article: Spell (AD&D)
Columns Mapped: Filtered Name;School of Magic;Level;Range;Material Component;Duration;Components;Area of Effect;Saving Throw;Source Book;Edition;Author;Notes;Reversed Spell
Load: OK
Mapping: OK

So far so good...

Here comes the first failure:

21:
Type: XLSX
Structure: v.03
Target Article: Spell (AD&D)
Columns Mapped: Filtered Name;School of Magic;Level;Range;Material Component;Duration;Components;Area of Effect;Saving Throw;Source Book;Edition;Author;Notes;Reversed Spell;Casting Time
Load: RWTopic>> failed to find RWContentsItem for ":"
Mapping: No fields mapped past Casting Time (column N in the spreadsheet).

Is it the name of the column in the spreadsheet?

22:
Type: XLSX
Structure: v.03
Target Article: Spell (AD&D)
Columns Mapped: Filtered Name;School of Magic;Level;Range;Material Component;Duration;Components;Area of Effect;Saving Throw;Source Book;Edition;Author;Notes;Reversed Spell;renamed Casting Time to CT in XLSX.
Load: RWTopic>> failed to find RWContentsItem for ":"
Mapping: No fields mapped past Casting Time (column N in the spreadsheet).

Nope.

Is it a problem with the field definition in the structure file?

23:
Type: XLSX
Structure: v.03
Target Article: Spell (AD&D)
Columns Mapped: Filtered Name;School of Magic;Level;Range;Material Component;Duration;Components;Area of Effect;Saving Throw;Source Book;Edition;Author;Notes;Reversed Spell;mapped Range to Casting Time target as well.
Load: OK
Mapping: OK

Nope.

Is it positional in the spreadsheet?

24:
Type: XLSX
Structure: v.03
Target Article: Spell (AD&D)
Columns Mapped: Filtered Name;School of Magic;Level;Range;Material Component;Duration;Components;Area of Effect;Saving Throw;Source Book;Edition;Author;Notes;Reversed Spell, Casting Time moved from column N to column J in XLSX.
Load: RWTopic>> failed to find RWContentsItem for ":"
Mapping: No fields mapped past Components (new column N in the spreadsheet).

Yep.

No idea why though.

I hope this helps.

-=Janrith

Farling March 21st, 2019 11:49 PM

Thanks for all the detail Janrith!
I'll have a look to see why column N (number 14) is causing a problem.
(Initial thought, column N could be number 13 if you start counting at 0, and 13 is the ASCII for code for carriage return [end of line].)

Farling March 22nd, 2019 03:36 AM

Version 2.4.2 is available

This fixes more bugs.
It is now possible to load a project in which you've used column 14 in your data table.
The parentage that you've set up for topics should also be saved and restored.

Many thanks to Janrith for his diligent hard work in finding the column 14 issue.

No previous saved projects will load successfully in this version.

Ash Nazg March 22nd, 2019 06:42 AM

Just a quick note to say, brilliant QA and debugging work to both Janrith and Farling. Very impressive!

And specifically @Farling: thanks for this amazing tool (which I'm surprised LWD did not built into RW), and for your continued support and tinkering that makes it consistently and (frequently!) better. *doffs cap*

~Ash

Farling March 22nd, 2019 09:30 AM

Quote:

Originally Posted by Janrith (Post 277019)
Feature request:

"New Project" option on the File Menu...

Would like to clean the slate w/o needing to exit and reload the tool.

Thank.

-=Janrith

Actually you only need to load a structure file (you could pick the same file). This will remove all the columns that you've configured.

Farling March 22nd, 2019 11:34 AM

Quote:

Originally Posted by Taronthir (Post 275485)
I haven't discovered yet how to make multi-domain tag entries in the csv/xls so they'll be interpreted properly.

"Tags (Multi-Domain)" snippets are not supported yet.

If you want multiple tags on a more normal "Tags" snippet, then you can do this by separating the tab labels with a comma in your source data cell.

Janrith March 22nd, 2019 10:07 PM

Downloaded 2.4.2.0 and tested. Loading the project file is working now EXCEPT:

1. Literal text in a Parent's Name and Suffix is not preserved when reloading.
2. Literal text in File Details/Credits field is not preserved when reloading.

The big meat in mapping worksheet columns to category fields is working WOOT!

Feature Request:
Any chance we can get the project filename appended to the app title on load? Having the * appended to the name when there are unsaved changes would be an added bonus :).

It would help to remember what file you are working in w/o having to open the Save Project As... dialog.

Thanks again for the awesome tool!

-=Janrith

Janrith March 22nd, 2019 10:23 PM

When you select Load Data, the tool appears to cache the data in memory and use that cache rather than re-reading the worksheet to build the rwexport file.

This means, if you subsequently update your source file (XLSX confirmed, probably the CSV will work the same), you need to select Load Data in order to refresh this cache or your updates won't make it to the rwexport file.

This is actually a good thing, since changing your column definition out from under your project definition could have alignment issues, depending on how the columns are being read from the file.

Haven't tested whether this cache is cleared when selecting a new structure definition. If the cache is maintained, it might be another argument for a New Project... option.

I'll test caching of CSV data and what happens when the structure is changed tomorrow... it's getting late.

Later.

-=Janrith

Farling March 22nd, 2019 11:11 PM

Quote:

Originally Posted by Janrith (Post 277145)
When you select Load Data, the tool appears to cache the data in memory and use that cache rather than re-reading the worksheet to build the rwexport file.

This means, if you subsequently update your source file (XLSX confirmed, probably the CSV will work the same), you need to select Load Data in order to refresh this cache or your updates won't make it to the rwexport file.

This is actually a good thing, since changing your column definition out from under your project definition could have alignment issues, depending on how the columns are being read from the file.

Haven't tested whether this cache is cleared when selecting a new structure definition. If the cache is maintained, it might be another argument for a New Project... option.

I'll test caching of CSV data and what happens when the structure is changed tomorrow... it's getting late.

Later.

-=Janrith

Hi,

Yes, when you select "Load Data" then it is read into memory and the file can be changed without changing the loaded data.

When you load a saved project, both the data file and the structure file are read from disk again - so any changes you've made in the CSV or XLSX file will be read in at that point.

The storage of the loaded data and the loaded structure are totally separate, so you can load a new structure file and use the already loaded data with that new file, or vice versa.

Janrith March 23rd, 2019 01:46 AM

Quote:

Originally Posted by Janrith (Post 277143)
Downloaded 2.4.2.0 and tested. Loading the project file is working now EXCEPT:

1. Literal text in a Parent's Name and Suffix is not preserved when reloading.
2. Literal text in File Details/Credits field is not preserved when reloading.

The big meat in mapping worksheet columns to category fields is working WOOT!

Feature Request:
Any chance we can get the project filename appended to the app title on load? Having the * appended to the name when there are unsaved changes would be an added bonus :).

It would help to remember what file you are working in w/o having to open the Save Project As... dialog.

Thanks again for the awesome tool!

-=Janrith

Addendum to #2 above. It looks like the Credits are actually still in the project file, the UI just keeps wiping out the field.

Repro:
(assumes a loaded project file with a Credit value populated)

a. Open File Details.
b. Enter data into the Credits field; click OK.
c. Click File Details. Credits is empty.
d. Click Cancel to avoid saving the empty field.
e. In the cmd line navigate to export directory
f. findstr /I Janrith "Test Import 34.rwexport"

result:

<credits>Janrith</credits>

Thanks again.

-=Janrith

Farling March 23rd, 2019 02:23 AM

Version 2.5 is available

When a project has been saved or loaded, the name of the project is displayed in the window title (requested in http://forums.wolflair.com/showpost....&postcount=511). (No star appears to indicate that the project has been modified.)

Fixes a bug where "fixed text" in a field definition was not being displayed after changing categories or loading a saved project (repoted in http://forums.wolflair.com/showpost....&postcount=511)

Fixes a bug where the credit box in the File Details windows was being populated with the Requirements text when the window was opened. (reported in http://forums.wolflair.com/showpost....&postcount=514)

Farling March 23rd, 2019 03:24 AM

Version 2.6 is available

A "*" is put in the window title bar if modifications have been made since the last load/save of a project.

On quitting the application, a prompt is displayed if a project has not been saved since the last change.

Janrith March 23rd, 2019 05:11 PM

Downloaded 2.6.0.0.

Literal text parent loading verified.
Credits field in File Details loading verified.

* (unsaved flag) working for target fields on main window (article, parent, true names, relationships). Does not yet apply to checkboxes or dropdowns (ether standard ones or the "tool icon" ones), or the File Details modal dialog. Also can't get it to prompt me to save my changes when moving to a different file once I've triggered the *.

I don't want to side track you on your feature priorities; just wanted to fill you in on my observations.

Thanks again.

-=Janrith.

Madmaxneo April 22nd, 2019 05:08 PM

This CSV tool looks like it may do the job I need it to do. But I'm not sure it will work.

I'd like to transfer pertinent character and NPC data into Realm Works from a character sheet (.xls spreadsheet).

Please note that I am using the HARP system and it is not in HeroLab.

I'd prefer if it was possible to just name certain cells withing the spreadsheet and the CSV tool simply import those specific cell values into specific snippets within RW.

My questions:
1. Does the CSV tool require one to have columns of data with headers or is it possible to pull only specific cell data from a single cell in a ".xls" spreadsheet (or from multiple singular cells)?
2. Do I still need to "save as" the ".xls" spreadsheet into a CSV file so this tool can read the info or does the CSV tool read .xls data also?
3. Is it possible to overwrite data when a character or NPC increases in level? Meaning can I just import the same character over and over again or would I need to delete the RW snippets/character entry every time someone goes up a level?

daplunk April 22nd, 2019 06:28 PM

My questions:
1. Does the CSV tool require one to have columns of data with headers or is it possible to pull only specific cell data from a single cell in a ".xls" spreadsheet (or from multiple singular cells)?
It works on a table. All columns must have a header. Think of it as one column per snippet. Each line in that table would align with a different topic.
2. Do I still need to "save as" the ".xls" spreadsheet into a CSV file so this tool can read the info or does the CSV tool read .xls data also?
You can save as XLSX or CSV
3. Is it possible to overwrite data when a character or NPC increases in level? Meaning can I just import the same character over and over again or would I need to delete the RW snippets/character entry every time someone goes up a level?
No this will not work as the tool does not save unique UUID's for future use. It creates a new one for each snippet during the import process.

Madmaxneo April 22nd, 2019 06:40 PM

Quote:

Originally Posted by daplunk (Post 278331)
My questions:
1. Does the CSV tool require one to have columns of data with headers or is it possible to pull only specific cell data from a single cell in a ".xls" spreadsheet (or from multiple singular cells)?
It works on a table. All columns must have a header. Think of it as one column per snippet. Each line in that table would align with a different topic.
2. Do I still need to "save as" the ".xls" spreadsheet into a CSV file so this tool can read the info or does the CSV tool read .xls data also?
You can save as XLSX or CSV
3. Is it possible to overwrite data when a character or NPC increases in level? Meaning can I just import the same character over and over again or would I need to delete the RW snippets/character entry every time someone goes up a level?
No this will not work as the tool does not save unique UUID's for future use. It creates a new one for each snippet during the import process.

Thanks for the reply and explanation.

For player characters I would have to delete their snippets then do the export/import thing each time they go up a level.

For the table requirement I can create a spreadsheet that can gather all the pertinent data from each of the character sheets so that it is easily formatted into a table for the CSV tool. It adds extra work to getting it all straightened out for the game, but I'll get it done eventually.


All times are GMT -8. The time now is 02:19 PM.

Powered by vBulletin® - Copyright ©2000 - 2024, vBulletin Solutions, Inc.
wolflair.com copyright ©1998-2016 Lone Wolf Development, Inc. View our Privacy Policy here.