![]() |
|
Senior Member
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,622
|
Hello RW Community,
I have been quietly working on a generic CSV import tool which (hopefully) requires no programming knowledge for the user. For non-ASCII characters, the CSV file should use the UTF-8 encoding (there's probably an option in Excel, or your other favourite spreadsheet program, to encode as UTF-8 rather than some other form of encoding). As of version 1.38, it is now possible to directly import Excel (xlsx) files. There is no longer the need to save them as CSV files. There are several steps to using the tool: 0. Download the installer from https://github.com/farling42/csv2rw/...RWimporter.exe 1. Prepare data from RealmWorks 1.1 Create a structure only export of the realm. (Go to the "Share" tab, select "Manage Exports"; in the new window select the "Export Realm Structure Only" tab, type something into the "Name" box, prese the "Save Changes" button, then press the "Full Export" button to generate the RWstructure file.) 1.1.1 (Optional) Within RealmWorks, if you want additional labelled snippets to be used during your import, then you should add these into the Realm Works category definition (possibly making a copy of a predefined category definition) before creating the structure only export. (Go to the "Manage" tab, select "Categories", choose the topic/article category which you want to use, select the tool icon and select "Create duplicate of category".) 1.2 If you want to load a collection of images and/or herolab portfolios and/or other data types, then the name of these files should be specified in one of the columns in the CSV file. A separate file will be loaded for each row of the CSV, although you can use the same filename in more than one row of the CSV if required. 2. Use the tool 2.1 Start up the CSV Import tool. 2.2 Press the "Load CSV" button and choose the file containing your CSV data. 2.3 Press the "Load Structure" button and choose the file containing the RealmWorks structure-only export file. 2.4 Select the required category type in the drop down menu below the "Load Structure" button to choose the category that you want your CSV data to be imported into. (This should be the "copy" that you made of the predefined category/article structure.) 2.5 Drag each of the CSV column names from the left panel to the corresponding place in the category in the right panel. 2.5.1 If the snippet should be revealed, then select the first radio box on the line (the circle). If the snippet should be marked as GM-only, then select the second radio box on the line. 2.5.2 If you want to remove a field from the category panel, then just right-click the field to remove it. 2.5.3 If you want a field to contain the same text in all occurrences of the topic/article, then you can add text directly into the field, or choose a particular tag from the drop-down list (instead of dragging one of the CSV column names across). 2.6 If you want all your topics/articles to be imported as children of another topic; then tick "Add Parent", choose a category for the parent, and enter a name for it (optionally with a prefix and suffix for the parent too). If you want to group entries under separate children, you can choose a particular CSV column to use for each group (e.g. you can group your spells with each school of spells under a different parent). 2.6.1 If you want a heirarchy created, then you can create multiple parents, choosing different CSV columns to fill in the title of the parent topics. 2.7 When you have all the fields defined, press the "GENERATE" button at the bottom-right of the window to generate the export file. 3 Import into RealmWorks 3.1 Import the generated file into your realm. (Go to the "Share" tab, select "Import", click on the box next to "File to Import" and select the rwexport file generated by the CSV tool. Then press the "Import" button.) ![]() 4 Complex tasks 4.1 Custom section names If you want the name of a section within a topic to be the text from a CSV column, then select "..." in the section title and set a column for the "First Section" box (leave the other two section boxes blank). 4.2 Multiple custom subsections If you want to have the same section duplicated multiple times, then select "..." and set a column for the First, Second, and optionally the Third section box. The First section defines the CSV column containing the title of the first section. The Second section defines the CSV column containing the title of the second section. The Third section (defaults to last column) defines the CSV column containing the title of the last section. The snippets within the section should be set to use CSV columns between First and Second column. For the second and subsequent sections, the same "offset" will be used for reading the text from the correct column of the CSV table. (example available in http://forums.wolflair.com/showpost....&postcount=295) NOTES It is no longer mandatory to make a copy of a predefined category, however if a copy is made (or a user defined category is used), then the additional greyed-out placeholder text will be displayed within this tool. The first line of the CSV file is assumed to have the field names in it. The bottom panel of the window shows the contents of the CSV file, so you can double-check that you are choosing the correct field. For the domain selection parts of a topic, you can drag the CSV field name directly onto the drop-down marker in the right panel; or if you want the plain text then you can drop it into the "annotation" part of the window. Note that if the value does not exist in the tags available from RealmWorks, then no value will be put into the RWexport file. A field from the CSV panel can be dragged to more than one place in the topic/article panel if you want the same data in more than one place (usually used to set both the tag selector and the annotation field for the same item). Daplunk has created one of his amazing HowTo videos which shows the tool in use. You can see it at https://youtu.be/A1i3FqHfi9E Donations for efforts can be made via any of Ko-Fi or Paypal Last edited by Farling; December 3rd, 2021 at 04:04 PM. Reason: Change download link to always point to latest download |
![]() |
![]() |
Senior Member
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,622
|
Release Notes
v2.18 Images can now be embedded in XLSX files. The data preview window will show only a blank cell for any cells containing images. The generated RWexport file will have the image stored in PNG format. v2.17 Change CSV file reading to default to UTF-8 codec, and read newlines properly (QFile::Text on QFile: ![]() v2.16 (There is no 2.15) This version adds a function "row.hasColumn(name)" to the JavaScript support for derived columns. v2.14 When a complex JSON file is loaded, there is now an option to choose the top-level array to be used as the data source. v2.13 Save and Load actions have been added for "derived" columns. This allows you to save the custom column definitions and re-use them in another import project. (This means you don't have to copy/paste each individual definition for a new project.) v2.12 Adds "derived" columns, where the value is based on a javascript expression that you specify. The javascript expression can examine the values of other columns in the current row using the syntax: row.column('columnname') The new derived columns can be managed via the new "custom columns" button. Fixes a bug that prevented excel files from being loaded. v2.11 Adds the ability to launch the tool by double-clicking on your project's csv2rw file. Adds basic support for JSON and YAML data files. The complex hierarchy is reduced to a simple overall set of column names, using "/" to separate each step of the hierarchy that contains each specific data value. v2.9 Aliases set up in a topic definition are now saved as part of the project. From now on, projects saved in earlier versions of tool will load into later versions of the tool. v2.8 Adds an entry in the "Options" menu that will force the file's format_version attribute to be set to 3. (Realm Works 254 appears to generate export files with format_version of 4, but can't read such files during import.) v2.7 User can now choose which sheet inside an Excel document to be used as the data source (only one can be used). Provide warning about unsaved work when attempting to load a new project, load a new data file, load a new structure file, or choosing a different worksheet in the current excel data set. Ensure that the data mappings are shown when loading a project. v2.6.1 This is a bug fix release. It fixes the problem where a cell that is not HTML which contains any < or > or & characters does not import properly. v2.6 A "*" is put in the window title bar if modifications have been made since the last load/save of a project. On quitting the application, a prompt is displayed if a project has not been saved since the last change. v2.5 When a project has been saved or loaded, the name of the project is displayed in the window title. Fixes a bug where "fixed text" in a field definition was not being displayed after changing categories or loading a saved project. Fixes a bug where the credit box in the File Details windows was being populated with the Requirements text when the window was opened. v2.4.2 This is a bug fix release. It is now possible to save projects that use column 14 from the table, and the saved project includes the parentage that you've set up. No projects saved with an earlier version can be loaded into this version. v2.4.1 This version is a bug fix release which gets the Save Project action to save data in a format that can be read properly by Load Project. v2.4 The internal warnings and critical messages are now reported in an error window - e.g. when a source data cell doesn't match a known value for tag, then this is reported in the error window. It is also now possible to configure a field to fill in the annotation for an asset snippet type without requiring you to also configure a field for that actual asset type (e.g. picture or smart image). v2.3 When reading an Excel spreadsheet, the default cell formatting is used as a starting point when trying to determine what formatting should be applied to text within each cell. v2.2 Annotations will now be produced on all snippet types (except multi-line and labeled text types). Numeric fields will now be checked that they contain only characters allowed in numbers. v2.1 When generating aliases, the tool now checks that the name of the alias does not match either the topic title or an alias within the same topic. A warning is displayed, and the alias is not added to the RWexport file. (This prevents a crash of Realm Works during import.) v2.0 GM Directions can now be generated from the import data. The tool is now a 64-bit tool (and uses the latest Qt 5.12.0). The tool has been renamed to RWImporter since it handles more than only CSV files. v1.49 This fixes a bug (introduced in 1.45) where a topic was being created more than once under different parents, where the name of parent is a shorter version of another parent. v1.48 An error dialog will be displayed if the same ROW of the source data would generate more than one topic in the output file. An output file will still be generated, but Realm Works will fail to import the file. A restriction was added when implementing relationships (in 1.46) which meant that each ROW of the source data can only generic one topic in the output file (otherwise it would be impossible to know with which of multiple topics a relationship was being created.) v1.47 Remove Minion_To_Master and Offspring_To_Parent relationships, since they are ignored by Realm Works on import. Create one relationship for each match between the KEY field and the SEARCH column. Add placeholder text to the relationship field boxes. v1.46 This version introduces relationships. This allows you to specify a particular column of the table which will contain a value that can be linked to another row of the table. (Note: old saved projects will not load into this version.) The names of the relationships are currently based on the actual relationship which will be put into the XML file, rather than the "human friendly" names which are used inside Realm Works. There are two fields which need to be filled in with a column. The first field identifies which column in the current row provides a search value. The second field identifies the column which will be searched (across the entire dataset) for a matching value - if a row is found, then a relationship will be created to that other topic. (Note that if you are using selective parents and don't the other topic doesn't end up in the output file, then RW will generate an error during import.) v1.45 Don't try to treat a parent topic title as a regular expression. It will now be treated as a fixed string, so "special" characters can now be used in topic titles (such as parentheses). v1.44 All categories are now shown in the category drop-down list (sub-categories were previously being skipped). It is now possible to set attributes on the public name of the topic/article (e.g. auto-accept, case-matching, match-priority and show-in-nav-pane). The name of the case-matching has been shortened from "Auto Correct" to "Correct" to make the field shorter. v1.43 This fixes a bug where a blank element was being put into the contents section when an "overlay" was present in the topic's definition in the structure. v1.42 (the Meaning of Life edition) This fixes a bug in that the reveal button on "other" names was set to be exclusive across all names. This is no longer the case. v1.41 This version simply presents Tag_Standard snippets the same way as Hybrid_Tag snippets, in that a CSV column can be assigned to choose the corresponding domain tag for the field. v1.40.1 A bug-fix release so that Realm Works doesn't crash when importing a file generated from an XLSX file containing multiple paragraphs in a single cell. (Multiple paragraphs are separated by blank lines in the cell.) v1.40 Fix XLSX file reading so that "<" are copied to Realm Works properly, as are cells which contain only HTML. v1.39 If you have BOLD, ITALIC, UNDERLINE and/or STRIKE-THROUGH in your XLSX files, then this formatting will be imported into Realm Works. v1.38 Provide the option to read XLSX files as an alternative to CSV files. v1.37 You can now save your current project, and load it back later. The names of the CSV and RWEXPORT files are saved as part of the project. Your mappings within the categories is also saved (although if no field is mapped to the topic's name, then that category won't get saved). v1.36 Fixes a bug where the import would fail if the category has child sub-categories (e.g. the "Location" category has "Settlement" and "Adventure Area" sub-categories; so any attempt to create "Location" topics would fail). v1.35 Fixes a bug where the import would fail if the category definition has sub-sections within sections. v1.34 It is now possible to create multiple content snippets in the main body of a section. Individual snippets can't be duplicated, but the general "contents" snippet of the section can now generate multiple snippets. v1.33 Allow a section to be marked as starting in the "collapsed" state. Allow a CSV column to be used as the name of a section within a topic. Allow a set of CSV columns to be used to create multiple sections within a topic. See http://forums.wolflair.com/showpost....&postcount=295 for more details on how to use this functionality. v1.32 A small update to add a menu option to choose what to use for the field separator in the CSV file. You can choose from comma, semi-colon, or the windows list separator. v1.31 The default separator in CSV files is now read from the Windows Regional Settings (the "list separator"). If you don't want to use your language's default list separator, it can be changed by opening windows Settings (windows 10) and finding the "Region & Language" section. Click on the "Administrative language settings" under "related settings". Go to the "Formats" tab, and click "Additional Settings...". Here you will see "List separator:" where you can change the default separator. v1.30 You can now define more than one category to be exported from the tool (e.g. spells and items). Each category definition can have its own set of parents. The new "key" icon in the title of the category allows you to specify which rows in the CSV table should be displayed in this category (based on a column having a specific "key" value in it). Note that separate parents are created for each category, so for example it isn't possible to create some spells and some items and have them appear under the same parent topic/article in Realm Works. v1.29 The same category can now be chosen for each level of parents. In theory (untested), you could even select the same category that you've chosen for the main topics. This removes the previous restriction that all parents had to be using a different category. v1.28 For each snippet, it is now also possible to set the Veracity of that snippet (True, Partially True, Untrue). The selection of snippet style has been merged into the new Options menu. v1.27 Adds support for numeric snippets. v1.26 The "Import Name" text box has been replaced with a button to open a File Details dialog in which the import name, as well as other details, can be entered. v1.25 Individual "<" are now treated properly so that they don't cause an error on import. However, any field which starts with "<" and finishes with ">" is left unchanged, since it is assumed to be HTML that you wish to have imported directly. v1.24 Don't include "overlay" elements in the generated RWexport file. (There is no v1.23) v1.22 You can now enter the name of the RW import before generating the output file. The installer has been changed from QTIF to NSIS. v1.21 You can now set the Snippet Style on the snippets that you're going to be importing. v1.20 It is now possible to specify a hierarchy of parents. This means you could have spells organised by sub-school under a more general school topic; or you could have all your traveller planets stored per sub-sector and then per sector. At the moment, a different category must be chosen for each parent, as well as for the main topic display. Icons have been added to buttons to make their actions clearer. v1.19 When defining a parent category, you now have the choice of being able to have more than one parent in your output file. By dragging one of the CSV columns into the parent's name field, then there will be one parent created for each unique string in that column of the CSV file. The rows of the CSV table which contain that same value will then appear under that parent. (For example, you could copy the "school" column of a spell CSV into the parent box, and then the RWexport file will define one parent topic/article for each spell school, and will have the spells for that school as children.) v1.18 (thread comment #166) True Names and Other Names can now be imported from a CSV file. You'll find a new "+Name" button at the end of the topic title's row. Pressing the button once will add a line where you can define the True Name for the topic. Pressing it again will add additional rows for defining Other Names for the topic. As usual, the relevant CSV column should be dragged to define the True/Other name. If the CSV field is empty on a particular row, then no alias will be added for that topic. Several Other Names can be added and the various Realm Works name flags can be set differently on each Other Name (revealed, Case Matching, Auto Accept, Shown in Nav, Priority). So depending on the contents of your CSV file, you could have one CSV column for "revealed" Other Names, and a different CSV column for "unrevealed + priority avoid" Other Names. v1.17 When reading images from a web URL, follow web redirects which are issued by the web server. A progress bar is also displayed when performing the GENERATE, so that you can see how quickly (or slowly) it is working. v1.16 When loading an image (see V1.11) it is now possible to specify a web URL instead of just a local file name. Note that the URL must point directly to the image, not to a web page which contains an image somewhere inside it. The GENERATE button will now be disabled when the file generation process is in progress. This helps you know when the generation is complete (accessing files using a web URL can be very slow). v1.14 When importing blocks of text, convert URLs so that they appear in Realm Works as clickable links. v1.13 A mistake with github meant that the wrong code was tagged, so ignore this version (typical for number 13 anyway!) v1.12 This version fixes a bug in the CSV reader where a " at the end of a line was not being handled properly. v1.11 Need to load HeroLab portfolio files for all your favourite NPCs? This version extends the file loading to allow loading data for the following snippet types: Portfolio, Statblock, Rich_Text, PDF, Audio, Video, HTML, Foreign. v1.10 Load images into the RWexport file... A column is chosen in the CSV file to use for the Picture or Smart-Image snippet. The text in the column is the name of the file to load as part of the GENERATE process. The file can either be a full or relative path to the file; if no path is given then the file must be in the same directory as the CSV file. If the file does not exist, then a warning will be generated during the GENERATE process. The resulting file can still be loaded into Realm Works, in which case the snippet will show the name of the file even though the image itself will remain blank. (Also, upgraded from Qt 5.8 to Qt 5.9.2 - this should have no impact on users.) v1.9 Multi-line quoted cells are now supported in the CSV file. Also, when you one of these fields then a blank line is created as a paragraph break when the data is being put into a Realm Works field which is either a Multi_Line or Label_Text field (the two fields which allow more than one line of text). v1.8 This release fixes an issue where the RWEXPORT file would contain one or more blank topics if the original Realm Works structure file contains a category defined as a child of another category. (E.g. Location in the basic Pathfinder category structure). v1.7 A small update which ignores blank lines in the CSV file. This prevents an empty topic being created in the RWEXPORT file which caused Realm Works to reject the import. v1.6 A minor update which remembers the last path you used for each of the three file dialogs (each is remembered separately). A tidy-up of code means that the text in the structure section is never mixed up with text in the topics section. Multi-line text is now possible. The lovely (R) has been added to all uses of the term Realm Works(R). v1.5 This release allows you to type in some free text into a snippet, or choose a tag for a choice field, instead of having CSV data appear in the field. You can't mix CSV data and free text in the same snippet (yet) though. It also fixes a bug where sometimes a partition in the structure part of the XML file would wrongly get text attached to it. v1.4 The entire topic, as well as the optional containing topic, can now be marked as REVEALED to players (using the circular button to the left of topic name). This updates the existing functionality where individual snippets could be marked as revealed. v1.3.1 A minor update to add validation to the new optional parent. You must enter a name, and you must choose a different category for the parent to the one chosen for the CSV data. v1.3 This release see the addition of an option to allow a parent containing topic/article to be specified for all the topics which are to be created from the CSV file. v1.2 The installer now adds the application to the Start Menu. Fixed a bug with the layout of elements when there was a blank snippet in the template. v1.1 Blank cells in the CSV file don't produce unnecessary elements in the output file. Ensure that the annotation child appears before the tag_assign child in relevant elements. Ensure the name of the topic/article is assigned one of the CSV fields. Remove the "GM only" button from each snippet, since it doesn't do anything. v1.0.0 The top drop-down menu does nothing. TO DO Improve the CSV value => Domain mapping so that partial matches are possible (comma separated lists of multiple tags is already supported). Won't Do The following items should really be handled by setting up the required category definitions within Realm Works before exporting the structure: Provide support for Labeled_Text fields. Allow the addition of additional text (and other) snippets to an imported category. Allow the addition of new sections and subsections to an imported category. The following idea can be achieved by re-organising the contents of the CSV file: Allow predefined text and CSV field names to be mixed together within certain text snippet types. Last edited by Farling; September 19th, 2021 at 04:12 AM. Reason: v2.14 |
![]() |
![]() |
Senior Member
Join Date: Dec 2010
Posts: 468
|
This looks like a very great addition to our toolkits. Can you elaborate on the instruction regarding how to do domain selection?
Any good CSv sources out there? |
![]() |
![]() |
Senior Member
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,622
|
Quote:
From the list of CSV fields in the left panel, drag the name of the field across onto the box that contains the black triangle (to the right of the field name). For example, with the Pathfinder game, if you have a CSV for spells, then the column for the "spell subschool" might contain the text "scrying". My import tool would take the text "scrying" and find the correct tag to put into RealmWorks for this text. |
|
![]() |
![]() |
Member
Join Date: Jan 2016
Location: Sydney, Australia
Posts: 48
|
I must be doing something wrong somewhere along the line as each time I go to import after I've generated the export file I get the message that the category does not exist, any advice? Tried with feats and also magic items.
|
![]() |
![]() |
Senior Member
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,622
|
Quote:
Is the message about the category not existing coming from my tool or from Realm Works? Which fields from a CSV are you trying to copy? It is best not to use fields which have lots of HTML mark-up in them since this will likely conflict with the HTML being added by the conversion tool, and also with what Realm Works might be expecting. |
|
![]() |
![]() |
Member
Join Date: Jan 2016
Location: Sydney, Australia
Posts: 48
|
Quote:
I'm just using basic text fields from the CSV files from http://www.pathfindercommunity.net/home/databases/, I have already managed to get the spells imported with EightBitz's script so I attempted to bring in the magic items and feats, both gave that error. |
|
![]() |
![]() |
Senior Member
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,622
|
Quote:
|
|
![]() |
![]() |
Member
Join Date: Jan 2016
Location: Sydney, Australia
Posts: 48
|
Quote:
![]() https://drive.google.com/drive/folde...Vk?usp=sharing |
|
![]() |
![]() |
Senior Member
Join Date: Mar 2013
Location: Greater London, UK
Posts: 2,622
|
Absinthe,
I'm trying to import your "feat text.rwexport" file at the moment into a newly created "pathfinder" realm. My attempt is currently at 23% and still going. What type of realm are you trying to import into? The first improvement to the import would be to drag the feat type across to the "selector" in the feat template. The selector is the box containing the triangle. (It will detect multiple tags for the same field in the CSV if they are separated by commas within the CSV field.) You can have the same CSV column appear more than once in the template. See the attached picture Last edited by Farling; April 17th, 2017 at 02:17 PM. |
![]() |
![]() |
![]() |
|
|