Question to Lone Wolf about Import/Export file details

[Zaphod Beebledoc]

Not sure how to put this, but here goes.

What details about the import/export files would you want us to discuss on the forums?

Farling's thread about his import from Obsidian Portal has got me excited about the prospect of importing my data from spreadsheet files.

I don't want to say the wrong thing or post screen shots of stuff you would rather be kept off the forum. Hence me asking. :)

[Acenoid]

Not sure were the big secrets are, if its about copyrighted content and such, you could always use one of those useful quick editors to blur certain text / images / passages out?

I think http://getgreenshot.org/ or https://getsharex.com/ are good.

Tried both.

[Zaphod Beebledoc]

I was thinking the more of technical details of the files, like the XML tags with the data inside. Most of the content I would be trying to put into RW is my own anyway...

[Farling]

There's nothing wrong with creating an XML file structure which matches the export file that you can create from a realm.

I must say that manually creating the XML file from other data is quite hard to do, particularly getting uuids sorted properly.

[Zaphod Beebledoc]

A thought came to me last night about uuids and signatures. Why not have a substitute for them them indicates to RW that a uuid or signature is needed?

[the_redbeard]

A thought came to me last night about uuids and signatures. Why not have a substitute for them them indicates to RW that a uuid or signature is needed?

Sounds like a valid feature request that could help those importing content.

[Zaphod Beebledoc]

Good point, I'll pop it on the Feature board...

[Parody]

A thought came to me last night about uuids and signatures. Why not have a substitute for them them indicates to RW that a uuid or signature is needed?
RW should either create anything you omit or report a problem with the import. You shouldn't have to put in a substitute.

A second import without IDs would probably duplicate everything rather than make any attempt at conflict resolution, but that's just a guess.

[kbs666]

I'm pretty confident that RW will not import a file that doesn't contain UUID's in a form that it expects.

[Farling]

A UUID has a very specific form in terms of hexadecimal characters and "-" separators. I don't think that RW follows the UUID rules about how the digits are chosen, since there is no need for the UUID to be unique worldwide outside of realmworks.

The only UUIDs I have in the OB script are those generated by RW in the first place when I exported the realm structure that I would use for the topics I was creating.

I would like to create my own UUIDs for the topics, so that I can re-import. I may have a look at this later today.

[Bidmaron]

Farling, can you explain your usage case? There is nothing stopping you from "reimporting" now

[Farling]

Farling, can you explain your usage case? There is nothing stopping you from "reimporting" now

All the snippets and topics need tagging with their own UUIDs which don't change from one import to another, otherwise all the existing snippets and topics will get duplicated by the second import.

[Zaphod Beebledoc]

I know this is a bit convoluted, but if you exported the topics once you have them in RW, you would have the UUID...

[Farling]

I know this is a bit convoluted, but if you exported the topics once you have them in RW, you would have the UUID...

You would then still need to match them back up to the original topics in the source XML :-o

[Bidmaron]

Farling what is your use case? You can export and reimport now

[Farling]

Farling what is your use case? You can export and reimport now

My use case is importing data from Obsidian Portal, where I am creating my own XML file to be imported. In this case I will have to create the UUIDs manually from some other uniqueness provided by Obsidian Portal (likely to be a hash of the URL of the thing being converted) so that a subsequent OP backup can be re-imported to update the existing data in the realm with the latest changes.

(Which is the whole point of this thread, import of our own data not export and import of RW data.)

[kbs666]

Hopefully Rob will answer the question if it is asked directly,

Can RW import an XML file if it was not created by RW but is in the correct form. We can surely provide some form of UUID's and signatures but they would not be what RW itself expects, obviously, unless LWD reveals how those are created.

[Bidmaron]

What is to stop you from creating a bogus realm with a bunch of blank entries, exporting them and comandeeeing the uuids for import?

[Silveras]

Farling is working from a previous answer to that question. He asked previously if manually-created XML is possible to import. It is.

As I understand the present conversation:

The problem he has is that if he wants to re-extract from the same Obsidian Portal source later, he has no way to automatically put the uuids in place in the XML other than 1) exporting the imported data from Realm Works just to get the uuids and 2) manually updating the new extract from Obsidian Portal with the uuids so that imported records from the new extract are recognized as updates of the existing topics, and new duplicate ones are not created.

[kbs666]

Farling is working from a previous answer to that question. He asked previously if manually-created XML is possible to import. It is.

As I understand the present conversation:

The problem he has is that if he wants to re-extract from the same Obsidian Portal source later, he has no way to automatically put the uuids in place in the XML other than 1) exporting the imported data from Realm Works just to get the uuids and 2) manually updating the new extract from Obsidian Portal with the uuids so that imported records from the new extract are recognized as updates of the existing topics, and new duplicate ones are not created.
Could you link to that post I missed it.

[Silveras]

I am reluctant to link to it as it is in the beta forum, and not everyone would have access to it.

Users with beta access can find the thread (2 pages) under the title "Import of own data".

[daplunk]

The thread will only be accessible to beta participants.

[kbs666]

Then how would the rest of us know that?

[AEIOU]

@kbs666: We can't. The beta testers live in two worlds and they're trying to share some of their other world knowledge here. Given the volume of posts I'm sure it's hard for them to remember what has been discussed in public. Especially when public and beta are having the same conversations.

The take-away for me is the beta team is actively thinking about this, testing this, and talking to the devs about this. Unlike us, they have a much more direct link to them. So it's a win-win for us.

[Silveras]

It is especially confusing when the conversations are in both places, because the beta members have agreed to NDAs. In general, if the information has been announced in public, the NDA no longer applies ... but remembering which details are and are not public makes me conservative on what I say in the public forums.

[ShadowChemosh]

It is especially confusing when the conversations are in both places, because the beta members have agreed to NDAs. In general, if the information has been announced in public, the NDA no longer applies ... but remembering which details are and are not public makes me conservative on what I say in the public forums.
Yep. Pretty much easier for me to say almost nothing on the main forums anymore. :)

Worse for me is how much NDA stuff I have for Hero Lab I am not suppose to say. :p

[AEIOU]

Knowing the conversations are underway is comforting. When betas are managed well they are very effective. A big thank you to the beta folks that are watching the public forums and bringing our concerns to the devs so they don't have to spend their time poring over every post.

[Farling]

Hopefully Rob will answer the question if it is asked directly,

Can RW import an XML file if it was not created by RW but is in the correct form. We can surely provide some form of UUID's and signatures but they would not be what RW itself expects, obviously, unless LWD reveals how those are created.

Yes, you can create an XML file in the correct format, which it should be possible to import into RealmWorks.

If you look at the XML output of my script then the UUID and signatures are not required on the topics which you are importing. (They are required on the elements in the structure definitions, but not in the actual topics.)

(If you want to be able to import data more than once, then you will need to be able to put some form of unique id in for each topic; and use that same unique id when you re-import it. I assume this is achieved using the UUID attribute on the topic elements.)

[Farling]

What is to stop you from creating a bogus realm with a bunch of blank entries, exporting them and comandeeeing the uuids for import?

This is possible. You would have to create a lot of uuids to cater for adding more topics later on subsequent exports. You could also assign pseudo-random uuids using a microsoft library from within powershell.

The problem in making a general purpose script is to make the effort the least number of steps possible for a general user. So any use of UUIDs should ideally not be known by the average user of the import script.

[Farling]

I am reluctant to link to it as it is in the beta forum, and not everyone would have access to it.

Users with beta access can find the thread (2 pages) under the title "Import of own data".

I was answering Zaphod's original post in this thread :-)

[rob]

Various notes regarding the import/export mechanism and assorted questions asked above...

1. We have a formal XSD for the import/export file structure. One of the tasks on our todo list is to properly annotate it with explanations of everything. My guess is that it will be another week or two before that gets done, though.

2. Anything to do with the XML structure can be freely discussed here. Even if it's something we shared privately with the Beta team prior to release. The XML structure is intended to be a public format. We just haven't gotten it documented yet. :)

3. When we designed the import/export mechanism, we never considered the use case of someone repeatedly importing material from another source. We figured that content would come in and then potentially cycle back and forth with another tool, but we never envisioned continually importing from the same source - which is what @Farling is doing with Obsidian Portal. Consequently, there is nothing in place right now to handle the one-time generation of uuids that can subsequently be re-used. This is something we'll need to consider and solve in the future.

4. The "original_uuid" and "signature" attributes are something that should NOT be used when importing from another source. They have a very specific use in conjunction with import and are for internal use only - at least, until we get a chance to re-think #3.

5. The uuids used by RW are of the same general format as standard guids, but their composition is unique to RW. If you generate your own uuids, you absolutely run a risk of rendering all your data unusable due to the uuids being invalid.

6. Use of uuids not explicitly generated by RW will present the opportunity for errors, and the validity of uuids is critical to the proper operation of RW. Consequently, we won't be sharing how the uuids are generated. The only real possibility here (that we've thought of so far) is to enable RW or some other tool to generate a block of uuids that can subsequently be used.

7. Generating uuids from one realm and then using them for other content in a different realm is going to create a vast sea of problems that even risks contaminating other realms in the database. Please don't do that, as the results could be disastrous.

I have a suggestion for how you might work around things on a temporary basis that I'll post in the Beta forums. It's something that would maybe work now, but it will NOT work in the future, so it's not something suitable for posting here. A proper long-term solution is needed for #3.

Hope this helps!

[Zaphod Beebledoc]

...Hope this helps!

Absolutely! Thanks! :):D