RealmWorks Database Format Documentation

[AutoDMC]

This is yet another Export/Printing thread! Kind of!

I ran a couple of grep experiments on the "backup" files in the Documents folder for my two "realms." I'm able to locate content from both of my realms within. It appears that each data point is tied to a unique ID number, in the format of:

[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}

I haven't yet determined the database format, or the delimiters used in the file. Unfortunately, the grep tool I have for Windows chokes on the database file every time I tap a key, making the process a bit long winded. There's some interesting strings in there, and I've tracked down fragments of text from my realms floating around, but I haven't yet been able to figure out how everything lines up.

Do you use some kind of recognized database format (LayerDB, Built in SQLite), or can you provide some sort of database format specification?

According to your blog, you've put export on the bottom of the feature list. Despite it being a hot topic on the forums, your survey got it ranked low. I can see why you'd do that, I really can.

So if you're not going to release an export tool, could we at least get some pointers on how to deconstruct the database file? I'm sure I'm not the only one of the forum able to throw grep and sed at a file; we just need some help disentangling the format.

I'm not just asking for the format to be an ass, I have reasons I'm going to rant about below. If you're the TL;DR type, you can stop here I guess.

I know you guys are all fired up on web tools and realm hosting and all that Web 2.0 jazz. And I may be interested in using RealmWorks as a tabletop manager for my game. I like the ability to manage the player's screen with popup windows and maps; I actually use a digital table (HDTV), and not having to constantly peer over the GM screen and try to flip and reverse mouse movements in my head is nice.

But here's the simple fact of the deal: I don't want to spend time putting data into something I can't get it out of. No matter how many whiz-bang gadgets you put into RealmWorks, if I can't make a proper export that I can manipulate outside of your tool, I simply can't be assed to use it.

I know you guys have made a "pledge;" you'll release an export tool as the very last action should the ship sink. I know you want this to be something that gives us trust but frankly? It doesn't do it for me.

Maybe I'm just jaded, but after being a PC gamer for a decade I don't trust software companies. (Full Disclosure: I work for a software company.) You can make all the promises you want, but unless you write the code, it's just words.

The difference between promising to release the web view or the content market, which I fully trust in, and promising to release an export tool, which I completely don't trust as far as I can throw a frost giant with a STR mod of -2, comes down to this:

The Web View, and the Content Market will make recurring money.

An Export is seen as something that won't make you any recurring money.

This is why I don't trust you: an Export to text files would be a rather simple feature to produce (relative to the rest of the software, of course. It could be a difficult task depending on your data structures), but because (at best) it would help make initial sales and, at worst, it'd just keep a few disgruntled users around who have already given you their cash...

...it tells me a lot that it's a feature promised to happen just before the ship sinks, and not a minute sooner.

I find myself having to leave your tool turned off, and go back to managing text files and loose images on a hard drive, because at least those I can manage using any tool I want. I have lots of experience juggling text files. Your tool is something I have to learn to embrace fully.

Additionally, I'm trying to package my gaming materials for use by other game masters. I want to put together some documents for them, and I'd love to use RealmWorks as a mind mapping tool. You guys want me to do this with your yet-to-be-released vaporware market; I get that. But you're not going to get me to invest in your market if I have to depend on your market. Again, I don't trust a database I can't get my data out of. An XML file I can write an XSLT to transform into paper documents. A SQLite database I can write queries to output reports I can compile into offline documents. Your file..? I can grep out random segments of text. Which I have to sort by hand. "strings" only goes so far in making sense of a binary blob.

I was naive. I pre-purchased based on the features, with the hope that I'd be able to use your $50 tool to map out my campaigns, throw some maps up on a secondary monitor, and then export my data for use later. My bad, I guess. Right now I've got two realms with like 15 data fragments put in. I've mapped like three characters and one mission. I can't be bothered to go any farther than that, because once I'm done working in RealmWorks, I have to duplicate the work elsewhere anyway. I might as well just do the work outside the tool you made.

I WANT to use your tool. I find the player's screen system to be great. But until I'm convinced my time invested in your tool won't become a black hole in the future because I decide to use a different table display or switch to paper handouts, I'm not going to invest my time in your tool.

And instead of telling people how much I love your software, I'll be saying things like "HeroLab is an awesome tool, you should buy it. But steer clear of RealmWorks, because you can't print or export your data from it."

TL;DR: If you're not going to bother to release an XML or text output engine until "it’s the last feature that ever gets into Realm Works (and even if Rob has to add the feature himself)," the least you can do is let us know some information about the file format so we can do the data recovery ourselves. The Internet is full of great programmers who know perl and sed and grep; if you really feel it's such a niche feature that only a few data nerds would be interested... at least give the data nerds the information they need to get the data out?

 

[AutoDMC]

For clarity:

I want to use your tool. I wouldn't be posting here if I didn't. I've blasted money on crap software I've never used again in the past, and will in the future.

There are some cool features, right now that I want to use. I've already used it at my table a couple of times to throw up a map and a few dossier documents, and it's pretty slick. And once I can allow my players access to my database without having to ask them to invest even more money into the game for the "player's edition," it'll be even more awesome. (I already require them to buy the handbooks and materials.)

But every time I open up the software, I fill out about a third of an article before I realize that every drop of effort I put into putting data into RealmWorks is a duplicated drop of data I need to put into putting the data elsewhere. One of my players likes handouts that they can scribble on, for example. My backup computer can't run RealmWorks, but I can get Dropbox to work on it, so I can access text files but not RealmWorks databases.

The confinement of my data into the proprietary database keeps me from using the tool I want to use. And until I can get my data out, I can't be bothered to put it in. I won't be paying for cloud service. I won't be buying content on your market. I won't be pitching player's editions or logging into your website. Until my data is actually my data, to use any tool I wish with it, I simply can't waste my time duplicating effort.

And instead of enjoying using your flexible awesome tool, instead I open it, sigh, close it, and then go write my notes in Notepad.

I love (I mean, seriously, love) HeroLab. I hand out printouts of character sheets for the 20+ games I run at conventions, and always pitch the product to my players. I've had a couple buy it and use it for themselves because of that. I can't in good faith pitch RealmWorks the same way, or use it at the table, until I can get my data out. At least I can get a PDF from HeroLab.

 

[Farling]

TL;DR: If you're not going to bother to release an XML or text output engine until "it’s the last feature that ever gets into Realm Works (and even if Rob has to add the feature himself)," the least you can do is let us know some information about the file format so we can do the data recovery ourselves. The Internet is full of great programmers who know perl and sed and grep; if you really feel it's such a niche feature that only a few data nerds would be interested... at least give the data nerds the information they need to get the data out?

The danger with LWD providing any information about the database format is that people will shout and scream when the format is changed to implement new features.

 

[Viking2054]

you also have to worry about people trying to pry out copyrighted material purchased through the marketplace. If that material was passed around and patched into other peoples RW or other databases that could be a problem for LWD.

 

[Acenoid]

I'm for full access to entered data that does not belong to 3rd party. Similar to any office application. But until there is some newer survey or discussion brought up by LWD I don't think there is much ground for newly discussions topic did come up in the past several times already...

 

[Parody]

The (local) database engine is Firebird; you can tell that by looking at the DLLs in the Realm Works install folder.

The ID numbers are GUIDs, so you're missing the 12 hex characters on the end. Even if they're always the same in that file, they're part of the identifier.

 

[AutoDMC]

Good catch, Parody. I prematurely ended my copy-paste. I also missed the Firebird DLLs, I was looking at the files themselves. My mistake!

I can recognize the worry over third party content, but as of yet there has never been a successful DRM system. If they want to put the third party content in it's own database, or slipstream it encrypted into the main database, that's fine. As long as they don't encrypt MY data, I wouldn't care if purchased third party data came out as gobbledy gook. Just as long as nobody is under the delusion that DRM will do anything but make the product more frustrating for legitimate users.

Additionally, I understand them passing up on an IMPORT function if they're worried about third party copyright issues, just not an EXPORT function. A huge XML file full of D&D5th edition content isn't any more useful than the Player's Handbook if I can't import it. I don't want to export my data as a backup, I want to export my data because it's mine, and i want to process it in other tools as well. They can give us EXPORT without IMPORT.

Finally, we don't need another survey. Question #33 on the Kickstarter FAQ states:

Realm Works will not have export capability in its initial release. As with most products, our focus in the initial launch is to have Realm Works do its own tasks exceptionally well, and to efficiently get material into the product. We wholly expect there will be demand for export to other products and formats, and we'll shift our focus to include those capabilities once we get the core product off the ground.

The core functionality of the product DOES work, as does cloud syncing with a player's edition. The core product is "off the ground." Now's the time to provide a feature they've been pushing off.

If I'd have known this feature was going to languish until the day Rob writes the code himself because they're closing down shop, I wouldn't have even been a backer, and I wouldn't have the angst now of wanting to use a tool, but not being able to because a core feature (for me) which was promised is still missing, a year and a half plus later.

Sure, I'd still be using text files and Adobe Illustrator maps but, please understand, that's the workflow I have to use now, still, despite owning RealmWorks because I can't export my data for processing.

 

[AutoDMC]

I've attempted to open the backup databases and the MASTER.REALM file in FlameRobin, which is supposed to be able to read Firebird embedded databases, however, I've been unable to get the client to read any of those files.

In theory, embedded database files should be able to be opened directly; the password SYSDBA supposedly gives "root" access to an embedded database, but I get errors saying the resource is invalid.

Anyone else given any effort toward opening these database files? Any pointers in getting access? I've got some experience in report writing if I can get access to the database.

 

[rob]

The core functionality of the product DOES work, as does cloud syncing with a player's edition. The core product is "off the ground." Now's the time to provide a feature they've been pushing off.

That argument can be made for dozens of features that users have asked for and that are on the todo list, not to mention outstanding capabilities that have been considered part of the core product vision from the beginning. A few that spring immediately to mind include the content market, web access, calendars, journals, individual player reveal, the list goes on and on...

We did the survey at the start of the year to find out which of these many capabilities are COLLECTIVELY considered most important by our users. We're prioritizing (and in some cases, re-prioritizing) our efforts based on the survey results. Our rationale for this approach has already been discussed at length elsewhere here on the forums, so I'm not going to rehash it all again in this thread. I encourage you to take a look at the other threads for more information.

If you HAVE to extract your content from Realm Works in order to use it NOW, then you are in the tiny minority of users. Unfortunately, that means the feature you consider mission critical isn't going to be added in the near term. Given your personal requirements, which aren't shared by the majority of our users, it's possible that Realm Works isn't a viable option for your needs at the present time. That's unfortunate, but it doesn't change our plans.

You've twisted our words for the sole purpose of hyperbole and saber-rattling. Sorry, I'm not going to engage beyond clarifying misinformation. If you don't trust us simply because we're a software company, that's entirely your choice - although it's one that seems NOT to be shared by the vast majority of our users.

Since you quoted a strategically snipped version of the statement we made, I’ll clarify it here for others. The pledge began with “Even if it’s the last feature that ever gets into Realm Works (and even if Rob has to add the feature himself)…” [emphasis mine]. That does NOT mean the plan is to never add the feature until the “ship is sinking”, as you’ve portrayed. It simply means that we pledge to ensure users can get their data out if that event should one day occur. You’ve chosen to read into it vastly more than what was said, and you’ve then distorted the entire matter here in this thread.

In a subsequent post, you asserted that we could readily provide export without supporting import. Well, the problem IS the export. If you can get published data out, you can then do ANYTHING with that data. You don’t have to put it back into Realm Works via an import. You could pretty it up and post it on a website, or give it to others. That’s what the publishers are worried about. They don’t care whether you can put the data back into Realm Works – they care whether you can get it out in the first place. So that complicates any efforts towards providing export, as does the fact that the data is split across dozens of tables, and anyone who wants to get their data out will want it nicely organized. When we provide export capabilities, we’ll do it well, and in a way that is truly useful to users who want the feature – not in a quick-and-dirty manner that would suffice for your needs but few other users.

As @Parody pointed out, we use Firebird as the database engine. However, that's not going to be sufficient for extracting data from the database manually. Key realm content within the database is encrypted, which was a decision driven by feedback during the early stages of development. There are common usage scenarios where GMs and players share the same computers in a household, so it was requested that we safeguard against tech-savvy players sneaking a peek at a GM's material by going straight to the database. Since we have to encrypt purchased content anyways, we simply encrypt key content regardless of its source.

Based on the sensationalized narrative you've already presented and your proclaimed distrust of honest folks just because they develop software, I won't be surprised if you take this information and characterize it in an alarmist manner. However, please understand that, by doing so, you are willfully working to undermine a product that you claim you want to use. That’s exactly what actively telling others to avoid the product amounts to, especially when it’s simply because you don’t trust software companies in general and the product lacks a feature you consider necessary to satisfy your self-professed jaded perception. If you think telling others to stay away will in some way pressure us to give you the feature you want now, I can assure you that's not going to work. All it will do is lessen the chance that the product evolves to a point where it's successful enough that we DO have the resources to add the feature you want. I don’t have any problems with you saying Realm Works lacks a feature YOU value, but let others make their own choice whether the product meets their requirements – which, if our survey results are any indication, will usually be very different from yours on the matter of export capability. Striving to scare users off because Realm Works lacks a feature few of them actually care about is essentially fear-mongering, and I respectfully ask that you not do that.

How you choose to proceed is entirely your option. What I ask is that you be mindful of what you're doing as you do it and consider whether your path will actually give you what you ultimately want.

Thanks to everyone for reading through to the end.

 

[Chemlak]

Thank you, Rob.

 

[Acenoid]

Haters gonna hate jokes aside, I think most of us understand that the current way the economy works today lw has to protect their investment. and to ensure that they will be able to take partners into the project, they a forced to ensure a high level of security.

I really like it by the way how lw reacts to critismn. Hopefully such threads will lead even to more users instead of scaring some off, as it shows how you deal with 'hot' topics.

I hope that sometime in the future the export functions will be added, maybe a good way to ensure safety for copyrighted material would be a separate key. And the other key could be set in the gui by the realm owner.

Ok, back to work; habe a nice Friday and of course an even better weekend

 

[rob]

I hope that sometime in the future the export functions will be added, maybe a good way to ensure safety for copyrighted material would be a separate key. And the other key could be set in the gui by the realm owner.

We're working through all these issues right now as we focus on the Content Market. So the basic distinction between copyrighted/published and user-created content is being designed into how it all works. It gets more complicated with the notion of users being able to share (i.e. publish) content to themselves so they can create a master realm that gets inherited into multiple instances for running different groups through the same world. But I believe we've got a handle on it all.

Please note that we're laying the foundation for this now - we're not actually implementing any export capabilities. The foundation has to be in place for many different aspects of Realm Works, and the export logic is simply one piece that will ultimately build on that foundation. I just don't want anyone misinterpreting my comments here, so forgive me if I'm restating something that you consider obvious.

 

[MaxSupernova]

Dude, stop pissing Rob off about export/print! It's the one feature I want the most! :-D

I keep trying to rein in my frustration about print/export/web view just so it doesn't turn into "That feature request they hate". I'm not very good at restraint, but I try.

Heck, if I kept getting abuse about something, it certainly wouldn't be on the top of my "want to do" list.

Hopefully Rob is a better man than I.

 

[AEIOU]

Thanks for the insights again, Rob. It's easy for us users to think something is simple. Heck, "export" is just one word. How much code can it take to make a button? But that button is just the tip of the iceberg that we can see. I don't even want to know how many countless hours have been spent planning and implementing the infrastructure already and how much is still ahead of you to code that export button many of us rabidly want.

 

[AutoDMC]

First off, I am apologizing directly to Rob, and to the rest of the Lone Wolf team, for the insulting tone of the last half of my posts. It's always recommended to not write when tired, a rule I constantly abuse, to my own demerit.

Secondly, I will give you credit for not banning my account. I'll admit, I've gotten rather used to the hot-headed insta-ban knee-jerk response some developers have for anyone who complains about their software. It says a lot about Rob and the crew here that I'm able to post this response.

However, I stand by my point that having a long-term license to my own data is not the same thing as owning my own data. Having my data locked in somebody else's system in such a way that I cannot extract it is not "owning" your own data. As nice as Realm Works is, and as much as I appreciate the complexity that goes into a programmed project (especially one as content driven as RealmWorks), I have to point out that not having data export as an early feature shows me what the company thinks my data is worth to them.

I recognize the complexity of third party content, but I also recognize the flexibility of a SQL database, and the ability to tell Firebird to encrypt specific columns or tables. In products I have worked on, many data nuggets have been encrypted and locked out to protect sensitive data (such as credit card numbers), but in no way was the ability to export a SQL dump ever taken away. And I've spent many, many months writing data export functionality; the complexity usually lies within permissions related to the data... beyond that it usually involves shoving a big data structure from the SQL server to a library that outputs files.

The upshot of this thread is I know that I can go back to my OneNote archives and OneCloud file sharing, using PoV aware mapping tools, secure in my knowledge that RealmWorks is, most definitely, not the product I need. Honestly, the BIGGEST feature of RealmWorks, the one that made me really, really desire an export, were the pre-structured ingame objects. I could create fleshed out content, easily, because the software had specific fields for specific questions. That was the killer feature for me; but I'm sure I'll find a workflow that doesn't require the software.

Thanks to those who spend the time reading to the bottom. If having a license to your own content is just fine for you, RealmWorks is a great product, and you will totally enjoy it's power. If you desire the ability to export your content, RealmWorks is not the product for you, and obviously will not be for a very long while (antangonism from the forums or not). Which is fine, you guys write the software you want to. I guess maybe I just suffer from a bit of buyer's remorse. Isn't the first time, won't be the last.

 

[rob]

However, I stand by my point that having a long-term license to my own data is not the same thing as owning my own data. Having my data locked in somebody else's system in such a way that I cannot extract it is not "owning" your own data. As nice as Realm Works is, and as much as I appreciate the complexity that goes into a programmed project (especially one as content driven as RealmWorks), I have to point out that not having data export as an early feature shows me what the company thinks my data is worth to them.

I must respectfully disagree with your conclusion here. This is NOT a reflection of what the company thinks of your data. Not having data export shows you what USERS (i.e. your peers) think the value is in getting their data out of Realm Works compared to all the other features we could be working on. If export had been a high priority across users within the recent survey we completed, it would have been prioritized accordingly. It wasn't.

Please don't personalize this, as such conclusions are fundamentally flawed. Your PEERS spoke very loudly. We're simply listening to them as a collective whole - instead of letting a few disgruntled voices from the crowd dictate our priorities.

And thanks for making this a much more civil discussion this time around.

 

[Viking2054]

Out of curiosity, how hard would it be to turn on/off the encryption of user entered data? I'm guessing that if he could simply turn off the encryption for his own content then he could theoretically write his own export/dump requests until you guys at LWD got around to giving us an export feature. Of course I would expect third party content we purchase off the market to not be affected by such a theoretically simple feature. You could potentially make the guy happy and just maybe he could provide some tools to the community as a stopgap for those that want to complain about exporting their stuff.

 

[rob]

Out of curiosity, how hard would it be to turn on/off the encryption of user entered data? I'm guessing that if he could simply turn off the encryption for his own content then he could theoretically write his own export/dump requests until you guys at LWD got around to giving us an export feature. Of course I would expect third party content we purchase off the market to not be affected by such a theoretically simple feature. You could potentially make the guy happy and just maybe he could provide some tools to the community as a stopgap for those that want to complain about exporting their stuff.

How do we ensure that users are only ever allowed to apply this to THEIR content? How do we make sure someone can't exploit this capability to readily gain access to published content? Any openings we introduce like this become potential attack vectors for hackers, so we have to make sure they are well-conceived and carefully protected against such attacks. That process requires lots of thought, and usually lots of accompanying work, turning what seems like a simple and useful solution on the surface into a typically very complex problem to solve correctly.

I'm not trying to be dismissive of the idea. It's both appreciated and a good one in principle. The gotcha is that the devil's in the details, and things that seem easy at first often become nasty to solve once we really dig into them. This particular idea is a perfect example, and we run into these gotchas all the time with a product as complex and diverse as Realm Works. In fact, these sorts of gotchas are the primary source of all the delays we've incurred over the past few years.

 

[Viking2054]

I understand completely, it just sounded like you had at least considered the idea when you mentioned that it was at the request of the beta testing team that encryption be used for player entered content.

Personally, I'd rather have a really good printing system that would allow me to print out my own creations in the form of a semi-polished module. But I'm just wishing, not expecting, at that since layout and design would probably require sending the information to a desktop publishing program like Publisher for layout.

 

[Azombiod]

Just to reply to this thread and countless others I've been reading... I too feel slightly duped although I do understand this is my fault for a lack of research. To me this seemed like it was going to be a tool like Tabletop Simulator. With a user community building mods and sharing ideas for campaign and eventually selling them if they wanted. This DRM obsession is made this software useless for me but more importantly I think the survey is disingenuous. I personally think based on reading these forums all day and a few other gaming forums that it does not reflect the wider view of the audience. Tabletop games only survive still because of the user base support of them. It's literally the easiest form of game to pirate and I download every book I get in PDF (even if not released as such) as a backup/ease of use. I play EotE and some guy had to do high res scans of all the books but still did it. Even if you say make it so you can't used pirated material well what stops someone from simply making the go bots version with a understanding to go get a third party program that runs a macro and changes the necessary entries. Perhaps that's a bit extreme but I think the point needs to be made that piracy is unstoppable and this industry still exists in spite of it not because of measures to protect against it. Personally the DRM nature and lack of access or ability to simply give away or trade your data much less edit or print it for yourself makes this product go from something that would have made you money on sales alone to most likely a mostly online tabletop with a niche of a already niche market. Personally, and this is just my thoughts a model similar to tabletop simulator should have been used with player driven content being the focus and simply removing content as cease and desist orders come in like every other company on the internet. Anyway, that's my two cents and I thank you for the product... I really had hopes for it and will keep checking back but in it's current form it's useless to me. One more point to add is that perhaps encryption should have been added during the marketplace update so we could actually make our own stuff in the meantime... unless you're argument is what if you make a book in our software and distribute it in which case I would argue does Adobe or Microsoft Office get sued for the exact same thing? Or forums get sued for the exact same thing? I'm just confused on that point I guess. Anyhow, sorry for the stream of conciseness and thanks for reading.