File compression?

I asked this in an email nearly a month ago now and I got the automated response but never saw a human's reply. Time to try my chances on the Lone Wolf forum

I've been looking at the .ab file format at a low level and it appears to be a quite simple archive format. The main thing I don't know but am interested in at the moment is what the compression format is for the files. Relic's SGA files in Dawn of War just use deflated/zlibed, but the .ab files don't seem to contain be deflated/gzipped/bzipped (or at least not that I've been able to work with so far).

Can anyone enlighten me on the compression algorithm used?

Thanks,

IBBoard

For AB, we use a commercial compression library. It uses an adapted version of the LZW algorithm. The import files themselves are files produced by this compression library, and since they are non-standard, they can't be accessed directly via other tools.

This is something we've had on the todo list to revise for many years, but it all works, so it's never actually been addressed.

Ah, so when the documentation says about being compatible with PKZip, it doesn't actually mean it is compatible with PKZip? The many joys of proprietary, closed-source formats.

Was there a particular reason that a commercial and modified version was used rather than an open and standards-based version?

The documentation says it's compatible? Please point me to that comment so that it can be corrected.

We began using the current compression library in 1998. That was 11 years ago. Back then, the LZW algorithm was still under patent protection, so the *only* option was a commercial product with appropriate licensing. Since then, everything has been working smoothly without incident, so there has been no reason to change things.

Like I said, it's something that's been on the todo list for a long time. But I don't usually spend time changing something that is working smoothly when I'm focused on developing new stuff.

I'm not on the right machine to check for v3 at the moment, but someone on my forums posted a quote from the v2 construction kit documentation that says:

The ABExport tool is used to package up one or more Army Builder files into a single import file, much like PKZip would accomplish. The created file is compressed using a compression algorithm that is compatible with the one used in PKZip.

Click to expand...

From what I remember of the surrounding text it basically hints that a .ab file is like a .zip, except that .zip wasn't used because it was felt that people would try to extract the content manually. I know that's not strictly true and is more of an analogy, but it does mean that the compression algorithm is at least semi-standardised.

As for the original decision, does that mean that rather than license a library that implemented the patented LZW compression you instead licensed a based-on-but-not-quite-compatible-with-LZW library that must also have been under patent? (if it implemented LZW it should surely be covered by the same patent).

Also, from my investigations in to the DLL I'm guessing that the library is compiled in to the EXE. I've found references to ZLib and Deflate within the EXE (hence why I thought it might be a standard compression), so I'm either using libraries that are stricter on the standards, or those comments in the EXE are lying. I don't suppose you can say anything about the compression used can you, given that it is proprietary?

I've been looking at this as well, and the best guess I've had at the .ab format is that it uses TTComp, which was a tool/library developed by SWFTE.

As I say, it's a best guess and it could be completely wrong. I've had no success trying the tools I've managed to get hold of that claim to be compatible because they don't run in a 64-bit environment and I haven't had the time to try it on a 32-bit machine/OS.

I just had a look at the old ABExport.exe from AB2 (since that seems to be what the majority of people I've talked to want compatibility with) and the copyright notices I can find in there that are relevant to the compression are for "Greenleaf Software", and things like "Internal 1 error in Greenleaf Decompression routine".

According to their product page, "Greenleaf ArchiveLib" is PKZip-compatible. It does say it has three compression engines, though, so I'm guessing Lone Wolf just picked the wrong one and made it all incompatible. I'm just glad I know about open source and don't have to pay their prices!

Greenleaf give some hints about their compression (the product brochure calls it "Greenleaf proprietary lossless engine, dictionary based, 5-level"), so it might be workable by other people without the library if that is what Lone Wolf used.


[edit] Oh, and if it is important, the reference I posted earlier is from ConstrKit.rtf from v2 of the app. There doesn't appear to be any similarly useful/interesting/technical documentation for v3 that I've found so far.

IBBoard said:

According to their product page, "Greenleaf ArchiveLib" is PKZip-compatible. It does say it has three compression engines, though, so I'm guessing Lone Wolf just picked the wrong one and made it all incompatible.

Click to expand...

Actually, we picked the only compression engine that allowed us to properly identify the archive as being an AB import file with game system and version information embedded, as opposed to simply a generic Zip file that could contain anything. Sadly, the Zip format didn't allow that capability - at least, not back in 1998 - so we were forced to use the format that was designed competently for our needs instead of the Zip format. We didn't "pick the wrong one".

Please don't go off making rash assumptions without first checking on the FACTS. I'm happy to answer questions about this stuff. But your language and comments are insulting - and all they do is demonstrate that you haven't bothered to get the FACTS before making your assertions.

I'm sorry you took offence, perhaps I didn't word that quite right. I wasn't making rash assumptions, I was making a developer's assessment of the situation with the file formats based on the information I have gathered.

My point was that some documentation for 2.x said the compression was PKZip compatible [1]. The data in the 2.x and 3.x files does not, in fact, appear to be PKZip compatible [2]. The Greenleaf ArchiveLib is PKZip-compatible [3]. ArchiveLib also contains one other compressor that isn't compatible with PKZip [4]. If ArchiveLib is used and the data cannot be extracted using standard compression and ArchiveLib supports additional compression formats then the logical conclusion is that the data was compressed with the non-standard compression [5]. If the documents say that the compression is PKZip compatible and the data is compressed with a non-standard compression then the logical conclusion is that the wrong compression format was chosen (for a given value of 'wrong' where it means 'not the one documented and not the one that would make things easier for anyone, including Lone Wolf, to work with in the future') [6].

Relic made Dawn of War (and previous) SGA files as a custom archive file format. They also used Zlib compression for the individual components within that archive. That lets them do what they want in terms of making it recognisable as one of their files while also using a standard compression format. Just because you use Zlib/deflate compression doesn't mean you have to make the whole thing a standard .zip file and handle its short-comings, and just because you want to make a non-standard file doesn't mean you can't use standard compression formats. I'm just trying to understand the documentation vs implementation difference and the decisions that got us where we are now.



1) which is true and I quoted the section from the document. I'll happily supply a copy of the document as well if you want.
2) which is true for all of the testing that I, and seemingly others, have done
3) which is true, and I provided a link to their product page
4) which is true - they have three, one is Zlib/PKZip, one is their custom format and one is an "uncompressed" format.
5) which is true by deductive logic based on the previous three points being true
6) which is also true by deductive logic based on the first and fifth point being true

The problem with your logic is you assume 1 is true when it is in fact not [1] according to the post Rob made prior to you saying that Lone Wolf chose the wrong compression. Therefore you did make a rash assumption that was invalid based upon the facts as presented to you when you made that statement.

1) In post 4, Rob asks if the documenation says that and then says it (by pronoun assocation rules would be the documentation) will be corrected, thereby stating that if the documentation says it is PKZip compatible it is in error.

Part of constructing an argument based on logic is taking the facts as they stand at the moment and deducing further facts based on them. The documentation makes a statement of truth about the compression format used in the file. With no further information, all we can do is assume that it is true. If the documentation for something makes a statement and you are evaluating something in relation to that statement then assuming the statement is true* is not a rash assumption.

If I had assumed the compression would be PKZip compatible if the documentation just said "it uses compression" then that would be a rash assumption as there are many different compression algorithms that are not PKZip compatible. What you're effectively saying is that you should ignore all statements in the documentation as any one of them could be false.

Besides, points 2, 3 and 4, and as a consequence point 5, already point out exactly what you have just repeated - the documentation is incorrect and the format is some different compression that is not, in fact, PKZip compatible.

As I said in my follow-up post, everything is based on the facts as I had them and with respect to the statement in the documentation and the future usability of the files the decision that was made could be considered to be the "wrong" one. Using zlib within a similar structure to what I had now would a) have allowed Lone Wolf development the freedom to move to other (potentially more efficient) compression libraries if they wanted, b) matched the statement in the document (which is what users read rather than source code) and c) made the file format more usable for others (which could improve the up-take of Army Builder as people build community tools around it, as I've done for Dawn of War and their SGA various "Chunky File" file formats).



* especially when it is in the 2.2c documentation - so it isn't as if it was in 1.x documentation and accidentally slipped in to the first revision, it was there for a while

Wow...while I understand (at the root of it) what your saying you are talking about Army Builder 2.2c which IIRC hasn't been supported for at least 4 years.

Your also criticizing a company for using something that you wouldn't use.

At the root of it..it's a Data file. There doesn't have to be anything efficient about it..you download it and it imports into Army Builder.

Lone Wolf really doesn't have to nor should they make a usable format for others to build tools for since it's Lone Wolf's proprietary software (licensed compression or not).

I'm just trying to get an idea as to what your asking about now.

But then it's my thinking..

I'm not really asking anything at the moment, since Lone Wolf have answered my question sufficiently for me to find out the full information (i.e. to track down that it is Greenleaf's custom compression format that was used).

The documentation I had was for 2.2c because, as I said, the community members I'm working with are 2.2c users. Also, from what I was able to find the AB3 construction kit was either lacking or hiding its similarly technical documentation. The compression used seemingly hasn't changed between 2.2 and 3 anyway, so any information on the compression is equally relevant to new files. All of the additional information in the files is very basic stuff, at least for the important bits.

As for making it a usable format, I'm not saying that they should, just that it can certainly be advantageous to them in the long run. Lots of companies have a very closed mind about "it is ours and no-one else can have it", even over simple data files, but a more open mind-set of "it's ours, but we'll make it interoperable so that we can encourage a community of tools that could increase uptake" can have positive consequences that aren't normally factored in to the bottom line.

As for criticizing, as I said in my last post and one of the others, I was making a developer's evaluation. Given all of the information to hand (including information from the Greenleaf website and my knowledge of other archive file formats for computer games) a different choice could easily have been made that would have provided more flexibility. The developers have already admitted that they have considered alternate compression but that it isn't a high priority, which implies that even they think a better choice could have been made, or that compatibility in the compression may be a better way forward.

As I said, though, I've asked my question (what is the compression?), I've got my answer (custom modification of LZW), I've done my research (it is Greenleaf's custom format) and I now know what I need to know.

IBBoard said:

mpression?), I've got my answer (custom modification of LZW), I've done my research (it is Greenleaf's custom format) and I now know what I need to know.

Click to expand...

There's not much "custom" about it, it's pure ARJ in a slightly different container

I didn't get too far with pulling apart the compression, but I've got someone else who's looking at it for the project. Do you have anything more on it being similar to ARJ? Any evidence or examples?

Thanks.

I'll take that as a "no" then!