File compression?

IBBoard · May 8th, 2009, 06:56 AM

I asked this in an email nearly a month ago now and I got the automated response but never saw a human's reply. Time to try my chances on the Lone Wolf forum

I've been looking at the .ab file format at a low level and it appears to be a quite simple archive format. The main thing I don't know but am interested in at the moment is what the compression format is for the files. Relic's SGA files in Dawn of War just use deflated/zlibed, but the .ab files don't seem to contain be deflated/gzipped/bzipped (or at least not that I've been able to work with so far).

Can anyone enlighten me on the compression algorithm used?

Thanks,

IBBoard

rob · May 12th, 2009, 08:04 PM

For AB, we use a commercial compression library. It uses an adapted version of the LZW algorithm. The import files themselves are files produced by this compression library, and since they are non-standard, they can't be accessed directly via other tools.

This is something we've had on the todo list to revise for many years, but it all works, so it's never actually been addressed.

IBBoard · May 13th, 2009, 11:41 PM

Ah, so when the documentation says about being compatible with PKZip, it doesn't actually mean it is compatible with PKZip? The many joys of proprietary, closed-source formats.

Was there a particular reason that a commercial and modified version was used rather than an open and standards-based version?

rob · May 15th, 2009, 03:37 PM

The documentation says it's compatible? Please point me to that comment so that it can be corrected.

We began using the current compression library in 1998. That was 11 years ago. Back then, the LZW algorithm was still under patent protection, so the *only* option was a commercial product with appropriate licensing. Since then, everything has been working smoothly without incident, so there has been no reason to change things.

Like I said, it's something that's been on the todo list for a long time. But I don't usually spend time changing something that is working smoothly when I'm focused on developing new stuff.

IBBoard · 02:36 AM

I'm not on the right machine to check for v3 at the moment, but someone on my forums posted a quote from the v2 construction kit documentation that says:

Quote:

The ABExport tool is used to package up one or more Army Builder files into a single import file, much like PKZip would accomplish. The created file is compressed using a compression algorithm that is compatible with the one used in PKZip.

From what I remember of the surrounding text it basically hints that a .ab file is like a .zip, except that .zip wasn't used because it was felt that people would try to extract the content manually. I know that's not strictly true and is more of an analogy, but it does mean that the compression algorithm is at least semi-standardised.

As for the original decision, does that mean that rather than license a library that implemented the patented LZW compression you instead licensed a based-on-but-not-quite-compatible-with-LZW library that must also have been under patent? (if it implemented LZW it should surely be covered by the same patent).

Also, from my investigations in to the DLL I'm guessing that the library is compiled in to the EXE. I've found references to ZLib and Deflate within the EXE (hence why I thought it might be a standard compression), so I'm either using libraries that are stricter on the standards, or those comments in the EXE are lying. I don't suppose you can say anything about the compression used can you, given that it is proprietary?

MilitaryCoo · May 17th, 2009, 03:42 AM

I've been looking at this as well, and the best guess I've had at the .ab format is that it uses TTComp, which was a tool/library developed by SWFTE.

As I say, it's a best guess and it could be completely wrong. I've had no success trying the tools I've managed to get hold of that claim to be compatible because they don't run in a 64-bit environment and I haven't had the time to try it on a 32-bit machine/OS.

IBBoard · 07:27 AM

I just had a look at the old ABExport.exe from AB2 (since that seems to be what the majority of people I've talked to want compatibility with) and the copyright notices I can find in there that are relevant to the compression are for "Greenleaf Software", and things like "Internal 1 error in Greenleaf Decompression routine".

According to their product page, "Greenleaf ArchiveLib" is PKZip-compatible. It does say it has three compression engines, though, so I'm guessing Lone Wolf just picked the wrong one and made it all incompatible. I'm just glad I know about open source and don't have to pay their prices!

Greenleaf give some hints about their compression (the product brochure calls it "Greenleaf proprietary lossless engine, dictionary based, 5-level"), so it might be workable by other people without the library if that is what Lone Wolf used.

[edit] Oh, and if it is important, the reference I posted earlier is from ConstrKit.rtf from v2 of the app. There doesn't appear to be any similarly useful/interesting/technical documentation for v3 that I've found so far.

rob · May 26th, 2009, 05:56 PM

Quote:

Originally Posted by IBBoard

According to their product page, "Greenleaf ArchiveLib" is PKZip-compatible. It does say it has three compression engines, though, so I'm guessing Lone Wolf just picked the wrong one and made it all incompatible.

Actually, we picked the only compression engine that allowed us to properly identify the archive as being an AB import file with game system and version information embedded, as opposed to simply a generic Zip file that could contain anything. Sadly, the Zip format didn't allow that capability - at least, not back in 1998 - so we were forced to use the format that was designed competently for our needs instead of the Zip format. We didn't "pick the wrong one".

Please don't go off making rash assumptions without first checking on the FACTS. I'm happy to answer questions about this stuff. But your language and comments are insulting - and all they do is demonstrate that you haven't bothered to get the FACTS before making your assertions.

IBBoard · May 27th, 2009, 10:50 AM

I'm sorry you took offence, perhaps I didn't word that quite right. I wasn't making rash assumptions, I was making a developer's assessment of the situation with the file formats based on the information I have gathered.

My point was that some documentation for 2.x said the compression was PKZip compatible [1]. The data in the 2.x and 3.x files does not, in fact, appear to be PKZip compatible [2]. The Greenleaf ArchiveLib is PKZip-compatible [3]. ArchiveLib also contains one other compressor that isn't compatible with PKZip [4]. If ArchiveLib is used and the data cannot be extracted using standard compression and ArchiveLib supports additional compression formats then the logical conclusion is that the data was compressed with the non-standard compression [5]. If the documents say that the compression is PKZip compatible and the data is compressed with a non-standard compression then the logical conclusion is that the wrong compression format was chosen (for a given value of 'wrong' where it means 'not the one documented and not the one that would make things easier for anyone, including Lone Wolf, to work with in the future') [6].

Relic made Dawn of War (and previous) SGA files as a custom archive file format. They also used Zlib compression for the individual components within that archive. That lets them do what they want in terms of making it recognisable as one of their files while also using a standard compression format. Just because you use Zlib/deflate compression doesn't mean you have to make the whole thing a standard .zip file and handle its short-comings, and just because you want to make a non-standard file doesn't mean you can't use standard compression formats. I'm just trying to understand the documentation vs implementation difference and the decisions that got us where we are now.

1) which is true and I quoted the section from the document. I'll happily supply a copy of the document as well if you want.
2) which is true for all of the testing that I, and seemingly others, have done
3) which is true, and I provided a link to their product page
4) which is true - they have three, one is Zlib/PKZip, one is their custom format and one is an "uncompressed" format.
5) which is true by deductive logic based on the previous three points being true
6) which is also true by deductive logic based on the first and fifth point being true

wookieegunner · May 28th, 2009, 06:34 PM

The problem with your logic is you assume 1 is true when it is in fact not [1] according to the post Rob made prior to you saying that Lone Wolf chose the wrong compression. Therefore you did make a rash assumption that was invalid based upon the facts as presented to you when you made that statement.

1) In post 4, Rob asks if the documenation says that and then says it (by pronoun assocation rules would be the documentation) will be corrected, thereby stating that if the documentation says it is PKZip compatible it is in error.

rob Senior Member Lone Wolf Staff Join Date: May 2005 Posts: 8,232 May 12th, 2009, 08:04 PM	For AB, we use a commercial compression library. It uses an adapted version of the LZW algorithm. The import files themselves are files produced by this compression library, and since they are non-standard, they can't be accessed directly via other tools. This is something we've had on the todo list to revise for many years, but it all works, so it's never actually been addressed.
#2

IBBoard Junior Member Join Date: May 2009 Posts: 9 May 13th, 2009, 11:41 PM	Ah, so when the documentation says about being compatible with PKZip, it doesn't actually mean it is compatible with PKZip? The many joys of proprietary, closed-source formats. Was there a particular reason that a commercial and modified version was used rather than an open and standards-based version?
#3