Feature Request: Make ddidownloader.exe faster using multithreading?

[Rasmon]

Hi all,

I ran ddidownloader.exe, and while it worked it was very slow (about 8 hours to complete). I have a fast Internet connection with lots of bandwidth (30 Mbps+) and my computer is a Core i7, 12 GB RAM, and a SSD.

I searched the forums and found a few other threads about this, where someone explained this tool has to send many requests for small amounts of data and that the connect/get/disconnect for so many requests is the problem.

Couldn't Lone Wolf Development make ddidownloader.exe multithreaded to speed things up? I know I can connect to DDI on multiple computers simultaneously, and even different browsers on the same computer, so why couldn't ddidownloader.exe open multiple connections on different threads to speed things up? Even going from a single thread to 3 - 5 threads seems like it would make a huge difference on the download time and 8 or more threads could likely reduce the time to under an hour.

Any chance someone at Lone Wolf Development could consider this and comment on this thread?

Thanks!

[Colen]

It's definitely possible to do this, but there's a reason we don't - we're being good citizens. The downloader intentionally hits the D&D Compendium web site no faster than a user who's quickly paging through content on the site.

This means that if a bunch of people are using our downloader at once, the load on the site isn't much higher than it is normally, because it's spread out over time. If the downloader grabbed stuff as fast as possible, using multiple requests at the same time, that'd create a much higher load than if a lot of people used it at once, which might be problematic for the servers running the Compendium.

Other 4e-related tools similar to the downloader have been blocked in the past, so we're assuming that the reason the downloader still works is because we try our best to make things easy on the servers by accessing things at a reasonable speed. Unfortunately that means things take a long time to download, but that seems to be the trade-off for being able to use the downloader at all.

Hope this helps!

[Rasmon]

Hi Colen,

It surely helps my understanding and I really appreciate the explanation and fast response! Thank you!

Slowly getting the data is better than not getting it at all! Too bad Wizards doesn't offer any type of partnership or accommodation for developers like Lone Wolf to make this type of thing easier/faster.

Anyone know if I can copy the 4e folder from one computer to another so I only have to download it once? I have HL on my desktop and laptop (Windows, though may get a Mac some day.)

Also, does anyone know how often I should update the DDI for HL?

Thanks again!

[cryptoknight]

You can copy the data files from one computer to another.
Should be Tools->Explore Folders->Game System Data Folder

just copy that to any other computer that has HL licensed and you're golden.

WotC tends to publish updates about once a month.... though generally I update every couple because I generally forget to do it more often.

[Rasmon]

Thanks cryptoknight!