MTG Salvation

Quote from brownballa55

I think it's broken because of from the vault: realms

Nope, you just need to edit the code list. You have a warning informing you that some codes are missing. You choose the code you want for FtV: Realms and you choose V12 for Magiccards.info code. It works perfectly.
I updated the extractor.

I'm pretty new to the game, my 14 year old son got me into it. I started coding a php/Postgres database last week, and I have been looking for something like this to download raw and complete data. I've scoured many pages, and as a developer, I appreciate the effort you have put into this program. Excellent work, especially the many useful export formats.

Can you describe your problem a little more thoroughly please? Do you see Return to Ravnica in the left sidebar? Does it have its set code? What happens if you check it off and then click "Actions" > "Download Main Data"?

If the Return to Ravnica is highlighted in red, it means some information are missing about that set, ie you need to set its code and its magiccards.info set code. After that, you can download the set properly.

FYI, you can download the last version of GEx. I updated the table of set codes, so that you'll get directly all the set codes, dates etc.

Foreign names are not available from magiccards.info for the moment, even if there are not present on the Gatherer.

Set code is RTR for magiccards.info as well, so you did nothing wrong. Just wait for the magiccards.info update.

It has been updated for 3 days.

First, I love this program. Second, I don't know if it was mentioned already or not, but if you use PHPMyAdmin to import the .sql file then you must first edit it (I used notepad++) or the foreign names will attempt to enter as plain text and not as characters (for non-Roman lettering names). Basically I added "character set utf8 collate utf8_unicode_ci". I changed these lines. (Lines 31-41 in my file).

Nback_id text character set utf8 collate utf8_unicode_ci,
Nname_CN text character set utf8 collate utf8_unicode_ci,
Nname_TW text character set utf8 collate utf8_unicode_ci,
Nname_FR text character set utf8 collate utf8_unicode_ci,
Nname_DE text character set utf8 collate utf8_unicode_ci,
Nname_IT text character set utf8 collate utf8_unicode_ci,
Nname_JP text character set utf8 collate utf8_unicode_ci,
Nname_PT text character set utf8 collate utf8_unicode_ci,
Nname_RU text character set utf8 collate utf8_unicode_ci,
Nname_ES text character set utf8 collate utf8_unicode_ci,
Nname_KO text character set utf8 collate utf8_unicode_ci,

Without this the foreign text shows as a bunch of question marks. (specifically the Asian card names). I think I changed more than was necessary, but wanted to point this out in case it helps anyone else.

I can't download Champions of Kamigawa for some reason. When I go download all + scans it gets all the spoiler text and settings for each language, then when it gets to downloading the data for every card it does 2 of them stopping at Akki Coalflinger and says done. Tried like 3-4 times and it consistently doesn't go past the second card.

Hi !

Thanks for the report.
Unformtunately, I can't fix it right now, because I noticed a more severe problem on DownloadFileAsync function, that downoad incomplete files.... (for Return to Ravnica for example : no foreign name, delayed downloads, no pricing. Downloading card numbers seems to block after 59 Ko...) This is particularly ennoying... I dont understand why it works fine for Dark Ascension, and bugs for Return to Ravnica....

Do you guys experience the same behavior ?

Edit: To illustrate a bit more, this code used to download all the list, now it is blocked after 59 Ko :

Dim WebClient As New System.Net.WebClient
WebClient.Encoding = System.Text.Encoding.UTF8

Dim pageURL As String = "http://magiccards.info/query?q=++e:rtr%2Fen&v=list&s=cname"
Dim uri As Uri = New Uri(pageURL)

WebClient.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20100101 Firefox/13.0.1")
WebClient.DownloadFileAsync(uri, "D:/LogsGatherer.txt")

WebClient.Dispose()
WebClient = Nothing

But the following one works:

Dim WebClient As New System.Net.WebClient
WebClient.Encoding = System.Text.Encoding.UTF8

Dim pageURL As String = "http://magiccards.info/query?q=++e:dka%2Fen&v=list&s=cname"
Dim uri As Uri = New Uri(pageURL)

WebClient.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20100101 Firefox/13.0.1")
WebClient.DownloadFileAsync(uri, "D:/LogsGatherer.txt")

WebClient.Dispose()
WebClient = Nothing

Any help is very welcome! As long as I can't fix this bug, Gatherer Extractor will not be available for download because providing something identified as buggy just does not make sense. I am sure you all understand...

You're probably going to run into situations where the problem cannot be fixed because it lies inside a control that you can't change. That's the trouble with doing serious work in VB really.

Well,
After a comprehensive debugging process, the problem comes from large pages of magiccards.info. The routine exposed before used to work perfecty, but not anymore. The problem seems to come from a connexion closing for pages >70 ko from magiccards.info. Indeed an error is raised when downloading async so that the event DownloadEnd is raised. The resulting webpage is incomplete and GEx is hence bugging.

I tried to use successfully a webbrowser control instead of the webclient (I noticed magiccards.info pages that could no longer be downloaded from the progam are still correctly downloaded by my web browser, so the problem was not a local unavailability of magiccards.info services). I can release a new version with that temporary solution, but I dont like it because it is not as efficient as an async download method from the webclient class. There are also no events to get info while downloading (% downloaded, file size etc.).

I would definitely like to know why the former code is not working: it comes either from the vb control itself as ark42 suggests or it comes from magiccards.info...

Honestly, I gave up on VB long ago because of things like this. Can you check the WebClient source to see what it is doing? Probably not. With C++/MFC, Microsoft at least gives you the source to controls so you can modify them if there are actually bugs in them. I know with downloading using Async and raw API calls from C++, you have to do a lot of work to make sure large files work right. Calling InternetReadFile() in a loop to fill up a small buffer will sometimes work, and othertimes fail because of IO blockage. Then you have to check for ERROR_IO_PENDING and block on a semaphore, that you release from the Async callback. The main thread will then have the buffer filled in at that point, even though InternetReadFile() returned a failure status code and didn't originally fill the buffer in. If the callback or IO blocking isn't done right, then exactly what you describe may happen - the download stops abruptly in the middle. Different users would see the error to different degrees too. If you have a super-fast internet connection, InternetReadFile() may never block for you, and you get the whole file. Slower connections will block more and fail sooner or more often.

Quote from Balthamos

I would love that, I was just starting to work on a project when this happened.

Great software, by the way.

I second that! Looking forward to a patch

I will release v3.6 probably this week-end. The patch is almost ready. Very little things remain to be fixed. But I am unavailable for GEx coding until Friday because of my job.

v3.6 has been released !

I hope the bug is correctly fixed now. I didn't have that much time to test. So your testing and report are very welcome!
Some new little things some of you required have also been added (predefined selection, delete sets etc.).

Enjoy !

You have to generate a patch for MWS, then import patch MWS. Be careful with Masterbase corrupting due to intense patching.

Thanks Balthamos,

Fixed! > v3.6a

Thanks for sharing this information... Now I am able to understand about magic fundamentals. I hope you would post more details on it.

Hi there,

I'm trying to get a database of every card to enter into my store's POS system and came across your program. However when I try to run it to get every card in each set, it only does three or four sets. I am clicking the "download all" button.

Am I doing something wrong? Is there's a procedure to download a complete database with card images?

NomadGameCenter, You need to press the button beside Download all data. It is "Download all data + Scans". Also, if you want the price then it is a checkbox in the main settings page.

Thanks - I tired that and it only collects and processes the first few sets.

It stops at the Coldsnap themed decks.

I was curious if it would be possible to add in Watermark as a field on on the card data. Gatherer has all this information available, for example http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=366290

Quote from DarthObiwan

I was curious if it would be possible to add in Watermark as a field on on the card data. Gatherer has all this information available, for example http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=366290

For sure it would !!! Added to my "to do" list for next update

It stops at the Coldsnap themed decks.

First make sure you don't have connection failure at home (Currently I have some that stop the download process for Time Out reason)
With the lastest version, I can't reproduce the bug. Are you using 3.6b ?

@chaudakh Awesome