Thanks rikimbo!
I didn't noticed that the way flavor texts are now managed on the Gatherer has been very slightly modified. The area is called "flavortextbox" instead of "cardtextbox". I fixed this in 3.15a
Please double check to make sure the flipped cards are displaying correctly.
On Homura, Human Ascendant, I see a back ID, and it's the number of itself.
On every other flip card, I see nothing in terms of back ID or how to distinguish it as a flip card (Magiccards.info requires the letter a after the image number, in order to pull the image correctly).
@ danbopes : fixed! This minor bug was due to an update of the WotC Gatherer so that Gatherer Extractor got lost in the extraction of some specific flipped cards.
@ swallowtail23 : this is due to a modification of the management of rulings on the WotC Gatherer. I updated the way rulings are extracted. This bug should be fixed.
@ Freezerburn26 : This is due to a modification of the management of converted mana cost on the WotC Gatherer. I fixed the bug.
v3.16 release with fix.
Please note that WotC gatherer's html markers and stuff are continuously modified, and Gatherer Extractor needs to be modified accordingly. That's why sometimes some columns remain blank or contain html code. Your reports are very appreciated to fix those running bugs.
@ swallowtail23 : this is due to a modification of the management of rulings on the WotC Gatherer. I updated the way rulings are extracted. This bug should be fixed.
@ Freezerburn26 : This is due to a modification of the management of converted mana cost on the WotC Gatherer. I fixed the bug.
v3.16 release with fix.
Please note that WotC gatherer is html marker and stuff are continuously modified, and Gatherer Extractor needs to be modified accordingly. That's why sometimes some columns remain blank or contain html code. Your reports are very appreciated to fix those running bugs.
No problems at all, and thank you for a great little app. Rulings are fixed - thanks!
@ NightCabbage : The set codes are not random, but are the result of the historical legacy. People are still very familiar to former set codes, such as A for Alpha, R for Revised, IA for Ice Age, TE for Tempest etc. I didn't want to change those set codes for the new official ones in order to prevent people from being confused with. Nevertheless, since Kamigawa, I'm using the official 3-letter code because it was introduced at that period. The only exceptions are Conflux and Guilpact for compatiblity reasons (CON and GPT are reserved names, depending on the OS), as well as some promotional sets because the official set codes are not at all straightforward or only short-lived (DDx for the Duel Decks... what about after DDZ, that occurs in the next 3 years ?). But you can customize the set codes as you want in the code editor; so feel free to replace the default ones by the official ones. Everything depends on the compatibility you want with the resulting database.
Thanks for fixing the last bugs. Unfortunately there's another error that hangs up the application when it tries to process legalities. It fails at the Tiny Commanders one. It says invalid string. I think it was doing the double cards with // in them
Ok, this bug is probably do to the local culture you use.
It works correclty for a culture as fr-FR, de-DE, but fails for en-US.
May you please test release 3.16a ? It is supposed to be fixed now.
Just thought I'd let you know of a bug I found in the flavour text... it's very common, any card with a quote should have the problem.
Example below:
Anaba Bodyguard
(UGH this silly forum isn't letting my post anything with a less than, greater than, or even ampersand in it... pretend the divs have their correct symbols around them...)
"They who challenge a minotaur enjoy the taste of their own blood." (/div)(div class='flavortextbox')-Mirri of the Weatherlight
Unfortunately I have already extracted the data into 129 separate xml files... so I'll have to remove this in my code lol
Speaking of which - it would be awesome if you had the option to output each set in the database to a separate xml file, so I don't have to do them all individually
Dear NightCabbage,
thanks for your report. This minor bug has been fixed in 3.16b.
For the automated extraction in individual xml files, it will be done in release 3.17.
Best regards
On some flip cards (Kamigawa block), the artist field is not being populated.
For example, BOK, Cunning Bandit. The artist should be Paolo Parente.
This seems to only happen when both sides of the card are displayed separately on Gatherer. ie. it does not happen for Callow Jushi.
-EDIT-
Sorry, I'll try to be more helpful!
I believe you're checking for the following...
ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_ctl03_ArtistCredit
ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_ctl04_ArtistCredit
However it looks like it is (maybe just sometimes? I'm not sure...):
ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_ctl02_ArtistCredit
ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_ctl03_ArtistCredit
Also, as part of my xml cleanup, I'm removing any "A" characters from the Color field. Not entirely sure why they're there in the first place, as Artifact isn't a colour?
Maybe make this an option, like you have for +H for hybrid and +O for gold? (and probably have it disabled by default)
v3.17 released!
-- Added export sets in sql format, with a file per set
-- Fixed minor bug in computing legalities for Tiny Leaders Commander
-- Fixed minor bug in pricing for OS culture with period-separated decimals
-- Fixed minor bug in exraction of flavor texts due to modifications on WotC Gatherer
-- Fixed minor bug in exraction of artists for flipped cards
@NightCabbage : I know "Artifact" is not a color, but is is classically mentionned in the color field, as for land with "L". It is a legacy from the MWS database, and before colorless cards were massively developped (such as in ROE).
I can make them an option. Consequently, you'd get C for those items (for colorless) and +A/+L for Artifacts/Lands? Does it make sense for you?
As long as there are options to disable the additional non-color types it should be fine.
Personally all I want in the color field are: WUBRG and I'll clear anything else out manually if necessary
All of that information is present in other ways...
- If it's an Artifact, it will have the Type of Artifact
- If it's a Land, it will have the Type of Land
- If it's colorless, it will have an empty Color field
As a side-note, my program is coming along nicely
Entire xml database compresses down to around 2.5MB
Not bad for over 24,000 cards (130 sets).
I have 1 entry per card, with sub entries for each printing - most efficient data structure (no repeated information).
And I can download an entire set of images in under 10 seconds.
Random question - should I limit this to be slower? ie. will magiccards.info or gatherer.wizards.com try to block anyone who downloads too many images too quickly? (I notice GEx downloads images quite slowly - is this the reason for that?)
There seems to be a bug or two in the export sql. I downloaded all data from all sets and now i'm trying to load it and there's a bug on the first insert for Nsets
It seems that if your values end in a NULL, you're putting an extra comma at the end of that. Easy fix for me, I do a find an replace and remove the comma.
Now when i run it i get an error that the column count doesn't match the value count for this line.
('Odyssey','OD','OD','09/2001','False','0','1','3','11','0',NULL,NULL,'0','0',NULL,NULL,'True','0.158')
Which is totally weird because I count 18 values and 18 columns. So I may need to do some digging there. Maybe i'll just try the xml and see if i have better luck with that one.
again love your app!!
**edit**
did some more digging. even if i delete that line from the query it still generates the same error. i would bet that there's something weird with an apostrophe some where in the first hundred lines, but i can't see it. my eyes hurt from trying to see it. i tried to past the insert query into this message, but got an error about non-latin unicode characters not being allowed. so maybe that's something??? let me know if you want me to email it to you and i'll be happy to send you the file.
oh, i also tried exporting each of the sets as a separate sql file. but that caused an error. i didn't think to grab a copy of the dump for you. sorry.
@vrspectre : Thanks for your bug report.
I agree, there is an extra comma that raises an error. This is fixed and available in the next release.
Concerning the other bugs, I did some digging and I didn't find what's wrong. I was able to successfully load the sql database in my tests. May you please send me the dump in Private Message so that I can see what is the issue? Thanks in advance
Feedback
Just downloaded everything using Gatherer Extractor Installer v3.18_June-2nd-2015.exe
tried to export per set as sql, get a run time error about unable to write file
exported as a single file worked
but my SQL server from godaddy will only accept 1000 records at a time per update
it was tedious but I would run 900 records and submit then run the next 900 ad nauseaum.
I didn't see an option in the interface to break the generated command down to blocks of commands,
ie every ### rows end and start new command.
question: I cannot find a way to extract token information with Gatherer Extractor
I finally found a note about a filter being in places to block token information... why? or better yet, why is it not a toggle?
using Gatherer Extractor Installer v3.18_June-2nd-2015.exe
@globeadue : the small bug you reported (thanks for that) has been fixed in 3.18a. The error was raised because of some sets containing '"' in their name, such as Archenemy "Schemes". As you know, '"' is a forbidden character for filenames in Windows.
The tokens are filtered becuse they are partly missing in many sets. As long as all the tokens are not stored in the WotC gatherer database, they will not be downloadable by GEx for the sake of consistency.
Best regards,
I didn't noticed that the way flavor texts are now managed on the Gatherer has been very slightly modified. The area is called "flavortextbox" instead of "cardtextbox". I fixed this in 3.15a
On Homura, Human Ascendant, I see a back ID, and it's the number of itself.
On every other flip card, I see nothing in terms of back ID or how to distinguish it as a flip card (Magiccards.info requires the letter a after the image number, in order to pull the image correctly).
@ Freezerburn26 : This is due to a modification of the management of converted mana cost on the WotC Gatherer. I fixed the bug.
v3.16 release with fix.
Please note that WotC gatherer's html markers and stuff are continuously modified, and Gatherer Extractor needs to be modified accordingly. That's why sometimes some columns remain blank or contain html code. Your reports are very appreciated to fix those running bugs.
No problems at all, and thank you for a great little app. Rulings are fixed - thanks!
I'm just wondering why you are using such random set codes, instead of the official 3 character codes?
It's annoying that magiccards.info is also using (different) random set codes...
But I'm not sure why you are, too?
The real set codes are easy to find:
http://mtg.wikia.com/wiki/Set
http://en.wikipedia.org/wiki/List_of_Magic:_The_Gathering_sets
I've just gone through and modified my list (luckily you provided the facility to be able to edit set codes!)
(the only exception being Conflux, as CON is a reserved name for Windows, so most places use CFX instead for this)
Cheers
It works correclty for a culture as fr-FR, de-DE, but fails for en-US.
May you please test release 3.16a ? It is supposed to be fixed now.
Thanks for having reported the issue.
Just thought I'd let you know of a bug I found in the flavour text... it's very common, any card with a quote should have the problem.
Example below:
Anaba Bodyguard
(UGH this silly forum isn't letting my post anything with a less than, greater than, or even ampersand in it... pretend the divs have their correct symbols around them...)
"They who challenge a minotaur enjoy the taste of their own blood." (/div)(div class='flavortextbox')-Mirri of the Weatherlight
Unfortunately I have already extracted the data into 129 separate xml files... so I'll have to remove this in my code lol
Speaking of which - it would be awesome if you had the option to output each set in the database to a separate xml file, so I don't have to do them all individually
thanks for your report. This minor bug has been fixed in 3.16b.
For the automated extraction in individual xml files, it will be done in release 3.17.
Best regards
On some flip cards (Kamigawa block), the artist field is not being populated.
For example, BOK, Cunning Bandit. The artist should be Paolo Parente.
This seems to only happen when both sides of the card are displayed separately on Gatherer. ie. it does not happen for Callow Jushi.
-EDIT-
Sorry, I'll try to be more helpful!
I believe you're checking for the following...
ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_ctl03_ArtistCredit
ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_ctl04_ArtistCredit
However it looks like it is (maybe just sometimes? I'm not sure...):
ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_ctl02_ArtistCredit
ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_ctl03_ArtistCredit
(also happens for ISD/DKA flip cards like this http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=262675 )
Also, as part of my xml cleanup, I'm removing any "A" characters from the Color field. Not entirely sure why they're there in the first place, as Artifact isn't a colour?
Maybe make this an option, like you have for +H for hybrid and +O for gold? (and probably have it disabled by default)
-- Added export sets in sql format, with a file per set
-- Fixed minor bug in computing legalities for Tiny Leaders Commander
-- Fixed minor bug in pricing for OS culture with period-separated decimals
-- Fixed minor bug in exraction of flavor texts due to modifications on WotC Gatherer
-- Fixed minor bug in exraction of artists for flipped cards
@NightCabbage : I know "Artifact" is not a color, but is is classically mentionned in the color field, as for land with "L". It is a legacy from the MWS database, and before colorless cards were massively developped (such as in ROE).
I can make them an option. Consequently, you'd get C for those items (for colorless) and +A/+L for Artifacts/Lands? Does it make sense for you?
As long as there are options to disable the additional non-color types it should be fine.
Personally all I want in the color field are: WUBRG and I'll clear anything else out manually if necessary
All of that information is present in other ways...
- If it's an Artifact, it will have the Type of Artifact
- If it's a Land, it will have the Type of Land
- If it's colorless, it will have an empty Color field
As a side-note, my program is coming along nicely
Entire xml database compresses down to around 2.5MB
Not bad for over 24,000 cards (130 sets).
I have 1 entry per card, with sub entries for each printing - most efficient data structure (no repeated information).
And I can download an entire set of images in under 10 seconds.
Random question - should I limit this to be slower? ie. will magiccards.info or gatherer.wizards.com try to block anyone who downloads too many images too quickly? (I notice GEx downloads images quite slowly - is this the reason for that?)
There seems to be a bug or two in the export sql. I downloaded all data from all sets and now i'm trying to load it and there's a bug on the first insert for Nsets
('15th Anniversary','15A','15ANN','00/0000','True','0','0','0','0','0','0','0','0','0',NULL,NULL,'False',NULL,)
It seems that if your values end in a NULL, you're putting an extra comma at the end of that. Easy fix for me, I do a find an replace and remove the comma.
Now when i run it i get an error that the column count doesn't match the value count for this line.
('Odyssey','OD','OD','09/2001','False','0','1','3','11','0',NULL,NULL,'0','0',NULL,NULL,'True','0.158')
Which is totally weird because I count 18 values and 18 columns. So I may need to do some digging there. Maybe i'll just try the xml and see if i have better luck with that one.
again love your app!!
**edit**
did some more digging. even if i delete that line from the query it still generates the same error. i would bet that there's something weird with an apostrophe some where in the first hundred lines, but i can't see it. my eyes hurt from trying to see it. i tried to past the insert query into this message, but got an error about non-latin unicode characters not being allowed. so maybe that's something??? let me know if you want me to email it to you and i'll be happy to send you the file.
oh, i also tried exporting each of the sets as a separate sql file. but that caused an error. i didn't think to grab a copy of the dump for you. sorry.
I agree, there is an extra comma that raises an error. This is fixed and available in the next release.
Concerning the other bugs, I did some digging and I didn't find what's wrong. I was able to successfully load the sql database in my tests. May you please send me the dump in Private Message so that I can see what is the issue? Thanks in advance
Just downloaded everything using Gatherer Extractor Installer v3.18_June-2nd-2015.exe
tried to export per set as sql, get a run time error about unable to write file
exported as a single file worked
but my SQL server from godaddy will only accept 1000 records at a time per update
it was tedious but I would run 900 records and submit then run the next 900 ad nauseaum.
I didn't see an option in the interface to break the generated command down to blocks of commands,
ie every ### rows end and start new command.
I finally found a note about a filter being in places to block token information... why? or better yet, why is it not a toggle?
using Gatherer Extractor Installer v3.18_June-2nd-2015.exe
The tokens are filtered becuse they are partly missing in many sets. As long as all the tokens are not stored in the WotC gatherer database, they will not be downloadable by GEx for the sake of consistency.
Best regards,
I could be just going crazy, but is Conspiracy (CNS) missing from the set list?My bad, it's listed under "Magic: The Gathering—Conspiracy".