Personally, I don't mind the prefixes being four character. Introducing a hyphen or other delimiter might cause some issues where people are parsing them as pat of other strings, though - /blah(\w{3,4})/ is a bit easier than /blah((?:\w-)\d{3})/, especially if you've got thngs like UUIDs muddying the waters
Private Mod Note
():
Rollback Post to RevisionRollBack
Minouris's Library - Collection manager and deck builder. It's nifty - Check it out!
I think the lowercase 'p' (at the front) is a good idea, it shows clearly that it's not an official code. Also, it's nice to see that you're doing away with "Promo Set for Gatherer." That classification on Gatherer always annoyed me, and it made extracting the promo-only cards, like Sewers of Estark, a real pain without hard-coding stuff.
magiccards.info just got updated so they're still operational if you're going to use the site. There will just be some delay about getting tokens and emblems and such. Have you tried getting in contact with the owners? They don't give out their database but they might be willing to work with you.
A couple other ideas I had:
1. A card-face attribute for sets, which would have the values "original", "Eighth Edition", and "Magic 2015" or something like that.
2. Removing the text attribute from basic lands. They really don't have text, the symbol in their text box is basically just a design/ space filler. I know that's what gatherer has, but it doesn't make much sense.
Also, Vorthospike, that's pretty cool! I've done some analysis of my own using the data, but that's quite an unusual thing to analyze.
So after more than 3 full days of work, I have just updated the site with a bunch of changes:
Many cards have had their flavor text and artists corrected for various typos and errors found on Gatherer.
Double newlines are now just single newlines. This affects 'text', 'flavor' and 'originalText' card properties. I am sorry if this causes some existing software out there that consume MTGJSON to break. I always try and keep possible 'breaking' changes to a minimum.
Added a 'reserved' property for cards that are reserved according to Wizard's Official Reprint Policy
Created Window specific links for CON.json and AllSetFiles.zip. Basically Windows doesn't allow any filenames named CON, so CON.json could not be downloaded on windows and AllSetFiles.zip couldn't be extracted either.
Basic lands no longer have a text attribute. According to rule 305.6 basic lands have an intrinsic ability.
UGL, ARN: Basic lands now correctly have the rarity "Basic Land".
UNH: Super Secret Tech now has rarity of 'Rare' to match it's colored symbol.
More details on some of these changes can be found in the quoted replies below.
My next task is to add promos and tokens, which should take a bit of work. I'll be using the lowercase p prefix for the promo set codes. Thank you everyone who gave some feedback on this!
So I just noticed something weird randomly, and I'm not sure what it's about, but the flavor text on Bloodshot Trainee is missing the exclamation point after "Hrrngh". I remeber seeing a missing exclamation point in another card's flavor text too. This isn't your fault, though, that's how Gatherer has it also, so I'm not sure if there's anything you can really do about it if it's prevalent at all.
So this was a MASSIVE pain. Turns out Gatherer is missing exclamation marks in flavor text for HUNDREDS of cards across many sets. Fixing these errors is what took the bulk of the multi-day effort.
Nonbreaking space (0x00A0) in Sanguine Praetor's flavor text
Line separator (0x2028) in Battle Hurda's flavor text
Horizontal Bar (0x2015) in Beast Within's flavor text (Heroes vs. Monsters) and in Orcish Lumberjack's (Heros vs. Monsters and Vintage Masters.) The two versions of the Lumberjack actually have their flavor text formatted differently on Gatherer, the VM one has a linebreak (and MTG-JSON doesn't pick up the quote attribution there.)
1. In flavor texts that have quotations with attributions, such as Dissipation Field's '"Strike me once, shame on you. Strike me twice . . . Well, let's just see if you can." —Kara Vrist, Neurok agent', an em dash (U+2014) is used, but I think the convention is to use that horizontal bar character (U+2015) I mentioned earlier that strangely showed up on like 2 cards. (Related info on this)
2. Niv-Mizzet's flavor text is probably supposed to have a degree sign (U+00B0), but for some reason uses a "Masculine Ordinal Indicator" sign (U+00BA); these look identical in some renderings but not in others.
The em dash is used in the vast majority of cards, as you mention only 2 of them had the horizontal bar. I decided to just stick with em dash for now, since even in the useful link you gave (thanks for the link!) several answers mention em dash as acceptable.
A couple other ideas I had:
1. A card-face attribute for sets, which would have the values "original", "Eighth Edition", and "Magic 2015" or something like that.
2. Removing the text attribute from basic lands. They really don't have text, the symbol in their text box is basically just a design/ space filler. I know that's what gatherer has, but it doesn't make much sense.
I took care of #2. As for the card-face attribute (which Wizards calls the 'frame') I've added it to my to-do list in the future, but it's lower priority than some other tasks I have on there. Thanks for the suggestions!
Awesome! I think getting rid of the double newlines was a good move. That's crazy Gatherer has all those typos... is there any way for someone to contact them and let them know? I guess the good news is, unlike card text, flavor text is immutable, so you only need to extract it once.
I have a remark regarding the basic land intrinsic attribute:
Rule 305.6. says: 'A land with a basic land type has the intrinsic ability...' (not only a basic land). So i think the snow-covered lands should lose their GWRBU abilities too.
Strange that gatherer does not do it like this. For example Taiga has its text in brackets and has a rule explanation, that the mana abilities are intrinsic because of its basic land types
Good catch. Along with the upcoming promos update, I'll update it so all basic lands have no text (including snow covered, etc)
As for 'Taiga' and other dual lands, the 'Types' line on Gatherer doesn't have "Basic" in it (Snow covered lands do though). According to Rule 305.9: Any land with the supertype “basic” is a basic land. Any land that doesn’t have this supertype is a nonbasic land, even if it has a basic land type.
So according to this 'Taiga' isn't considered a basic land, and thus rule 305.6 doesn't apply to it (and other dual lands).
It's great you're adding promo's! If you're looking for card images or card lists for promo's, you could always compare deckbox.org (for example https://deckbox.org/editions/145-prerelease-events for prerelease cards) with magiccards.info, they do seem to have the (latest) additions magiccards.info. The images are not of great quality though. There is also a lot of information on magiccardmarket (for example https://www.magiccardmarket.eu/Products/Singles/Prerelease Promos for prerelease cards) but it's impossible to sort in a convenient way. Compliments for your awesome work and let us know if we can help (with providing HQ images for example).
Check out Magic Album by slightlymagic.net. It's run by the site admin over there. Since that's the source of all the high quality images, his pictures torrent includes everything that has been released by the team in an organized collection.
Is there any way that you can make it so that things like onlineOnly have an entry for every set? It would make arranging the data much easier if I could extract the value of onlineOnly rather than write a function to handle missing values.
Is there any way that you can make it so that things like onlineOnly have an entry for every set? It would make arranging the data much easier if I could extract the value of onlineOnly rather than write a function to handle missing values.
I could do this of course but there are lots of other optional fields that exist and I feel like if I broke convention for this field, that it opens door to having to break convention for other fields when a developer request, which in the case of card fields would quickly bloat up the JSON. Also the convention of not including the field if it doesn't need to be included would then have an exception and that doesn't make me tingle inside. I'm not trying to be difficult, I just feel it starts down a slippery slope.
As a programmer, I am curious what language and JSON parsing library you are using that requires a function in this use case. Most JSON libs I've seen for a few different languages have support for a 'default' value when the key isn't present.
Is there any way that you can make it so that things like onlineOnly have an entry for every set? It would make arranging the data much easier if I could extract the value of onlineOnly rather than write a function to handle missing values.
I could do this of course but there are lots of other optional fields that exist and I feel like if I broke convention for this field, that it opens door to having to break convention for other fields when a developer request, which in the case of card fields would quickly bloat up the JSON. Also the convention of not including the field if it doesn't need to be included would then have an exception and that doesn't make me tingle inside. I'm not trying to be difficult, I just feel it starts down a slippery slope.
As a programmer, I am curious what language and JSON parsing library you are using that requires a function in this use case. Most JSON libs I've seen for a few different languages have support for a 'default' value when the key isn't present.
I wasn't aware that the convention in JSON was to leave out implied information. That certainly helps keep the size of the file down.
In R there is a package (rjson) that reads JSON files and returns a (fairly literal) output of their contents. Every object and array becomes a list with the values stored as strings (for the sake of speed). Fields that aren't included are not listed. Parsing the results to becomes a dataframe (which is much easier to work with) requires writing my own program which recognizes that a missing value corresponds to a specific meaning (FALSE, 0, NA).
The way I interpret it, I don't think rule 305.6 is actually relevant here. It's not whether lands have the ability or not, it's just whether there's anything printed on the card. The dual lands have their mana abilities in reminder text, so they're rightly in the text field. But basic lands don't have a printed ability, and they don't have reminder text. They just have a special design in their text box. Gatherer puts the color code of the symbol in its text field for some reason, and it probably shouldn't be there.
Also, I didn't know about the onlineOnly attribute, that's pretty cool. Have you considered doing "paperOnly" or something similar?
As a programmer, I am curious what language and JSON parsing library you are using that requires a function in this use case. Most JSON libs I've seen for a few different languages have support for a 'default' value when the key isn't present.
Can't speak for Vorthospike, but I have this problem in PHP with the json_decode() function - it just translates what's there for each record, without trying to infer any structure from the surrounding records. I usually have to check using property_exists() before accessing a properrty.
Private Mod Note
():
Rollback Post to RevisionRollBack
Minouris's Library - Collection manager and deck builder. It's nifty - Check it out!
Just found a small issue with flavor text: excluding un-sets, in 482 total versions among 151 differently-named cards, there's an issue with newlines after quote attributions. Sometimes the newline shows up (correctly), sometimes it doesn't. For example, on Windstorm, the M10 version has: (and I replaced the em-dashes with ASCII hyphens)
"We don't like interlopers and thieves cluttering our skies."
-Dionus, elvish archdruid
And the M14 version has:
"We don't like interlopers and thieves cluttering our skies."-Dionus, elvish archdruid
And again, this is what Gatherer does. The two cards look exactly the same, and the newline is obviously supposed to be there. Just another Gatherer typo...
Some cards that only have one printing (such as Zephyr Charge) have this issue, but in most cases it's on cards that have multiple printings, and the later versions have it wrong.
Just did some more research, and found that this error occurs only in the following sets: [Portal, Urza's Saga, Urza's Destiny, Magic 2014 Core Set x 71, Duel Decks: Heroes vs. Monsters x 14, Commander 2013 Edition x 66] (This is a Multiset, the numbers are the number of times the issue shows up; no number = only once.)
Just found a small issue with flavor text: excluding un-sets, in 482 total versions among 151 differently-named cards, there's an issue with newlines after quote attributions. Sometimes the newline shows up (correctly), sometimes it doesn't. For example, on Windstorm, the M10 version has: (and I replaced the em-dashes with ASCII hyphens)
"We don't like interlopers and thieves cluttering our skies."
-Dionus, elvish archdruid
And the M14 version has:
"We don't like interlopers and thieves cluttering our skies."-Dionus, elvish archdruid
And again, this is what Gatherer does. The two cards look exactly the same, and the newline is obviously supposed to be there. Just another Gatherer typo...
Some cards that only have one printing (such as Zephyr Charge) have this issue, but in most cases it's on cards that have multiple printings, and the later versions have it wrong.
Just did some more research, and found that this error occurs only in the following sets: [Portal, Urza's Saga, Urza's Destiny, Magic 2014 Core Set x 71, Duel Decks: Heroes vs. Monsters x 14, Commander 2013 Edition x 66] (This is a Multiset, the numbers are the number of times the issue shows up; no number = only once.)
I was thinking of standardizing the MTGJSON flavor text so there is always a single newline before the em-dash before the quote attribution. Early cards actually had just spaces before the em-dash (Jade Statue in LEA at http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=24) but I don't see any harm in changing this to the way all recent MTG cards are printed.
I would probably say only the new versions should be fixed. I see why it might make sense to "errata" the flavor text on older cards to keep them in line, but I probably wouldn't do it, I don't know.
Anyway, I checked out out the three cards I found from the three older sets; they are Wind Drake from Portal, Serra Zealot from Urza's Saga, and the last one was picked up by accident, it was Goblin Berserker, where there is actually a just a quoted word followed by an em-dash. As for the other two: if you look at the card, Wind Drake definitely doesn't have a space, but with Serra Zealot it's hard to tell because the quote goes to the end of the line.
However, I think all the flavor texts from M14, Heroes vs Monsters, and Commander 2013 probably should be changed to have a line break, since it's obviously how it appears on the cards, and with almost all of them, there is an older version with the same flavor text that does have the line break.
I would probably say only the new versions should be fixed. I see why it might make sense to "errata" the flavor text on older cards to keep them in line, but I probably wouldn't do it, I don't know.
Anyway, I checked out out the three cards I found from the three older sets; they are Wind Drake from Portal, Serra Zealot from Urza's Saga, and the last one was picked up by accident, it was Goblin Berserker, where there is actually a just a quoted word followed by an em-dash. As for the other two: if you look at the card, Wind Drake definitely doesn't have a space, but with Serra Zealot it's hard to tell because the quote goes to the end of the line.
However, I think all the flavor texts from M14, Heroes vs Monsters, and Commander 2013 probably should be changed to have a line break, since it's obviously how it appears on the cards, and with almost all of them, there is an older version with the same flavor text that does have the line break.
I was on the fence, so I'm cool leaving the non-newline ones that were printed that way alone. I'll work on getting the sets you mentioned with gatherer errors you mentioned fixed. Thanks for the help!!
Ok, after several days of work, a huge update! I just added 27 new promo sets! Each set has the set type "promo" and a 4 letter set code beginning with a lowercase 'p'.
With these new promo sets, I have removed HHO and PPR. HHO is now pHHO and PPR is just gone because it's cards are spread across several of the new promo sets.
Also added set: From the Vault: Annihilation (2014) (V14)
Lastly, I fixed some bugs with rulings having extra spaces in them, foreign names starting with XX, newlines not appearing correctly before the em-dash in the flavor text and snow covered basic lands now correctly have no card text.
1. The Naughty & Nice promo card uses colorless mana symbols without curly brackets (i.e., "1{B}{B}" and "1{W}{W}").
2. What do you think about a "designer" attribute on the M15 cards, rather than having the designer credit in the flavor text?
3. Would it be possible to have a way to tell which expansions are available online? There's an "onlineOnly" field, but could there be a "physicalOnly" field, or just do "isOnline" and "isPhysical", or something like that? (I'm actually not too familiar with Magic Online, are the available cards just each block set and core set since Mirage, plus the Masters sets? If so this might be unnecessary.)
Private Mod Note
():
Rollback Post to RevisionRollBack
To post a comment, please login or register a new account.
Minouris's Library - Collection manager and deck builder. It's nifty - Check it out!
Ahh, good idea on the lowercase p! I like that idea. Now I just need to decide which side the "p" should go
Minouris's Library - Collection manager and deck builder. It's nifty - Check it out!
http://rednaxela1618.wordpress.com/2014/08/12/weird-facts-from-mtg-json-art/
1. A card-face attribute for sets, which would have the values "original", "Eighth Edition", and "Magic 2015" or something like that.
2. Removing the text attribute from basic lands. They really don't have text, the symbol in their text box is basically just a design/ space filler. I know that's what gatherer has, but it doesn't make much sense.
Also, Vorthospike, that's pretty cool! I've done some analysis of my own using the data, but that's quite an unusual thing to analyze.
More details on some of these changes can be found in the quoted replies below.
My next task is to add promos and tokens, which should take a bit of work. I'll be using the lowercase p prefix for the promo set codes. Thank you everyone who gave some feedback on this!
So this was a MASSIVE pain. Turns out Gatherer is missing exclamation marks in flavor text for HUNDREDS of cards across many sets. Fixing these errors is what took the bulk of the multi-day effort.
Fixed!
The em dash is used in the vast majority of cards, as you mention only 2 of them had the horizontal bar. I decided to just stick with em dash for now, since even in the useful link you gave (thanks for the link!) several answers mention em dash as acceptable.
I took care of #2. As for the card-face attribute (which Wizards calls the 'frame') I've added it to my to-do list in the future, but it's lower priority than some other tasks I have on there. Thanks for the suggestions!
Good catch. Along with the upcoming promos update, I'll update it so all basic lands have no text (including snow covered, etc)
As for 'Taiga' and other dual lands, the 'Types' line on Gatherer doesn't have "Basic" in it (Snow covered lands do though). According to Rule 305.9: Any land with the supertype “basic” is a basic land. Any land that doesn’t have this supertype is a nonbasic land, even if it has a basic land type.
So according to this 'Taiga' isn't considered a basic land, and thus rule 305.6 doesn't apply to it (and other dual lands).
Thanks for the links! I'm finding this site also very useful: http://www.magiclibrarities.net/rarities.html
I could do this of course but there are lots of other optional fields that exist and I feel like if I broke convention for this field, that it opens door to having to break convention for other fields when a developer request, which in the case of card fields would quickly bloat up the JSON. Also the convention of not including the field if it doesn't need to be included would then have an exception and that doesn't make me tingle inside. I'm not trying to be difficult, I just feel it starts down a slippery slope.
As a programmer, I am curious what language and JSON parsing library you are using that requires a function in this use case. Most JSON libs I've seen for a few different languages have support for a 'default' value when the key isn't present.
I wasn't aware that the convention in JSON was to leave out implied information. That certainly helps keep the size of the file down.
In R there is a package (rjson) that reads JSON files and returns a (fairly literal) output of their contents. Every object and array becomes a list with the values stored as strings (for the sake of speed). Fields that aren't included are not listed. Parsing the results to becomes a dataframe (which is much easier to work with) requires writing my own program which recognizes that a missing value corresponds to a specific meaning (FALSE, 0, NA).
Also, I didn't know about the onlineOnly attribute, that's pretty cool. Have you considered doing "paperOnly" or something similar?
Can't speak for Vorthospike, but I have this problem in PHP with the json_decode() function - it just translates what's there for each record, without trying to infer any structure from the surrounding records. I usually have to check using property_exists() before accessing a properrty.
Minouris's Library - Collection manager and deck builder. It's nifty - Check it out!
"We don't like interlopers and thieves cluttering our skies."
-Dionus, elvish archdruid
And the M14 version has:
"We don't like interlopers and thieves cluttering our skies."-Dionus, elvish archdruid
And again, this is what Gatherer does. The two cards look exactly the same, and the newline is obviously supposed to be there. Just another Gatherer typo...
Some cards that only have one printing (such as Zephyr Charge) have this issue, but in most cases it's on cards that have multiple printings, and the later versions have it wrong.
Just did some more research, and found that this error occurs only in the following sets: [Portal, Urza's Saga, Urza's Destiny, Magic 2014 Core Set x 71, Duel Decks: Heroes vs. Monsters x 14, Commander 2013 Edition x 66] (This is a Multiset, the numbers are the number of times the issue shows up; no number = only once.)
I was thinking of standardizing the MTGJSON flavor text so there is always a single newline before the em-dash before the quote attribution. Early cards actually had just spaces before the em-dash (Jade Statue in LEA at http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=24) but I don't see any harm in changing this to the way all recent MTG cards are printed.
Thoughts?
Anyway, I checked out out the three cards I found from the three older sets; they are Wind Drake from Portal, Serra Zealot from Urza's Saga, and the last one was picked up by accident, it was Goblin Berserker, where there is actually a just a quoted word followed by an em-dash. As for the other two: if you look at the card, Wind Drake definitely doesn't have a space, but with Serra Zealot it's hard to tell because the quote goes to the end of the line.
However, I think all the flavor texts from M14, Heroes vs Monsters, and Commander 2013 probably should be changed to have a line break, since it's obviously how it appears on the cards, and with almost all of them, there is an older version with the same flavor text that does have the line break.
I was on the fence, so I'm cool leaving the non-newline ones that were printed that way alone. I'll work on getting the sets you mentioned with gatherer errors you mentioned fixed. Thanks for the help!!
With these new promo sets, I have removed HHO and PPR. HHO is now pHHO and PPR is just gone because it's cards are spread across several of the new promo sets.
Also added set: From the Vault: Annihilation (2014) (V14)
Lastly, I fixed some bugs with rulings having extra spaces in them, foreign names starting with XX, newlines not appearing correctly before the em-dash in the flavor text and snow covered basic lands now correctly have no card text.
1. The Naughty & Nice promo card uses colorless mana symbols without curly brackets (i.e., "1{B}{B}" and "1{W}{W}").
2. What do you think about a "designer" attribute on the M15 cards, rather than having the designer credit in the flavor text?
3. Would it be possible to have a way to tell which expansions are available online? There's an "onlineOnly" field, but could there be a "physicalOnly" field, or just do "isOnline" and "isPhysical", or something like that? (I'm actually not too familiar with Magic Online, are the available cards just each block set and core set since Mirage, plus the Masters sets? If so this might be unnecessary.)