As for art generation: you could always train the network on a greater library of art using more or less pretagged "databases" and imageboards. Devianart, Furaffinity, E926 (rather well tagged that one) you could use to get a ton of Dragons, unicorns, Furries (leonids, nacatls, Rhox, Nezumi, Kitsune), animals etc. tagging the entire MTG art catalogue by hand would be ... hard unless you get a dozen or so people that do that.
I guess one could use some object identifying Software to find and seperate out (so say just the sword or just a shield from a picture with a knight) objects (like the Google service that that Identified Talcos and his Ferret) from existing cards and pictures as well giving the Art NN further things to learn from which might be good for art composition.
Good points!
By the way...
Mediocre news: I don't have too much to show yet with the art generation.
Good news: I've definitively established how to do it right!
So, I was experimenting with the Eyescream package (thanks to researchers at Facebook). I was inspired by their ability to generate images of virtually anything given the right data, and that they state it's possible to map word-embeddings (like my content vectors) to images (though they haven't implemented that part yet). I'm missing some packages needed to do the big version of their network, but I can do the small one. I got about one epoch of training out of it and then ran into some weird bug. I tried training the network on creatures, but since I wasn't able to complete the training, all I can get out of it are vague, ethereal blobs (like the image I attached, I took a very low resolution sample with a custom script). I'm fairly confident that I was able to teach the network that creature artworks have figures in the foreground and stuff in the background, but that's all I was able to get. I'm sure if I trained more I could get things looking better, but I'll have to figure out why I got that error.
EDIT: A few interesting finds. I've combined them with generated flavor text as well (I sometimes primed the flavor text with a relevant word from the card, just to see what I would get).
Sandstorm G
Exile target creature card from your graveyard.
Put a 2/2 green Wolf creature token onto the battlefield. Wolves fear the lightning, always necrogenaly, what it makes no difference. Leave nothing unguarded in Scarwood.
#(Whisper: "Wolves")
Show of Warpary 2
Artifact 2, T: Exile target creature you control, then return that card to the battlefield under your control. Warpary was dead. In just one night, the dwarven forces had return the the line, he traverse grows on which they are fearsomes to the Auriok tempered from here, even scaring the power of Shiv.
#(Whisper: "Warpary was") Evidently this artifact could not save Warpary herself against the dwarves.
Sunbringer's Touch 2W
Enchantment
Whenever an opponent casts a creature spell, uncast it unless that player pays XX, where X is the number of cards in all graveyards with the same name. The Sunbringer extinction for revove area to the grave, and it is to whire it comes in a single play for power.
#(Whisper: "The Sunbringer")
Ruby Leaf 1R
Instant
Ruby Leaf deals 3 damage to target creature or player.
Whenever one or more creatures you control deals combat damage to a player, if Ruby Leaf is in your graveyard, you may return it to your hand. "The leaves of destruction has make upon a rootombult. Now they will become the flame." - Lim-Dul, the Necromancer
#(Whisper: "\"The leaves") I just liked the name Ruby Leaf. But that flavor text is too perfect, even if it's disjointed.
I can't stop laughing at the flavor text the machine can come up with. It's just too good sometimes.
I think the Ruby leaf is a bit Undercosted, Volcanic Hammer does 3 damage for 2 so the leaf should be a bit more expensive say RRR. "Show of Warpary" is awesome, it would work very well with the old Kicker Battlemages like "Stormscape Battlemage" assumed the Kicker-effect would applie again. I could see myself playing Sunbringers touch.
Overall there is some quite good combo-material lately in these cards.
That's true, actually. What I think is happening is that shuffling the fields has weakened the length restrictions on text. Longer text bodies give more opportunities for self-reference, which is why we're seeing more cards like Sunbringer's Touch rhan we did in the past.
A thought: what if the card database had cards duplicated in amounts that corresponded to their as-fan in booster packs? A common would be trained on more times than an uncommon, and so on. Would this teach the network that some effects/cmcs/etc show up more frequently in packs than others, leading to better pack generation for, say, drafts?
Or is the correct approach to start training it on rarities, then once it understands those, building packs by whispering rarities to the netowrk?
A thought: what if the card database had cards duplicated in amounts that corresponded to their as-fan in booster packs? A common would be trained on more times than an uncommon, and so on. Would this teach the network that some effects/cmcs/etc show up more frequently in packs than others, leading to better pack generation for, say, drafts?
Or is the correct approach to start training it on rarities, then once it understands those, building packs by whispering rarities to the netowrk?
That can work, but would also up the training time and would definitely encourage overfitting on certain features. My thought would be to do as you suggested and teach it card rarity and then I can just ask it to generate, say, rares, and filter from there according to our needs.
As I'm waiting for my latest experiment to finish cooking, I'm thinking of the dawn of computer science, when people would punch cards, put them in the machine, and would have to wait 1-2 days to get to know the outcome. Now similar processing is almost instantaneous, but it's happening again with neural nets: I have many things I'd like to try NOW but I have to wait because of processing power being what it is. I'm wondering if a new era of engineer-cooks may be coming about now.
The problem we're seeing is that we don't have the economies of scale that big companies like Google do. The neural network problems we're tackling can in many cases be parallelized across many GPUs or broken down and similarly distributed across large clusters of machines. A lot of NN research is coming out of big tech companies these days because they have limitless amounts of data for training and funding to match their ambitions.
- The splicing question is actually simple: there are 3 candidate lengths after which splicing may happen (depending on what the code actually does):
* seq_length
* batch_size * seq_length, that's the size of one batch
* batch_size * seq_length * (validation_frac / training_frac), that's the size of a run of training batches before one validation batch comes up
If it's the third case we're good, if it's the first we're screwed. ... Talcos?
It should be the first option. At least, as far as I know from studying the code.
- it's good to have p/t and manacost correlated but it could be the case that the network simply learns to associate many p/t's with many costs without developping any abstract notion of size / cmc. Which seems rather probable after all. Which is a bad news for getting ability cost better assessed (this one is more complicated since more expensive cards may have less expensive cost for a given ability as a trade-off).
The machine is a very, very, very lazy creature. It seeks to learn how to emulate Magic cards as effectively as possible with the least amount of mental effort.
For example, I can teach an LSTM network to add two ten digit numbers together and give me the answer, and it has no trouble learning how to do it. However, the moment you get up to 11 or 12 digits, you start seeing a drop-off in accuracy. What happens is that by only training on numbers up to ten digits, it develops a "mental loop" that performs the carry operation that far. It hasn't learned a general rule for n-digit numbers because in training, it never had to worry about that.
To be fair, most human mental math capabilities only go so far. At some point, when the numbers get large enough, people switch over to a hard-coded algorithm performed on an external, permanent memory device (e.g. paper).
For the same reason, teaching it to assess trade-offs is going to be tricky. However, it may be the case that it's smarter than we know. For example, in my own analysis of Magic's creatures, I found that the flying ability was worth 0.3414 CMC, as in that's how much flying contributes to CMC on average. We could probably do similar tests on a dump of the network's output to see how it weights those abilities.
@Talcos, you're under the false assumption that the format I've posted the cards into is the input format. That's not the case at all, it's heavily postprocessed by the postprocess.vim script. In particular the repetition you've seen on that card with Morph is the result of that script NOT deleting repeated repeaters due to approximate repetition (which I want to be able to see to better assess what's happening with a card), normally all cards begin with two repeaters, then there's one for each \n in text, then one again at the end just before manacost and name, and the script takes those 3 last infos and puts them at the beginning. Uppercases are put back on types and abilities etc. I've meant to prepare a description I'll post someday, you can look at input.txt from my files in the meantime if you want. I don't have the training parameters with me but you can also ask the checkpoints if you want to know now.
Oh, sorry! My bad. I misunderstood what I was reading.
About color sense: did you take a look at the dumps, or just the highlights? In the first case, that's enough for me now if there's a need for an analysis. It's just so obvious that the absolute costs are way off, no need for nothing more to know that, but last time I played MTG was probably '98 so my color sense...
Not yet, no. Just the highlights thus far. With large quantities of cards I prefer to let my machine do the statistics work rather than trust my own judgement, haha.
EDIT: I figured out the image generation training bug. Running a small test on just goblin cards to see how things go. I'll check back on it in a few hours. I have it set up to generate test images after every epoch, so I can see what the network is and isn't learning.
So, I've been making some good progress. I've attached a picture for reference. On the left is a sample taken from the picture Mindstatic. I take that picture and compute a vector describing it and then the generator has to decode that description to reconstruct the original image. And it does.. pretty good, actually! It's missing some details, but it got the whole "spellcaster looking stage left holding a magic staff" part, so that's a success. And the image was not part of the training data, so I know it's not cheating.
And by varying the input, I should be able to get different outputs... once it's fully trained... and once I write a better sampling script. And even if this doesn't pan out, I have the deconvolution code that I can tailor to our needs.
But really what I want to do next is add in rarity and a way to produce packs of cards for drafting purposes. I like that the network can generate tons of variants of common-level cards, like these:
Deflection Plague 1B
Instant
Target creature gets -3/-3 until end of turn. If you control a Wizard, draw a card. Suddenly he realized that everything was connected to the leg stone...
Dragonscale Blessing 1G
Instant
Untap target creature. It gets +3/+3 until end of turn. "Now I lay our language of idiot magic stole life, we don't mean just your career."
Akroma's Favor 1W
Instant
Convoke
Creatures you control get +1/+1 until end of turn. Akroma found that gravity is the one treat.
Suddenly he realized that everything was connected to the leg stone...
This might very well be one of the greatest quotes in magic history.
Seriously though, the way this has evolved is genuinely amazing.
Haha, I'm glad you've enjoyed the adventure we've been going on.
And if you think all this is impressive, just wait and see what the state-of-the-art will look like, say, five to ten years from now. What we're doing here will seem primitive by comparison. I'm really excited about the possibilities.
Really? I'm thinking it could very well be the case that most of the problems are I'm trying to improve upon are due to splicing. There's evidence enough that the network can learn from rather limited data. The python-code guys certainly had no splicing. I want NO MORE SPLICING. How can I get that?
1) will I be ok if I make sure no card crosses the boundaries right after every k*seq_lenth'th character? Any possible relaxation on this constraint?
2) how does this splicing occur exactly, what part is stichted to what? If it's deterministic I don't understand why we don't first scramble input.txt around just the right way that the right ends will be back together again after splicing. (karpathy would have done that if possible... no?)
I'm thinking we're using code aimed at high GPU performance on data that tolerates a bit of moving around, while we want reliability on data that makes no sense when spliced together (plus I'm CPU anyway lol).
EDIT to come when I have the time, some highlights prepared already on my google drive.
The simplest way to avoid this problem is to lengthen every card entry with a sequence of special padding characters such that every line is of length 500 (maybe a little more, depdending on your input format). That covers almost every card except outliers like Dance of the Dead. Set that value to be the sequence length. That way, every card gets considered separately, with no chopping up or splicing of input. The issue with that is that the training script will iterate over those padding characters and train on them, which is bad because the average card is about 140 characters long, which means that training on every card would involve an average of 360 useless forward/backward passes of the network, which would ridiculously lengthen the time it takes to train said network.
The fix for that problem is to add in a few lines of code that check to see whether we have reached a padding character, and if so, to discontinue the training process for that particular card/sequence. That way the data gets loaded in correctly but we aren't negatively affected by padding when it comes to training time.
I haven't had any time to do that yet though, in part because my dissertation work eats up a lot of my time, haha. My current efforts are split between image/flavor generation and doing testing with rarities for the purpose of creating draftable packs/sets. I will get to it eventually, unless you want to go ahead and try and I might be able to offer some helpful advice.
Woo I'm back from a long camping trip.
I am still working on the fundamentals of a web front end (link). I've written a (currently ugly) card system. I've disabled card image pulling for now due to technical reasons*.
Next goal is to 100% fix card text formatting and improve the look of the card.
Following that will be pdf page generation.
And after that a real interface for card generation instead of copy and pasting text.
Ugly Example:
*This is the most bizarre issue. I have no issues with curl, wget, or basic python2 or 3 sockets. However urllib and requests libraries have an 18 second delay before they send the GET packet for anything that needs DNS lookups. I am stumped.
Woo I'm back from a long camping trip.
I am still working on the fundamentals of a web front end (link). I've written a (currently ugly) card system. I've disabled card image pulling for now due to technical reasons*.
*This is the most bizarre issue. I have no issues with curl, wget, or basic python2 or 3 sockets. However urllib and requests libraries have an 18 second delay before they send the GET packet for anything that needs DNS lookups. I am stumped.
Welcome back! I hope you enjoyed your camping trip. Since you left, I've been having fun with flavor text generation and art generation.
I'm still figuring out how to sample the art-generating network. Feeding it random inputs just ends up with this tangled mess of organic looking shapes. Trying to see how I can get things to be a little more coherent.
Right now I'm training a network with rarity added in, to hopefully enable the construction of draft-able sets.
As for your problem, I'm still stumped as well. I can't think of a reason why you'd be experiencing that issue. But I'll ponder on it some more, might consult the internet, see if I can name a culprit.
Right now I'm training a network with rarity added in, to hopefully enable the construction of draft-able sets.
If this works for you, could you try one with NWO-era commons as their own rarity, or otherwise tagged? That might be useful for making sets that look like Wizards' sets. I'm also curious if the network can discern any of the NWO rules. Card-length would be the easiest, I bet. It might also figure out not to put as many on-board combat tricks at NWO-common.
Right now I'm training a network with rarity added in, to hopefully enable the construction of draft-able sets.
If this works for you, could you try one with NWO-era commons as their own rarity, or otherwise tagged? That might be useful for making sets that look like Wizards' sets. I'm also curious if the network can discern any of the NWO rules. Card-length would be the easiest, I bet. It might also figure out not to put as many on-board combat tricks at NWO-common.
That's a very good idea, definitely something to try. I'll wait until the current training process finishes, of course, to see how well things work without the added tags.
I would guess the RNN will have some problems with Super-rares (or however they are called) because there are so few and they were introduced pretty late.
That's actually something I'm going to be doing some analysis on. For example, there are only 5 cards with the Grandeur ability, yet that doesn't stop the network from creating cards with abilities of the form "Grandeur — Discard another card named THIS: ". But then, that's just rote memorization, doesn't take a lot of brain power.
To prevent overlap with other commonly used letters I represented rarities as {O,N,A,Y} (taking the second letter)/ I suspect that what will happen is that the RNN will learn that "Y" is just a synonym for "A". For me, an even more interesting question is what the network learns about the distinction between commons and uncommons. Some data analysis is in order.
If the results are stable enough with respect to rarity, then we can look into scripts for set generation. We could generate a huge dump of cards (with priming if desired), and then pick cards out according to our needs. Here content vectors will be extremely useful:
Sylvan Redisent 1G
Instant
Target creature gets +4/+4 and gains hexproof until end of turn.
Slightly better than Titanic Growth. It'll do. Once the set skeleton is filled out, then we can make a pass over the resulting set to clothe them with flavor text and (if I am successful) artwork.
EDIT: I took that card from one of the dumps hardcast_sixdrop posted, and not long after I saw that one, I spotted a very small creature that I very much appreciated:
Orim's Avenger 2WW
Creature - Human Soldier
Cascade
1/3
I'm confident that the network can produce enough variety in a large pool of cards that we can be very flexible with the sets we create. Priming of course can take us the rest of the way, but having a solid foundation is important.
Orim Avenger is Rather odd, Cascade only appears on Multicolour cards so far. Also there are "Timeshifted" Cards from the Timespiral block which have theyr own "rarity" as "Timeshifted" cards.
edit: Just occured to me if you make a draft, how about applying the Planar Chaos/Time shift Carddesign? Would be even cooler if we could come up with an overarching narrative that goes with it
I forgot about the timeshifted cards. I suspect that if they did make it into the set, they'd be labeled as "I" rarity cards. But I just made a sweep of the input corpus and couldn't find it. I think hardcast_sixdrop filters those cards out with his encoding script, or that they are otherwise left out because he only wants one version of a card in the corpus and a version from an earlier set gets chosen first). In any case, thanks for bringing that to my attention.
And that's a fun idea. I'd love to try and take the cards and their semi-coherent flavor text and weave it into something a little more cohesive, haha.
I'm curious how alternate win/lose conditions are handled, as well as what sorts of annihilator cards are generated.
The next version of the network is still in training and I ought not disturb the machine. However, we can look at card dumps from previous versions of the network to give you an idea. If you go here, thanks to hardcast_sixdrop, you'll find a collection of network checkpoints and text files containing card samples. Later versions of the network tend to deliver better results than earlier ones, so I dug through a few of the more recent dumps to answer your question:
As for cards that cause players to lose/win the game, they occur about as rarely as they do in the real game. But because they are so rare, it's obvious that the network doesn't really understand the gravity of the words "lose" and "win":
Nightshade Angel 3B
Creature - Angel
Flying
Whenever Nightshade Angel deals combat damage to a player, you win the game.
Madness 3B
1/3
#See what I mean?
Ten-Caller's Leficuon 7
Artifact T: Choose target opponent. That player discards his or her hand. If that player can't, you lose the game.
#"In response to the activation, I cast One with Nothing. You lose!"
As for cards with the Annihilator ability, it usually shows up on big Eldrazi (sometimes Golem/Construct) creatures with intimidating names:
Skittering Into the Dead 13
Legendary Creature - Eldrazi
Annihilator 4
When Skittering Into the Dead leaves the battlefield, draw a card.
10/10
Slayers' Colossus 11
Artifact Creature - Golem
Annihilator 3 1R Slayers' Colossus gets +3/+0 until end of turn.
9/9
Note that among real Magic cards, the annihilator ability has only shown up on Eldrazi cards, the network is comfortable giving the ability to pricey golems and constructs also. This is because the machine identifies a certain semantic equivalence between a card like Darksteel Colossus (big, colorless, and powerful) and Artisan of Kozilek (also big, colorless, and powerful). This I think is one of the strengths of the machine, because it can make these sorts of connections and can bend rules in order to keep things fresh and interesting. Of course, by the same token, we're still struggling to get it to obey color discipline outside of basic stuff like guaranteeing that blue gets counterspells and red gets damage spells. It's doing better than it has in the past, but it still has a ways to go.
Now, at high sampling temperatures (around 0.9), you start to see annihilator elsewhere:
Oril, Murk Pordil 1RR
Legendary Creature - Centaur Human
Annihilator 3
Sacrifice Oril, Murk Pordil: Oril deals 2 damage to target creature or player.
2/1
But that's extremely rare at lower sampling temperatures, indicating that the network resists the idea of giving a creature annihilator unless it is big and imposing enough to justify it.
Talcos, are you sure there's splicing involved even with no validation batches? My 100% training network currently in learning performs better than expected (could also be due to instabilities in the learning process)
Next idea: stop the training at regular points, rerandmize the card order, and relaunch
I would hate to say that I know anything for certain only to be wrong, so I'll take a look at the code (perhaps this evening) and see if I can give you a satisfactory answer.
And yes, rerandomizing the card order would be good, but it would require some rewrites of the existing code. Right now the card corpus goes in, gets split up into training sequences by the loader, and then gets used. We'd have to randomize the original text and then re-sequence everything with the loader if we wanted to do that. It's definitely do-able though, and would help prevent the network from picking up on any spurious relationships.
---
EDIT: Oh, and to Croxis, I still don't know why you're having the issues with the 18+ second delays. We might try running your code in a different environment (different machine, different network) to see if the problem still happens. If it does, then it's either an issue with your client code or the library code. If the problem goes away, then it's something particular to the circumstances in which you are running your code.
EDIT: Oh, and to Croxis, I still don't know why you're having the issues with the 18+ second delays. We might try running your code in a different environment (different machine, different network) to see if the problem still happens. If it does, then it's either an issue with your client code or the library code. If the problem goes away, then it's something particular to the circumstances in which you are running your code.
It only happens on my vps. Urllib (which is in the standard lib) and requests in a docker container on my vps have normal response times. It is something very specific to my VPS environment. Normally I would just nuke my server (yay separate home partitions!) but no one can pay me enough money to setup postfix again.
Cards look a bit better, although this supposedly correct font still looks awful. My internet search says matrix bold is the correct font, but if anyone knows of any alternatives I would love to know about it/have it. I am having issues with counter text. I'll probably need harddrop to look at my python3 version of his code to see where I am going wrong. Right now I'm working on generating pdf pages for downloading and printing.
Private Mod Note
():
Rollback Post to RevisionRollBack
Proud to be saving the world since 1984 -- I also have an open source website to make AI generated magic cards. Source code
Finally finished reading this whole thread. What an amazing project. I'm on Windows and too lazy to have dual-boot for Linux, so I'll just keep reading the interesting cards posted here. I have two questions.
1. I have seen a number of cards with "uncast" as part of the card text. I know the RNN has made up keywords like "tromple" and "fuseback" but this one seems weird because it makes contextual sense as as substitute for "counter target spell", and that it's usually paired with other text that relates to countering spells (such "unless its controller pays 2"). Where is it getting this from? Is it really just coincidence that it both invented a word and uses it in a way that makes sense based on what that word would actually mean?
2. About the training process... Well, you guys seem to have it working much more reliably now, but I was wondering if, when training for scratch, it would benefit by starting with a a simplified list, for example all the basic vanilla creatures or ones that have only simple abilities like trample or flying, and let it design cards just based on that. When it seems to have mastered that, add new cards into the training pool, like sorceries and instants, again picking just simple cards like Lightning Bolt, Dark Ritual, etc. Let it make new cards with the new data combined with the creature data it already mastered, and then keep adding new cards to the training pool of increasing complexity. Since a NN is supposed to try to simulate how human brains learn, it seems it might benefit best from learning the same way humans learn (in steps, from easy to difficult complexity). Has anyone considered trying this approach?
Talcos, are you sure there's splicing involved even with no validation batches? My 100% training network currently in learning performs better than expected (could also be due to instabilities in the learning process)
Next idea: stop the training at regular points, rerandmize the card order, and relaunch
If I'm not mistaken, the code breaks up the data into contiguous chunks of size seq_length. So if you choose your seq_length to be 200 and your data consists of two 300-length cards, the training sees that as three separate cards, where the middle card is made up of the last 100 characters of the first card and the first 100 characters of the second card. And as far as the machine knows, all three of those are real, valid cards.
Finally finished reading this whole thread. What an amazing project. I'm on Windows and too lazy to have dual-boot for Linux, so I'll just keep reading the interesting cards posted here. I have two questions.
1. I have seen a number of cards with "uncast" as part of the card text. I know the RNN has made up keywords like "tromple" and "fuseback" but this one seems weird because it makes contextual sense as as substitute for "counter target spell", and that it's usually paired with other text that relates to countering spells (such "unless its controller pays 2"). Where is it getting this from? Is it really just coincidence that it both invented a word and uses it in a way that makes sense based on what that word would actually mean?
You are correct. Uncast is a standin for counter target spell. IIRC they found that the neural net understood "uncast" better than "counter target spell." It will just be a matter for formatting in conversion scripts.
EDIT: The web front end now renders PDFs (hardcoded to American letter size at the moment) of cards. Github and site are both updated. Still having issues with tokens, counters, and tap symbols.
"Uncast" was a preemptive disambiguation in order to try and avoid the network getting confused between countering spells and putting counters on things.
Well, it wasn't so much preemptive as reactionary - I think it solved some of our problems with permanents countering spells and instants/sorceries getting counters on them.
What is the current "sophisticated" way to generate cards from checkpoints? I know you worked on a script to manage some of the parameters and whisper suggestions but, at 50 pages, this is quite the topic to plow through
Also there are issues with gpu checkpoints unusable with just cpu methods. Has this been resolved too?
Private Mod Note
():
Rollback Post to RevisionRollBack
Proud to be saving the world since 1984 -- I also have an open source website to make AI generated magic cards. Source code
Dahammer4
Artifact - Equipment (R)
Choose one - Equipped creature gets +4/+0; or Double Strike; or "When this creature deals combat damage to a creature, exile that creature"; or t, unattach ~ from this creature: ~ deals 5 damage to target creature." t: Equipped creature gains flying until end of turn.
Equip 3
What is the current "sophisticated" way to generate cards from checkpoints? I know you worked on a script to manage some of the parameters and whisper suggestions but, at 50 pages, this is quite the topic to plow through
Also there are issues with gpu checkpoints unusable with just cpu methods. Has this been resolved too?
That's about as sophisticated as we have right now. Scripts for set generation will push the envelope on that, however.
Also, I promise that I'll release a gpu-to-cpu conversion script soon. Sorry for not having it sooner, haha.
How is your generator different from this one, which has been around for quite a while?
That requires a manually constructed BNF grammar. It's very fragile and difficult to extend. The RNN, meanwhile, is trained from scratch and can be altered by feeding it different inputs during training. Furthermore, the RNN also has the advantage of automatically inferring semantics, letting it make greater leaps in terms of the variety of the cards it creates (e.g. realizing that Eldrazi are in some ways synonymous with large Golem creatures). In short, the RNN removes most of the human engineering from the process and can deliver much better results.
For example, the only way to get the BNF-grammar-driven generator to produce cards with mechanics from a new set is to rewrite or extend the grammar. How do I do it? I simply show it the cards and let it study them.
EDIT: Oh, and priming is literally impossible. You can't really push it in any direction other than the select few that the algorithm wants to go in.
EDIT(2): If it sounds like I'm being harsh, I'm not. Churchill's work is very ingenious. The main thing that I'm doing differently is that I'm taking the human out of the equation (for the most part). He has to nurse his brainchild. Mine is an independent hunter.
Finally finished reading this whole thread. What an amazing project.
I'm glad you've found it so interesting! I hope that this has been an entertaining prelude to the more serious issue of a looming unemployment crisis. But that's a discussion for another time, haha.
2. About the training process... Well, you guys seem to have it working much more reliably now, but I was wondering if, when training for scratch, it would benefit by starting with a a simplified list, for example all the basic vanilla creatures or ones that have only simple abilities like trample or flying, and let it design cards just based on that. When it seems to have mastered that, add new cards into the training pool, like sorceries and instants, again picking just simple cards like Lightning Bolt, Dark Ritual, etc. Let it make new cards with the new data combined with the creature data it already mastered, and then keep adding new cards to the training pool of increasing complexity. Since a NN is supposed to try to simulate how human brains learn, it seems it might benefit best from learning the same way humans learn (in steps, from easy to difficult complexity). Has anyone considered trying this approach?
You're absolutely right. We call that curriculum learning. For example, to teach an RNN to understand Python code, it's best to start by teaching it arithmetic, then logic, and then to bring all that together in programs of increasing complexity.
The only problem here is that while we have infinitely many statements of math and logic to draw upon, we don't have many Magic cards. That being said, we have shown that it's possible to expand the corpus with feeder networks, and this could provide ample material for cirruculum-style learning. You bring up a good point.
---
So, I concluded training of the latest version of the network with rarity added in and field shuffling disabled (for now). I took some samples and the results look decent (with respect to rarity), but I'll need to do some analysis to see how well it gets the concept. For example, there's clearly a difference between rare cards and common cards, but does the network really understand the role of uncommons? That's something that statistics can help us to determine. When I have the opportunity, that is. At this point I'm spending a lot of time working on my dissertation, writing about the mathematics underpinning our Magic-generating RNNs. Truth be told, the experiments that we've been doing here with Magic have actually been immensely beneficial for my PhD research.
Anyway, we've settled the question of whether we can create content that is complex and beautiful. We've determined that is possible. The question that remains is whether or not we can create fun experiences for players. Beyond all the technical challenges, that's the real goal.
I'm eager to do some playtesting in the near future. That's been my goal since day one. But here's the catch: neither the machine nor I know anything about set design. If anyone can recommend a set skeleton of some kind, or a way of conjuring one, I can take that skeleton and write scripts that take the output from the network (possibly after priming it) and select cards automatically to fill certain roles in the set.
Like I said before, we have numerous tools at our disposal to express a set skeleton programatically:
* Since we can evaluate cards based on their semantic content using content vectors, it's possible to specify rules like "I need a pump spell like Giant Growth".
* We can do direct queries like "I need a white 5-drop creature at common with high toughness."
* We can prime the network to generate output that fills certain needs and then selects among the results. This means we can ask for stuff like "give me a Rat lord and cards that care about Rats" or "give me a metalcraft theme in red and white."
* We can take cards that the network produces and colorshift our way to cycles (e.g turning one dual-land into five).
* We can be conservative with our selections, like skipping cards with undefined X's.
It's also possible for us to generate flavor and artwork for the cards (as you have seen before), but I'll set that aside for now and return to it later. What we'll do is have a set and then run the set through scripts that call the flavor-generating/art-generating networks to clothe the cards prior to their use.
So if anyone has any suggestions, I'm all ears.
---
A few examples from the latest network. I'm noticing that the rarity seems appropriate more often than not. I wanted to showcase three at each rarity, let me know what you think.
Heat Shadow 4U
Instant (Common)
Put target artifact, creature, or land on top of its owner's library.
Spirit Marsh
Land (Common)
Spirit Marsh enters the battlefield tapped.
When Spirit Marsh enters the battlefield, sacrifice it unless you return an untapped creature you control to its owner's hand. T: Add RG to your mana pool.
Death's Herald 1BB
Creature - Zombie (Uncommon)
Whenever an opponent casts a white spell, put a 1/1 green saproling creature token onto the battlefield.
2/2
Time Control 2U
Instant (Uncommon)
Counter target spell unless its controller pays 4.
Brood Force 2G
Creature - Insect (Uncommon)
Flash
When Brood Force enters the battlefield, put a +1/+1 counter on target creature.
2/2
Stoneshaper Bow 4
Artifact (Rare) 5, T: Choose one:
* Destroy target nonblack creature. It can't be regenerated.
* Put three 1/1 white Kithkin creature tokens onto the battlefield.
* You gain one life.
* Destroy target artifact or enchantment.
Monstrous Charm 2R
Instant (Rare)
Monstrous Charm deals 4 damage to target creature or player.
After this main phase, there is an additional combat phase. Shuffle Monstrous Charm into its owner's library.
Keeper of the Thorns 3U
Creature - Human Wizard (Rare) 1U, T: Untap all creatures you control.
1/1
Memory Conscience 2WW
Enchantment (Mythic)
Whenever a nontoken creature enters the battlefield under your control, you may pay 1. if you do, put a 3/3 green Beast creature token onto the battlefield.
Battle-caller of the Horde 6BBB
Creature - Demon (Mythic)
Flying 1B, sacrifice a creature: Battle-caller of the Horde deals damage equal to its power to target creature or player.
8/8
Thornwind Angel 4WW
Creature - Angel (Mythic)
Flying
Whenever Thornwind Angel deals combat damage to a player, you may search your library for an artifact card with converted mana cost 5 or less and put it onto the battlefield. Then shuffle your library.
5/5
So it looks like the rarity thing is working pretty well. Not perfect, but it's decent. Next are a few unusual cards that I wanted to bring to your attention. I've attached flavor text for each of them (I primed the first word in each sentence and let it go from there).
Wall of Conschiet W
Creature - Human Soldier (Common)
Whenever Wall of Conschiet is dealt damage, you may put that many 2/2 green Bear tokens onto the battlefield. Bears sweatte for their wives' soil.
1/1
Trained Agent 3R
Creature - Goblin Warrior (Rare)
At the beginning of each upkeep, if no spells were cast last turn, transform Trained Agent. Goblins are cautioned ofter seeking to itself, they can't be eatenable to serve the beasts.
3/2
///
Savage Dragon
Creature - Dragon
Flying
When Savage Dragon enters the battlefield, you gain 2 life. Dragons emerge from the moment wizards say that boggart gone. His home.
6/6
---
Oh, and one more thing. Questions were raised earlier about the possibility of generating D&D spells. I looked into that and I feel like the neural architecture we're using is less than ideal for the task. The problem is that they exhibit micro-narrative structures for which our network has difficulty keeping track of. For example, we want:
This spell lets you breathe underwater.
-> You can swim under the ocean.
-> Breathing returns to normal after 1d8 minutes.
And our network is likely to give us outputs like:
This spell lets you breathe underwater.
-> Fish live underwater.
-> You can command the fish for 1d6 rounds.
Solving this sort of problem is challenging given the current state-of-the-art. And by that I mean we'll probably figure it out within the next few months. That's how fast all this is moving.
Orim Avenger is Rather odd, Cascade only appears on Multicolour cards so far. Also there are "Timeshifted" Cards from the Timespiral block which have theyr own "rarity" as "Timeshifted" cards.
edit: Just occured to me if you make a draft, how about applying the Planar Chaos/Time shift Carddesign? Would be even cooler if we could come up with an overarching narrative that goes with it
It occurs to me that with the time travel/alternate reality storyline in Tarkir, it might be interesting to take the concept to Dominaria and create a war between the various 'Walkers, all trying to edit the history of the plane to their own advantage. Since it's sort of the prime plane/hub of the multiverse, changes to Dominaria's timeline have a ripple effect throughout the other planes as well - for instance, killing Karn during the Phyrexian invasion would prevent Mirrodin from ever existing, and thus no New Phyrexia. Or intervening in the Brothers' War prevents the Invasion from taking place at all, since Yawgmoth never captures Mishra. Etc.
And the storyline could conclude with a big ol' reset button so it will never be obsoleted by the official continuity.
Edit: And as far as the flavor... You know how Coldsnap took the storyline, themes, and mechanics of the Ice Age block and updated them to a Modern design standard? What if the RNN-generated set did the same with multiple older sets? Arabian Nights, Antiquities, Legends, The Dark, Fallen Empires, Homelands, Ice Age, Mirage, Tempest, Mercadian Masques, Invasion, Onslaught... All the pre-Modern sets, given the Coldsnap treatment.
We'd have to come up with some mechanics, and I'm thinking that rather than have the RNN generate names and flavor text completely at random, we'd have someone write bits of flavor text and let the RNN decide which cards to stick them on. And with names, we could write a list of words that the RNN should look at x% of the time when naming a card, with stuff like "Krovikan", "Thallid", and "of Ith", which it would then prime the card name with either as a prefix or suffix.
I was a big lore buff back in the olden days of MtG, so I could probably help in creating both.
Good points!
By the way...
Mediocre news: I don't have too much to show yet with the art generation.
Good news: I've definitively established how to do it right!
So, I was experimenting with the Eyescream package (thanks to researchers at Facebook). I was inspired by their ability to generate images of virtually anything given the right data, and that they state it's possible to map word-embeddings (like my content vectors) to images (though they haven't implemented that part yet). I'm missing some packages needed to do the big version of their network, but I can do the small one. I got about one epoch of training out of it and then ran into some weird bug. I tried training the network on creatures, but since I wasn't able to complete the training, all I can get out of it are vague, ethereal blobs (like the image I attached, I took a very low resolution sample with a custom script). I'm fairly confident that I was able to teach the network that creature artworks have figures in the foreground and stuff in the background, but that's all I was able to get. I'm sure if I trained more I could get things looking better, but I'll have to figure out why I got that error.
EDIT: A few interesting finds. I've combined them with generated flavor text as well (I sometimes primed the flavor text with a relevant word from the card, just to see what I would get).
Sandstorm
G
Exile target creature card from your graveyard.
Put a 2/2 green Wolf creature token onto the battlefield.
Wolves fear the lightning, always necrogenaly, what it makes no difference. Leave nothing unguarded in Scarwood.
#(Whisper: "Wolves")
Show of Warpary
2
Artifact
2, T: Exile target creature you control, then return that card to the battlefield under your control.
Warpary was dead. In just one night, the dwarven forces had return the the line, he traverse grows on which they are fearsomes to the Auriok tempered from here, even scaring the power of Shiv.
#(Whisper: "Warpary was") Evidently this artifact could not save Warpary herself against the dwarves.
Sunbringer's Touch
2W
Enchantment
Whenever an opponent casts a creature spell, uncast it unless that player pays XX, where X is the number of cards in all graveyards with the same name.
The Sunbringer extinction for revove area to the grave, and it is to whire it comes in a single play for power.
#(Whisper: "The Sunbringer")
Ruby Leaf
1R
Instant
Ruby Leaf deals 3 damage to target creature or player.
Whenever one or more creatures you control deals combat damage to a player, if Ruby Leaf is in your graveyard, you may return it to your hand.
"The leaves of destruction has make upon a rootombult. Now they will become the flame." - Lim-Dul, the Necromancer
#(Whisper: "\"The leaves") I just liked the name Ruby Leaf. But that flavor text is too perfect, even if it's disjointed.
I can't stop laughing at the flavor text the machine can come up with. It's just too good sometimes.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
That's true, actually. What I think is happening is that shuffling the fields has weakened the length restrictions on text. Longer text bodies give more opportunities for self-reference, which is why we're seeing more cards like Sunbringer's Touch rhan we did in the past.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
Or is the correct approach to start training it on rarities, then once it understands those, building packs by whispering rarities to the netowrk?
That can work, but would also up the training time and would definitely encourage overfitting on certain features. My thought would be to do as you suggested and teach it card rarity and then I can just ask it to generate, say, rares, and filter from there according to our needs.
The problem we're seeing is that we don't have the economies of scale that big companies like Google do. The neural network problems we're tackling can in many cases be parallelized across many GPUs or broken down and similarly distributed across large clusters of machines. A lot of NN research is coming out of big tech companies these days because they have limitless amounts of data for training and funding to match their ambitions.
It should be the first option. At least, as far as I know from studying the code.
The machine is a very, very, very lazy creature. It seeks to learn how to emulate Magic cards as effectively as possible with the least amount of mental effort.
For example, I can teach an LSTM network to add two ten digit numbers together and give me the answer, and it has no trouble learning how to do it. However, the moment you get up to 11 or 12 digits, you start seeing a drop-off in accuracy. What happens is that by only training on numbers up to ten digits, it develops a "mental loop" that performs the carry operation that far. It hasn't learned a general rule for n-digit numbers because in training, it never had to worry about that.
To be fair, most human mental math capabilities only go so far. At some point, when the numbers get large enough, people switch over to a hard-coded algorithm performed on an external, permanent memory device (e.g. paper).
For the same reason, teaching it to assess trade-offs is going to be tricky. However, it may be the case that it's smarter than we know. For example, in my own analysis of Magic's creatures, I found that the flying ability was worth 0.3414 CMC, as in that's how much flying contributes to CMC on average. We could probably do similar tests on a dump of the network's output to see how it weights those abilities.
Oh, sorry! My bad. I misunderstood what I was reading.
Not yet, no. Just the highlights thus far. With large quantities of cards I prefer to let my machine do the statistics work rather than trust my own judgement, haha.
EDIT: I figured out the image generation training bug. Running a small test on just goblin cards to see how things go. I'll check back on it in a few hours. I have it set up to generate test images after every epoch, so I can see what the network is and isn't learning.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
And by varying the input, I should be able to get different outputs... once it's fully trained... and once I write a better sampling script. And even if this doesn't pan out, I have the deconvolution code that I can tailor to our needs.
But really what I want to do next is add in rarity and a way to produce packs of cards for drafting purposes. I like that the network can generate tons of variants of common-level cards, like these:
Deflection Plague
1B
Instant
Target creature gets -3/-3 until end of turn. If you control a Wizard, draw a card.
Suddenly he realized that everything was connected to the leg stone...
Dragonscale Blessing
1G
Instant
Untap target creature. It gets +3/+3 until end of turn.
"Now I lay our language of idiot magic stole life, we don't mean just your career."
Akroma's Favor
1W
Instant
Convoke
Creatures you control get +1/+1 until end of turn.
Akroma found that gravity is the one treat.
We'll see what happens.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
Haha, I'm glad you've enjoyed the adventure we've been going on.
And if you think all this is impressive, just wait and see what the state-of-the-art will look like, say, five to ten years from now. What we're doing here will seem primitive by comparison. I'm really excited about the possibilities.
The simplest way to avoid this problem is to lengthen every card entry with a sequence of special padding characters such that every line is of length 500 (maybe a little more, depdending on your input format). That covers almost every card except outliers like Dance of the Dead. Set that value to be the sequence length. That way, every card gets considered separately, with no chopping up or splicing of input. The issue with that is that the training script will iterate over those padding characters and train on them, which is bad because the average card is about 140 characters long, which means that training on every card would involve an average of 360 useless forward/backward passes of the network, which would ridiculously lengthen the time it takes to train said network.
The fix for that problem is to add in a few lines of code that check to see whether we have reached a padding character, and if so, to discontinue the training process for that particular card/sequence. That way the data gets loaded in correctly but we aren't negatively affected by padding when it comes to training time.
I haven't had any time to do that yet though, in part because my dissertation work eats up a lot of my time, haha. My current efforts are split between image/flavor generation and doing testing with rarities for the purpose of creating draftable packs/sets. I will get to it eventually, unless you want to go ahead and try and I might be able to offer some helpful advice.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
I am still working on the fundamentals of a web front end (link). I've written a (currently ugly) card system. I've disabled card image pulling for now due to technical reasons*.
Next goal is to 100% fix card text formatting and improve the look of the card.
Following that will be pdf page generation.
And after that a real interface for card generation instead of copy and pasting text.
Ugly Example:
*This is the most bizarre issue. I have no issues with curl, wget, or basic python2 or 3 sockets. However urllib and requests libraries have an 18 second delay before they send the GET packet for anything that needs DNS lookups. I am stumped.
Welcome back! I hope you enjoyed your camping trip. Since you left, I've been having fun with flavor text generation and art generation.
I'm still figuring out how to sample the art-generating network. Feeding it random inputs just ends up with this tangled mess of organic looking shapes. Trying to see how I can get things to be a little more coherent.
Right now I'm training a network with rarity added in, to hopefully enable the construction of draft-able sets.
As for your problem, I'm still stumped as well. I can't think of a reason why you'd be experiencing that issue. But I'll ponder on it some more, might consult the internet, see if I can name a culprit.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
If this works for you, could you try one with NWO-era commons as their own rarity, or otherwise tagged? That might be useful for making sets that look like Wizards' sets. I'm also curious if the network can discern any of the NWO rules. Card-length would be the easiest, I bet. It might also figure out not to put as many on-board combat tricks at NWO-common.
That's a very good idea, definitely something to try. I'll wait until the current training process finishes, of course, to see how well things work without the added tags.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
That's actually something I'm going to be doing some analysis on. For example, there are only 5 cards with the Grandeur ability, yet that doesn't stop the network from creating cards with abilities of the form "Grandeur — Discard another card named THIS: ". But then, that's just rote memorization, doesn't take a lot of brain power.
To prevent overlap with other commonly used letters I represented rarities as {O,N,A,Y} (taking the second letter)/ I suspect that what will happen is that the RNN will learn that "Y" is just a synonym for "A". For me, an even more interesting question is what the network learns about the distinction between commons and uncommons. Some data analysis is in order.
If the results are stable enough with respect to rarity, then we can look into scripts for set generation. We could generate a huge dump of cards (with priming if desired), and then pick cards out according to our needs. Here content vectors will be extremely useful:
"The set needs to have a green pump spell. Find something related to Giant Growth, Predator's Strike, and Resize."
And then the machine chooses a card like:
Sylvan Redisent
1G
Instant
Target creature gets +4/+4 and gains hexproof until end of turn.
Slightly better than Titanic Growth. It'll do. Once the set skeleton is filled out, then we can make a pass over the resulting set to clothe them with flavor text and (if I am successful) artwork.
EDIT: I took that card from one of the dumps hardcast_sixdrop posted, and not long after I saw that one, I spotted a very small creature that I very much appreciated:
Orim's Avenger
2WW
Creature - Human Soldier
Cascade
1/3
I'm confident that the network can produce enough variety in a large pool of cards that we can be very flexible with the sets we create. Priming of course can take us the rest of the way, but having a solid foundation is important.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
I forgot about the timeshifted cards. I suspect that if they did make it into the set, they'd be labeled as "I" rarity cards. But I just made a sweep of the input corpus and couldn't find it. I think hardcast_sixdrop filters those cards out with his encoding script, or that they are otherwise left out because he only wants one version of a card in the corpus and a version from an earlier set gets chosen first). In any case, thanks for bringing that to my attention.
And that's a fun idea. I'd love to try and take the cards and their semi-coherent flavor text and weave it into something a little more cohesive, haha.
The next version of the network is still in training and I ought not disturb the machine. However, we can look at card dumps from previous versions of the network to give you an idea. If you go here, thanks to hardcast_sixdrop, you'll find a collection of network checkpoints and text files containing card samples. Later versions of the network tend to deliver better results than earlier ones, so I dug through a few of the more recent dumps to answer your question:
As for cards that cause players to lose/win the game, they occur about as rarely as they do in the real game. But because they are so rare, it's obvious that the network doesn't really understand the gravity of the words "lose" and "win":
Nightshade Angel
3B
Creature - Angel
Flying
Whenever Nightshade Angel deals combat damage to a player, you win the game.
Madness 3B
1/3
#See what I mean?
Ten-Caller's Leficuon
7
Artifact
T: Choose target opponent. That player discards his or her hand. If that player can't, you lose the game.
#"In response to the activation, I cast One with Nothing. You lose!"
As for cards with the Annihilator ability, it usually shows up on big Eldrazi (sometimes Golem/Construct) creatures with intimidating names:
Skittering Into the Dead
13
Legendary Creature - Eldrazi
Annihilator 4
When Skittering Into the Dead leaves the battlefield, draw a card.
10/10
Slayers' Colossus
11
Artifact Creature - Golem
Annihilator 3
1R Slayers' Colossus gets +3/+0 until end of turn.
9/9
Note that among real Magic cards, the annihilator ability has only shown up on Eldrazi cards, the network is comfortable giving the ability to pricey golems and constructs also. This is because the machine identifies a certain semantic equivalence between a card like Darksteel Colossus (big, colorless, and powerful) and Artisan of Kozilek (also big, colorless, and powerful). This I think is one of the strengths of the machine, because it can make these sorts of connections and can bend rules in order to keep things fresh and interesting. Of course, by the same token, we're still struggling to get it to obey color discipline outside of basic stuff like guaranteeing that blue gets counterspells and red gets damage spells. It's doing better than it has in the past, but it still has a ways to go.
Now, at high sampling temperatures (around 0.9), you start to see annihilator elsewhere:
Oril, Murk Pordil
1RR
Legendary Creature - Centaur Human
Annihilator 3
Sacrifice Oril, Murk Pordil: Oril deals 2 damage to target creature or player.
2/1
But that's extremely rare at lower sampling temperatures, indicating that the network resists the idea of giving a creature annihilator unless it is big and imposing enough to justify it.
I would hate to say that I know anything for certain only to be wrong, so I'll take a look at the code (perhaps this evening) and see if I can give you a satisfactory answer.
And yes, rerandomizing the card order would be good, but it would require some rewrites of the existing code. Right now the card corpus goes in, gets split up into training sequences by the loader, and then gets used. We'd have to randomize the original text and then re-sequence everything with the loader if we wanted to do that. It's definitely do-able though, and would help prevent the network from picking up on any spurious relationships.
---
EDIT: Oh, and to Croxis, I still don't know why you're having the issues with the 18+ second delays. We might try running your code in a different environment (different machine, different network) to see if the problem still happens. If it does, then it's either an issue with your client code or the library code. If the problem goes away, then it's something particular to the circumstances in which you are running your code.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
Choose one of these judge of creation:
Make Strionic Resonator shine!
You can not grasp the true form of Ashiok's attack!
It only happens on my vps. Urllib (which is in the standard lib) and requests in a docker container on my vps have normal response times. It is something very specific to my VPS environment. Normally I would just nuke my server (yay separate home partitions!) but no one can pay me enough money to setup postfix again.
Cards look a bit better, although this supposedly correct font still looks awful. My internet search says matrix bold is the correct font, but if anyone knows of any alternatives I would love to know about it/have it. I am having issues with counter text. I'll probably need harddrop to look at my python3 version of his code to see where I am going wrong. Right now I'm working on generating pdf pages for downloading and printing.
1. I have seen a number of cards with "uncast" as part of the card text. I know the RNN has made up keywords like "tromple" and "fuseback" but this one seems weird because it makes contextual sense as as substitute for "counter target spell", and that it's usually paired with other text that relates to countering spells (such "unless its controller pays 2"). Where is it getting this from? Is it really just coincidence that it both invented a word and uses it in a way that makes sense based on what that word would actually mean?
2. About the training process... Well, you guys seem to have it working much more reliably now, but I was wondering if, when training for scratch, it would benefit by starting with a a simplified list, for example all the basic vanilla creatures or ones that have only simple abilities like trample or flying, and let it design cards just based on that. When it seems to have mastered that, add new cards into the training pool, like sorceries and instants, again picking just simple cards like Lightning Bolt, Dark Ritual, etc. Let it make new cards with the new data combined with the creature data it already mastered, and then keep adding new cards to the training pool of increasing complexity. Since a NN is supposed to try to simulate how human brains learn, it seems it might benefit best from learning the same way humans learn (in steps, from easy to difficult complexity). Has anyone considered trying this approach?
If I'm not mistaken, the code breaks up the data into contiguous chunks of size seq_length. So if you choose your seq_length to be 200 and your data consists of two 300-length cards, the training sees that as three separate cards, where the middle card is made up of the last 100 characters of the first card and the first 100 characters of the second card. And as far as the machine knows, all three of those are real, valid cards.
You are correct. Uncast is a standin for counter target spell. IIRC they found that the neural net understood "uncast" better than "counter target spell." It will just be a matter for formatting in conversion scripts.
EDIT: The web front end now renders PDFs (hardcoded to American letter size at the moment) of cards. Github and site are both updated. Still having issues with tokens, counters, and tap symbols.
What is the current "sophisticated" way to generate cards from checkpoints? I know you worked on a script to manage some of the parameters and whisper suggestions but, at 50 pages, this is quite the topic to plow through
Also there are issues with gpu checkpoints unusable with just cpu methods. Has this been resolved too?
Sigpic by Rivenor
Artifact - Equipment (R)
Choose one - Equipped creature gets +4/+0; or Double Strike; or "When this creature deals combat damage to a creature, exile that creature"; or t, unattach ~ from this creature: ~ deals 5 damage to target creature."
t: Equipped creature gains flying until end of turn.
Equip 3
Courtesy of Crepes
[OMC] Omerium's Collapse
That's about as sophisticated as we have right now. Scripts for set generation will push the envelope on that, however.
Also, I promise that I'll release a gpu-to-cpu conversion script soon. Sorry for not having it sooner, haha.
That requires a manually constructed BNF grammar. It's very fragile and difficult to extend. The RNN, meanwhile, is trained from scratch and can be altered by feeding it different inputs during training. Furthermore, the RNN also has the advantage of automatically inferring semantics, letting it make greater leaps in terms of the variety of the cards it creates (e.g. realizing that Eldrazi are in some ways synonymous with large Golem creatures). In short, the RNN removes most of the human engineering from the process and can deliver much better results.
For example, the only way to get the BNF-grammar-driven generator to produce cards with mechanics from a new set is to rewrite or extend the grammar. How do I do it? I simply show it the cards and let it study them.
EDIT: Oh, and priming is literally impossible. You can't really push it in any direction other than the select few that the algorithm wants to go in.
EDIT(2): If it sounds like I'm being harsh, I'm not. Churchill's work is very ingenious. The main thing that I'm doing differently is that I'm taking the human out of the equation (for the most part). He has to nurse his brainchild. Mine is an independent hunter.
I'm glad you've found it so interesting! I hope that this has been an entertaining prelude to the more serious issue of a looming unemployment crisis. But that's a discussion for another time, haha.
You're absolutely right. We call that curriculum learning. For example, to teach an RNN to understand Python code, it's best to start by teaching it arithmetic, then logic, and then to bring all that together in programs of increasing complexity.
The only problem here is that while we have infinitely many statements of math and logic to draw upon, we don't have many Magic cards. That being said, we have shown that it's possible to expand the corpus with feeder networks, and this could provide ample material for cirruculum-style learning. You bring up a good point.
---
So, I concluded training of the latest version of the network with rarity added in and field shuffling disabled (for now). I took some samples and the results look decent (with respect to rarity), but I'll need to do some analysis to see how well it gets the concept. For example, there's clearly a difference between rare cards and common cards, but does the network really understand the role of uncommons? That's something that statistics can help us to determine. When I have the opportunity, that is. At this point I'm spending a lot of time working on my dissertation, writing about the mathematics underpinning our Magic-generating RNNs. Truth be told, the experiments that we've been doing here with Magic have actually been immensely beneficial for my PhD research.
Anyway, we've settled the question of whether we can create content that is complex and beautiful. We've determined that is possible. The question that remains is whether or not we can create fun experiences for players. Beyond all the technical challenges, that's the real goal.
I'm eager to do some playtesting in the near future. That's been my goal since day one. But here's the catch: neither the machine nor I know anything about set design. If anyone can recommend a set skeleton of some kind, or a way of conjuring one, I can take that skeleton and write scripts that take the output from the network (possibly after priming it) and select cards automatically to fill certain roles in the set.
Like I said before, we have numerous tools at our disposal to express a set skeleton programatically:
* Since we can evaluate cards based on their semantic content using content vectors, it's possible to specify rules like "I need a pump spell like Giant Growth".
* We can do direct queries like "I need a white 5-drop creature at common with high toughness."
* We can prime the network to generate output that fills certain needs and then selects among the results. This means we can ask for stuff like "give me a Rat lord and cards that care about Rats" or "give me a metalcraft theme in red and white."
* We can take cards that the network produces and colorshift our way to cycles (e.g turning one dual-land into five).
* We can be conservative with our selections, like skipping cards with undefined X's.
It's also possible for us to generate flavor and artwork for the cards (as you have seen before), but I'll set that aside for now and return to it later. What we'll do is have a set and then run the set through scripts that call the flavor-generating/art-generating networks to clothe the cards prior to their use.
So if anyone has any suggestions, I'm all ears.
---
A few examples from the latest network. I'm noticing that the rarity seems appropriate more often than not. I wanted to showcase three at each rarity, let me know what you think.
Heat Shadow
4U
Instant (Common)
Put target artifact, creature, or land on top of its owner's library.
Spirestorm Shaman
2G
Creature - Elf Archer (Common)
Forestwalk
2/2
Spirit Marsh
Land (Common)
Spirit Marsh enters the battlefield tapped.
When Spirit Marsh enters the battlefield, sacrifice it unless you return an untapped creature you control to its owner's hand.
T: Add RG to your mana pool.
Death's Herald
1BB
Creature - Zombie (Uncommon)
Whenever an opponent casts a white spell, put a 1/1 green saproling creature token onto the battlefield.
2/2
Time Control
2U
Instant (Uncommon)
Counter target spell unless its controller pays 4.
Brood Force
2G
Creature - Insect (Uncommon)
Flash
When Brood Force enters the battlefield, put a +1/+1 counter on target creature.
2/2
Stoneshaper Bow
4
Artifact (Rare)
5, T: Choose one:
* Destroy target nonblack creature. It can't be regenerated.
* Put three 1/1 white Kithkin creature tokens onto the battlefield.
* You gain one life.
* Destroy target artifact or enchantment.
Monstrous Charm
2R
Instant (Rare)
Monstrous Charm deals 4 damage to target creature or player.
After this main phase, there is an additional combat phase. Shuffle Monstrous Charm into its owner's library.
Keeper of the Thorns
3U
Creature - Human Wizard (Rare)
1U, T: Untap all creatures you control.
1/1
Memory Conscience
2WW
Enchantment (Mythic)
Whenever a nontoken creature enters the battlefield under your control, you may pay 1. if you do, put a 3/3 green Beast creature token onto the battlefield.
Battle-caller of the Horde
6BBB
Creature - Demon (Mythic)
Flying
1B, sacrifice a creature: Battle-caller of the Horde deals damage equal to its power to target creature or player.
8/8
Thornwind Angel
4WW
Creature - Angel (Mythic)
Flying
Whenever Thornwind Angel deals combat damage to a player, you may search your library for an artifact card with converted mana cost 5 or less and put it onto the battlefield. Then shuffle your library.
5/5
So it looks like the rarity thing is working pretty well. Not perfect, but it's decent. Next are a few unusual cards that I wanted to bring to your attention. I've attached flavor text for each of them (I primed the first word in each sentence and let it go from there).
Wall of Conschiet
W
Creature - Human Soldier (Common)
Whenever Wall of Conschiet is dealt damage, you may put that many 2/2 green Bear tokens onto the battlefield.
Bears sweatte for their wives' soil.
1/1
Trained Agent
3R
Creature - Goblin Warrior (Rare)
At the beginning of each upkeep, if no spells were cast last turn, transform Trained Agent.
Goblins are cautioned ofter seeking to itself, they can't be eatenable to serve the beasts.
3/2
///
Savage Dragon
Creature - Dragon
Flying
When Savage Dragon enters the battlefield, you gain 2 life.
Dragons emerge from the moment wizards say that boggart gone. His home.
6/6
---
Oh, and one more thing. Questions were raised earlier about the possibility of generating D&D spells. I looked into that and I feel like the neural architecture we're using is less than ideal for the task. The problem is that they exhibit micro-narrative structures for which our network has difficulty keeping track of. For example, we want:
This spell lets you breathe underwater.
-> You can swim under the ocean.
-> Breathing returns to normal after 1d8 minutes.
And our network is likely to give us outputs like:
This spell lets you breathe underwater.
-> Fish live underwater.
-> You can command the fish for 1d6 rounds.
Solving this sort of problem is challenging given the current state-of-the-art. And by that I mean we'll probably figure it out within the next few months. That's how fast all this is moving.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
It occurs to me that with the time travel/alternate reality storyline in Tarkir, it might be interesting to take the concept to Dominaria and create a war between the various 'Walkers, all trying to edit the history of the plane to their own advantage. Since it's sort of the prime plane/hub of the multiverse, changes to Dominaria's timeline have a ripple effect throughout the other planes as well - for instance, killing Karn during the Phyrexian invasion would prevent Mirrodin from ever existing, and thus no New Phyrexia. Or intervening in the Brothers' War prevents the Invasion from taking place at all, since Yawgmoth never captures Mishra. Etc.
And the storyline could conclude with a big ol' reset button so it will never be obsoleted by the official continuity.
Edit: And as far as the flavor... You know how Coldsnap took the storyline, themes, and mechanics of the Ice Age block and updated them to a Modern design standard? What if the RNN-generated set did the same with multiple older sets? Arabian Nights, Antiquities, Legends, The Dark, Fallen Empires, Homelands, Ice Age, Mirage, Tempest, Mercadian Masques, Invasion, Onslaught... All the pre-Modern sets, given the Coldsnap treatment.
We'd have to come up with some mechanics, and I'm thinking that rather than have the RNN generate names and flavor text completely at random, we'd have someone write bits of flavor text and let the RNN decide which cards to stick them on. And with names, we could write a list of words that the RNN should look at x% of the time when naming a card, with stuff like "Krovikan", "Thallid", and "of Ith", which it would then prime the card name with either as a prefix or suffix.
I was a big lore buff back in the olden days of MtG, so I could probably help in creating both.