I originally tried using utf-8 character output, which didn't work well with the RNN (it doesn't use utf-8). Apparently it isn't trivial to change in Lua either, so I had to rewrite my scripts a bit.
The decoder isn't working properly right now, but it gets things "mostly right". I added a comment above the culprit function. I just haven't had the time to fix it yet; I know it is going to be one of those "duh" moments when I figure out why it's messing up.
EDIT: I would be a little sad if the encoding scheme prevented the invention of new keywords and creature types, but I understand why it might be desirable. What about names though? Would there be a way to ensure that names were still generated on a character by character basis? I guess by just not encoding them?
My thought exactly. We should probably only encode the commonly used words and phrases, that sort of thing, rather than a comprehensive encoding. Not sure which is being used here though. I'm sure Andrew can explain.
I posted the code now. Short answer though: I'm encoding everything. I thought it would be interesting to see the results using a hammer before I tried out the scalpel.
EDIT: I would be a little sad if the encoding scheme prevented the invention of new keywords and creature types, but I understand why it might be desirable. What about names though? Would there be a way to ensure that names were still generated on a character by character basis? I guess by just not encoding them?
My thought exactly. We should probably only encode the commonly used words and phrases, that sort of thing, rather than a comprehensive encoding. Not sure which is being used here though. I'm sure Andrew can explain.
I posted the code now. Short answer though: I'm encoding everything. I thought it would be interesting to see the results using a hammer before I tried out the scalpel.
Of course, you're right. Good thinking. Thanks for posting everything. I'll be sure to take a look at everything this evening.
Just discovered this thread and I've read through it all. I've been interested in Machine Learning and have built my own fair share of Neural Networks myself over the past 2 years. I'm very impressed so far and I want to pat everyone on the back for contributing to such a neat project. I don't have the time to do this myself unfortunately, but I'm happy to maybe contribute some ideas based on my experiences. First thing's first, why hasn't anyone tried doing each Type separately? I feel like that would:
a.) Get much better results because each type just does very different things. If you do an "only creatures" run, you should always get P/T, never get equip costs, and probably produce more reasonable names, subtypes, effects, etc. Also, lands won't have these overpowered abilities I've been seeing that probably came from spells. Spells won't have Flying. I think it will train much quicker and better.
b.) Give more diversity. Instead of seeing mostly creatures and occasional other types, you'll have the same amount of creatures, lands, artifacts, etc. If you want to draft, you can handpick the best cards of each type and proportion the types appropriately for a set.
Just discovered this thread and I've read through it all. I've been interested in Machine Learning and have built my own fair share of Neural Networks myself over the past 2 years. I'm very impressed so far and I want to pat everyone on the back for contributing to such a neat project. I don't have the time to do this myself unfortunately, but I'm happy to maybe contribute some ideas based on my experiences. First thing's first, why hasn't anyone tried doing each Type separately? I feel like that would:
a.) Get much better results because each type just does very different things. If you do an "only creatures" run, you should always get P/T, never get equip costs, and probably produce more reasonable names, subtypes, effects, etc. Also, lands won't have these overpowered abilities I've been seeing that probably came from spells. Spells won't have Flying. I think it will train much quicker and better.
b.) Give more diversity. Instead of seeing mostly creatures and occasional other types, you'll have the same amount of creatures, lands, artifacts, etc. If you want to draft, you can handpick the best cards of each type and proportion the types appropriately for a set.
Keep up the great work
I'm not sure a) is really that noticeable. I've very infrequently gotten creatures without subtypes or P/T or non-creatures with them. It might affect the lands thing, sure, but in theory the network, if complex enough, should be able to divide these things itself.
As for b), about half the results are creatures, and a cube is generally ~50% creatures anyways.
Private Mod Note
():
Rollback Post to RevisionRollBack
My Moderator Helpdesk
Currently Playing:
Legacy: Something U/W Controlish EDH Cube
Hypercube! A New EDH Deck Every Week(ish)!
Just discovered this thread and I've read through it all. I've been interested in Machine Learning and have built my own fair share of Neural Networks myself over the past 2 years. I'm very impressed so far and I want to pat everyone on the back for contributing to such a neat project. I don't have the time to do this myself unfortunately, but I'm happy to maybe contribute some ideas based on my experiences. First thing's first, why hasn't anyone tried doing each Type separately? I feel like that would:
a.) Get much better results because each type just does very different things. If you do an "only creatures" run, you should always get P/T, never get equip costs, and probably produce more reasonable names, subtypes, effects, etc. Also, lands won't have these overpowered abilities I've been seeing that probably came from spells. Spells won't have Flying. I think it will train much quicker and better.
b.) Give more diversity. Instead of seeing mostly creatures and occasional other types, you'll have the same amount of creatures, lands, artifacts, etc. If you want to draft, you can handpick the best cards of each type and proportion the types appropriately for a set.
Keep up the great work
I mentioned earlier that partitioning can work very well, but with the right training scheme and preprocessing of the data, and so forth, I have not seen a need for it. The creatures I generate in my runs always have power and toughness, my sorceries and instants (almost) never have flying, etc. Part of the reasoning is that I may want to see a creature with an ability inspired by an ability seen on an artifact. Much of the card text seen on one type can be seen on others, and I didn't think it wise to start by breaking the input into small subsets. That was my reasoning, anyway.
EDIT: I think deficiencies in the output can be resolved by generating simple inputs to drive key points home and then moving on to training with the corpus. That would ensure that equipment cards have equip abilities, etc. I've seen curriculum learning do wonderful things in the past for other problems.
EDIT (2): Specifically, I was most recently inspired by the recent success of such an approach in Zaremba and Sutskever 2015. Deep LSTM network, complex inputs, and so-so results made much better with such a training regime.
EDIT (3): Sorry for all the edits. Anyway, last thing is that large, self-partitioning networks can deliver state-of-the-art results without the need for manual partitioning. Consider what we've seen with things like image classification tasks (where you have many, many mutually exclusive categories of inputs). Sure, the manual partitioning can work, but we've seen such an explosion in GPU power that I'm not sure the savings are even worth it. Correct me if I'm wrong though.
Calestare Scrikes - 1W
Enchantment
At the beginning of your upkeep, you may put a +1/+1 counter on target creature you control.
You know what's weird? As far as I can tell, no card with this effect actually exists. I feel like I must have made a mistake Googling it, because it seems like a super-obvious line of text, but I got nothing. Considering Honor of the Pure, it might even be costed correctly. What the heck??
Here's a weird idea, but might removing card names from the input file be a good idea? We've pretty much established that the RNN is incapable of coming up with decent card names anyway. So it might lead to better results the same RNN size, because there's less information to process.
Any thoughts from those of you who actually know what they're talking about?
Haha, you're correct about your assumptions and I don't think I can improve upon your analysis. However, I will say that having the names makes the cards seem more real, and it gives us a way of talking about them as we generate them (as opposed to "card #221323 looks nice"). There's something magical about having the name, even if the name is gibberish. Also, you have to admit that "Slidshocking Krow" is a very interesting name. That and every so often it makes a connection between name and function, like "Akroma, Legocy Guildmage" being a legendary creature with two colors.
While I agree that it's not necessary, I maintain that manually partitioning will get better results faster. The only downside is what you said, that creatures won't have abilities inspired by other types. Perhaps I'll give it a try myself in about a month after I've moved and settled in.
Here are a few cards I liked from skimming the end of MagicDad's results:
Tolarian King 3UU
Legendary Creature — Human Wizard
Mythic Rare
$THIS doesn't untap during your untap step.
At the beginning of your upkeep, you may gain control of target land or s creature with power 2 or less. If a creature dealt damage this way would die this turn, exile it instead.
3/2
# Tolarian King is a blue Legendary Human Wizard mythic rare that gains control of things? Impressive!
Flockwater Well 2UU
Creature — Faerie Wizard
Uncommon
Flying
Whenever $THIS deals combat damage to a player, you may pay 3BR. If you do, put a +1/+1 counter on each of onto the battlefield.
Prevent all combat damage that would be dealt to $THIS by artifact lands. 5GG: Until end of turn, $THIS becomes a Human Nomad Artificer with Devourer 1/7.
2/2
# Wow, flavor-splosion! I like that it has Devourer 1/7. Devour 7 things, get one +1/+1 counter. Ignore the fact that there's no way for the Devour to trigger.
Dream Illusion UU
Enchantment — Aura
Uncommon
Enchant creature
Enchanted creature can't be blocked.
When enchanted creature becomes the target of a spell or ability, sacrifice $THIS.
When $THIS leaves the battlefield, draw a card.
# Wow. This is incredibly on-point.
Vault of the Wild 1GG
Enchantment
Rare 1R, Sacrifice a creature: Draw a card.
Abbey Warrior 3B
Creature — Sliver
Uncommon
All Slivers have "{2}: This creature gets +0/+1 until end of turn."
Whenever Reveal a card in play, you win the game.
4/3
# Seemed great besides the name... then the sudden win the game clause
Warden of the Onubol 3
Artifact Creature — Horse
Mythic Rare
$THIS's power and toughness are each equal to the number of nonbasic lands that player controls.
When $THIS enters the battlefield, if you have 20 or less life, you may search your library for a card named $THIS, reveal that card, put it into your hand, then shuffle your library. 1/4
Warrior of Claws 2BR
Creature — Hag
Uncommon
Morph 2WU (You may cast this card face down as a 2/2 creature for {3}. Turn it face up any time for its morph cost.)
When $THIS is turned face up, each opponent guts a winct in the card's name into his or her library.
3/4
# Guts a winct? *Shudders*
Duckcaster Goldmage (R/W)
Creature — Human Warrior
Common
Whenever you cast a Spirit or Arcane spell, $THIS gets +3/+3 until end of turn. G: $THIS gets +1/+0 until end of turn.
2/3
# I guess Spirit and Arcane spells are ducks? Also this has grass-breathing! Chase common right here.
Goblin Sorcerer R
Creature — Goblin Shaman
Common
{T}, Sacrifice $THIS: Add RR to your mana pool.
1/1
# Is this already a card? If not, it should be.
Yawgmoth's Soul 4B
Sorcery
Uncommon
Search your library for up to two basic land cards, reveal those cards, and put one onto the battlefield tapped and the other into your hand. Then shuffle your library.
# You guys, all Yawgmoth really wants to do is Cultivate.
Guerrilla Protector (R/W)
Creature — Insect
Common
Flying R, {T}: Regenerate target creature.
1/1
# Move over Mother of Runes! I like that it has Protector in the name.
Yeah, for purposes of this thread (or the Twitter bot), the names are great. But if somebody wants to create a "RoboRosewater cube" or something, they will probably need to make up names themselves anyway. Pronouncing some of the RNN's tongue twisters might hurt.
It's definitely an idea with some merit.
Private Mod Note
():
Rollback Post to RevisionRollBack
My Moderator Helpdesk
Currently Playing:
Legacy: Something U/W Controlish EDH Cube
Hypercube! A New EDH Deck Every Week(ish)!
This card should probably have some more unrelated lines of rules text added to it. Also, once again with the paying X to no effect. I think it's really strange that so far none of the iterations seems to have learned that every card with one X on it needs to have at least one more X on it.
Undefined X is another drink in the RNN drinking game.
Dream Illusion UU
Enchantment — Aura
Uncommon
Enchant creature
Enchanted creature can't be blocked.
When enchanted creature becomes the target of a spell or ability, sacrifice $THIS.
When $THIS leaves the battlefield, draw a card.
This thing is blowing my mind. It's a 100% blue effect, it has "Illusion" in the name and has the illusion effect of Illusionary Armor, it replaces itself so it doesn't have the Aura-card-disadvantage issue or penalize you too harshly when your opponent removes it immediately, and in my judgment it has an appropriate cost and rarity. If the name were a little different, I would totally believe this was a spoiled card from Magic Origins. If the system starts producing things like this consistently, maybe Mark Rosewater's job is in jeopardy.
Oh, and a question - is this sort of AI capable of learning higher-level pattern matching, like replicate costs always matching the card's mana cost even if that cost isn't something that's shown up on a replicate card, or if it decides to give something advisorwalk it should add reminder text of "(This creature can't be blocked as long as defending player controls an Advisor.)"? Can it extrapolate structures, or only assemble things out of literal patterns it's already seen?
Undefined X is another drink in the RNN drinking game.
Drink whenever:
You get an ability missing half of its effect, like equipment without an equip cost, kicker without an "if it was kicked" clause, X payment without the X defined, etc
You get a rambling, incoherent line
You get P/T or an ability that makes zero sense on an Instant/Sorcery (like "prevent all damage that would be dealt to $THIS", or an activated ability that gives flying)
You get no P/T on a creature
You get a mismatched reminder text
Private Mod Note
():
Rollback Post to RevisionRollBack
My Moderator Helpdesk
Currently Playing:
Legacy: Something U/W Controlish EDH Cube
Hypercube! A New EDH Deck Every Week(ish)!
And the old standby: "and then shuffle your library" when you haven't searched your library. Drink twice if you shuffle your library after putting something on the top or bottom of it.
A classic part of the drinking game (and something that shows the merit of splitting up the training group into card types for at least a portion of the training) is:
Creatures with abilities like "Target artifact gains shroud" with no ETB trigger.
AND
Instants with abilities like "When $THIS enters the battlefield put a +1/+1 counter on it."
I do agree that we can't afford to eliminate too much of the training sample, and that we want to empower the NN to cross-reference spell abilities when making creatures and vice versa. But so far, it seems very bad at learning that the abilities take different different forms and syntax on different types. At the very least, I think that splitting things up into spells, creatures, and non-creature permanents might give some interesting results, especially if we can fit the NN for the three separately and then somehow combine the training into a single supernet with initially weak connections between the three sections. It's been a while since I studied machine learning though.
I'm not sure a) is really that noticeable. I've very infrequently gotten creatures without subtypes or P/T or non-creatures with them. It might affect the lands thing, sure, but in theory the network, if complex enough, should be able to divide these things itself.
As for b), about half the results are creatures, and a cube is generally ~50% creatures anyways.
I think he has a good idea there, actually. Temperature really skewed things towards creatures for me, and very few of the lands that were not practically basics really worked.
Undefined X is another drink in the RNN drinking game.
Drink whenever:
You get an ability missing half of its effect, like equipment without an equip cost, kicker without an "if it was kicked" clause, X payment without the X defined, etc
You get a rambling, incoherent line
You get P/T or an ability that makes zero sense on an Instant/Sorcery (like "prevent all damage that would be dealt to $THIS", or an activated ability that gives flying)
You get no P/T on a creature
You get a mismatched reminder text
A classic part of the drinking game (and something that shows the merit of splitting up the training group into card types for at least a portion of the training) is:
Creatures with abilities like "Target artifact gains shroud" with no ETB trigger.
AND
Instants with abilities like "When $THIS enters the battlefield put a +1/+1 counter on it."
I do agree that we can't afford to eliminate too much of the training sample, and that we want to empower the NN to cross-reference spell abilities when making creatures and vice versa. But so far, it seems very bad at learning that the abilities take different different forms and syntax on different types. At the very least, I think that splitting things up into spells, creatures, and non-creature permanents might give some interesting results, especially if we can fit the NN for the three separately and then somehow combine the training into a single supernet with initially weak connections between the three sections. It's been a while since I studied machine learning though.
A classic part of the drinking game (and something that shows the merit of splitting up the training group into card types for at least a portion of the training) is:
Creatures with abilities like "Target artifact gains shroud" with no ETB trigger.
AND
Instants with abilities like "When $THIS enters the battlefield put a +1/+1 counter on it."
I do agree that we can't afford to eliminate too much of the training sample, and that we want to empower the NN to cross-reference spell abilities when making creatures and vice versa. But so far, it seems very bad at learning that the abilities take different different forms and syntax on different types. At the very least, I think that splitting things up into spells, creatures, and non-creature permanents might give some interesting results, especially if we can fit the NN for the three separately and then somehow combine the training into a single supernet with initially weak connections between the three sections. It's been a while since I studied machine learning though.
I think it's worth experimenting with multiple approaches, never know what we might get.
Once I get everything set up properly, my plan is to continue with the architecture we have and generate cookie-cutter inputs that thoroughly teach the basics to the network before introducing it to the real cards. I agree that part of the issue is that we are trying to get one network to serve many purposes, which is possible but also makes the training process harder. At the same time, I feel like we can get more robust and interesting inputs if we have some kind of "cross-chatter" between the different input domains. So a half-way solution like what you're suggesting may be worth considering if it can give us the best of both worlds. I guess we'll just have to try things and see. :-D
I've been watching this thread for quite awhile and I'm thrilled at the progress and samples that are being generated. I'm interested in generating a bunch of samples myself (I have a pretty powerful rig that could pull it off) but must admit I only have basic c++ knowledge. Eventually, could this project set a goal to have some kind of downloadable compilation of files/guide to running the generator? I'm over-simplifying it but basically something like "download this, open it, set system resource allocation, then hit GO".
If this isn't something too distracting, I think it would be worth editing into the first post as I'm sure there are other people interested that could use a guide with the setting up process.
I've been watching this thread for quite awhile and I'm thrilled at the progress and samples that are being generated. I'm interested in generating a bunch of samples myself (I have a pretty powerful rig that could pull it off) but must admit I only have basic c++ knowledge. Eventually, could this project set a goal to have some kind of downloadable compilation of files/guide to running the generator? I'm over-simplifying it but basically something like "download this, open it, set system resource allocation, then hit GO".
If this isn't something too distracting, I think it would be worth editing into the first post as I'm sure there are other people interested that could use a guide with the setting up process.
Someone posted this a few pages back, I believe.
Basically:
1) http://www.psychocats.net/ubuntu/virtualbox - Follow these directions to set up VirtualBox with 64-bit Ubuntu. Allocate at least 2GB RAM and as much Hard Drive Space as you can spare. (Skip this step if you're already in Ubuntu or OSX)
2) http://torch.ch/docs/getting-started.html#_ - Follow these directions in your terminal to download and install torch
3) In a terminal, type "luarocks install nngraph" and "luarocks install optim" to install the lua modules rnn needs
4) https://github.com/karpathy/char-rnn - Download RNN from here
5) https://drive.google.com/file/d/0B2S21COYh9P8M1hEeFYwUDZOMWM/view?usp=sharing - Download the dataset and put it into the data folder in the RNN package you downloaded
6) In a terminal, "cd /<path to where you downloaded RNN>"
7) In the same terminal, "th train.lua -data_dir data/<the folder you put the data in> -gpuid -1 -rnn_size <size> -num_layers <layers> -dropout <dropout>" to start training. This will start creating checkpoint files in the cv folder
8) In a new terminal, at any point, "th sample.lua cv/<checkpoint file name> -gpuid -1 -length <length> -temperature <temp>" to generate cards. You'll have to reformat the output to be more usable, but that's it in a nutshell.
As for what parameters to use
rnn_size - The default is 200 I believe. Talcos ran at 600. I'm kicking off a run at 1000 soon. This will depend on your hardware and how long you're willing to wait
num_layers - Defaults to 2, Talcos ran at 3, I'm going for 6. Like rnn_size will depend on your hardware and patience
dropout - I've tried .25, Talcos ran at .5. .5 is probably fine. Too much lower and you'll "overfit"
length - Default is 2000 characters, which is a good 10 cards. Increase this for more
temperature - how "risky" the created cards will be. Play around with this, anyhwere from .2 to .8 to see different results.
Private Mod Note
():
Rollback Post to RevisionRollBack
My Moderator Helpdesk
Currently Playing:
Legacy: Something U/W Controlish EDH Cube
Hypercube! A New EDH Deck Every Week(ish)!
I've been watching this thread for quite awhile and I'm thrilled at the progress and samples that are being generated. I'm interested in generating a bunch of samples myself (I have a pretty powerful rig that could pull it off) but must admit I only have basic c++ knowledge. Eventually, could this project set a goal to have some kind of downloadable compilation of files/guide to running the generator? I'm over-simplifying it but basically something like "download this, open it, set system resource allocation, then hit GO".
If this isn't something too distracting, I think it would be worth editing into the first post as I'm sure there are other people interested that could use a guide with the setting up process.
Someone posted this a few pages back, I believe.
Basically:
1) http://www.psychocats.net/ubuntu/virtualbox - Follow these directions to set up VirtualBox with 64-bit Ubuntu. Allocate at least 2GB RAM and as much Hard Drive Space as you can spare. (Skip this step if you're already in Ubuntu or OSX)
2) http://torch.ch/docs/getting-started.html#_ - Follow these directions in your terminal to download and install torch
3) In a terminal, type "luarocks install nngraph" and "luarocks install optim" to install the lua modules rnn needs
4) https://github.com/karpathy/char-rnn - Download RNN from here
5) https://drive.google.com/file/d/0B2S21COYh9P8M1hEeFYwUDZOMWM/view?usp=sharing - Download the dataset and put it into the data folder in the RNN package you downloaded
6) In a terminal, "cd /<path to where you downloaded RNN>"
7) In the same terminal, "th train.lua -data_dir data/<the folder you put the data in> -gpuid -1 -rnn_size <size> -num_layers <layers> -dropout <dropout>" to start training. This will start creating checkpoint files in the cv folder
8) In a new terminal, at any point, "th sample.lua cv/<checkpoint file name> -gpuid -1 -length <length> -temperature <temp>" to generate cards. You'll have to reformat the output to be more usable, but that's it in a nutshell.
As for what parameters to use
rnn_size - The default is 200 I believe. Talcos ran at 600. I'm kicking off a run at 1000 soon. This will depend on your hardware and how long you're willing to wait
num_layers - Defaults to 2, Talcos ran at 3, I'm going for 6. Like rnn_size will depend on your hardware and patience
dropout - I've tried .25, Talcos ran at .5. .5 is probably fine. Too much lower and you'll "overfit"
length - Default is 2000 characters, which is a good 10 cards. Increase this for more
temperature - how "risky" the created cards will be. Play around with this, anyhwere from .2 to .8 to see different results.
If I get super ambitious, I might write up a more detailed guide. I suppose it might also be possible to look into creating a packaged installer/script for a bunch of it that takes a lot of the manual steps out, but that's a bigger project that I'm prepared to take on this moment.
I appreciate the heads up, I must have missed the steps on the previous pages. I'll toy around with it over the next few days.
Edit: Followed your steps and kept receiving an error in terminal when trying to run the train command. Backtracked my steps a bit and found out that Torch wasn't installing correctly due to iPython being wrong version. Grabbed that but still getting a specific error. Idk, I'm a total linux n00b here, so if anybody wants to send a PM to me and let me know what I have wrong here, I won't derail this thread.
Get following info after entering "th train.lua -data_dir data/home/lasture/Desktop/char-rnn-master/data -gpuid -1 -rnn_size 600 -num_layers 3 -dropout .5"
/home/lasture/torch/install/bin/luajit: cannot open <data/home/lasture/Desktop/char-rnn-master/data/input.txt> in mode r at /home/lasture/torch/pkg/torch/lib/TH/THDiskFile.c:484
stack traceback:
[C]: at 0x7f63e017baa0
[C]: in function 'DiskFile'
./util/CharSplitLMMinibatchLoader.lua:130: in function 'text_to_tensor'
./util/CharSplitLMMinibatchLoader.lua:38: in function 'create'
train.lua:89: in main chunk
[C]: in function 'dofile'
...ture/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:131: in main chunk
[C]: at 0x00406670
I notice in the data file Wildfire linked, X is surrounded by curly braces when it's part of a cost, but in other contexts it's simply the letter by itself. I'm wondering if the RNN would grasp how X effects work faster if X was encoded the same way in all contexts. Or should a sufficiently-trained RNN be able to grasp the relationship between "{X}" and "X" without the help?
I started a new run this morning with a new dataset. I split out the mana cost as someone suggested, stripped out reminder text, and tweaked the order of fields a bit again. I also have a much larger set this time. Basically Beta + everything from Mirage onwards, or something like that anyway. I cut some stuff, non-standard layout, un-sets, Sol Ring, Ancestral Recall and the original Moxen, etc.
Settings:
-rnn_size 300 -layers 3 -seq_length 100
Example card from my dataset:
n:Tibor and Lumia|c:[Blue,Red]|m:{1}{1}{U}{R}|cmc:4|t:[Legendary Creature]|p:3/3|s:[Human Wizard]|r:Whenever you cast a blue spell, target creature gains flying until end of turn.\nWhenever you cast a red spell, $THIS deals 1 damage to each creature without flying.
On my way to work I had a flash of insight, the mana cost really should be {U}{R}{1}{1} if we want to make it easier to learn the colors based on the mana cost.
https://github.com/aarnott50/mtg-rnn-encoder
I originally tried using utf-8 character output, which didn't work well with the RNN (it doesn't use utf-8). Apparently it isn't trivial to change in Lua either, so I had to rewrite my scripts a bit.
The decoder isn't working properly right now, but it gets things "mostly right". I added a comment above the culprit function. I just haven't had the time to fix it yet; I know it is going to be one of those "duh" moments when I figure out why it's messing up.
I posted the code now. Short answer though: I'm encoding everything. I thought it would be interesting to see the results using a hammer before I tried out the scalpel.
Of course, you're right. Good thinking. Thanks for posting everything. I'll be sure to take a look at everything this evening.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
a.) Get much better results because each type just does very different things. If you do an "only creatures" run, you should always get P/T, never get equip costs, and probably produce more reasonable names, subtypes, effects, etc. Also, lands won't have these overpowered abilities I've been seeing that probably came from spells. Spells won't have Flying. I think it will train much quicker and better.
b.) Give more diversity. Instead of seeing mostly creatures and occasional other types, you'll have the same amount of creatures, lands, artifacts, etc. If you want to draft, you can handpick the best cards of each type and proportion the types appropriately for a set.
Keep up the great work
cEDH: [G(U/R) Animar] - [(U/B)(G/W) Redless Wheels] - [(G/U)(W/B) Redless Pod] - [(B/G)W Ghave Metapod]
Over 20% of artifacts cost 3 mana. So this isn't super surprising.
I'm not sure a) is really that noticeable. I've very infrequently gotten creatures without subtypes or P/T or non-creatures with them. It might affect the lands thing, sure, but in theory the network, if complex enough, should be able to divide these things itself.
As for b), about half the results are creatures, and a cube is generally ~50% creatures anyways.
Currently Playing:
Legacy: Something U/W Controlish
EDH Cube
Hypercube! A New EDH Deck Every Week(ish)!
I mentioned earlier that partitioning can work very well, but with the right training scheme and preprocessing of the data, and so forth, I have not seen a need for it. The creatures I generate in my runs always have power and toughness, my sorceries and instants (almost) never have flying, etc. Part of the reasoning is that I may want to see a creature with an ability inspired by an ability seen on an artifact. Much of the card text seen on one type can be seen on others, and I didn't think it wise to start by breaking the input into small subsets. That was my reasoning, anyway.
EDIT: I think deficiencies in the output can be resolved by generating simple inputs to drive key points home and then moving on to training with the corpus. That would ensure that equipment cards have equip abilities, etc. I've seen curriculum learning do wonderful things in the past for other problems.
EDIT (2): Specifically, I was most recently inspired by the recent success of such an approach in Zaremba and Sutskever 2015. Deep LSTM network, complex inputs, and so-so results made much better with such a training regime.
EDIT (3): Sorry for all the edits. Anyway, last thing is that large, self-partitioning networks can deliver state-of-the-art results without the need for manual partitioning. Consider what we've seen with things like image classification tasks (where you have many, many mutually exclusive categories of inputs). Sure, the manual partitioning can work, but we've seen such an explosion in GPU power that I'm not sure the savings are even worth it. Correct me if I'm wrong though.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
You know what's weird? As far as I can tell, no card with this effect actually exists. I feel like I must have made a mistake Googling it, because it seems like a super-obvious line of text, but I got nothing. Considering Honor of the Pure, it might even be costed correctly. What the heck??
Haha, you're correct about your assumptions and I don't think I can improve upon your analysis. However, I will say that having the names makes the cards seem more real, and it gives us a way of talking about them as we generate them (as opposed to "card #221323 looks nice"). There's something magical about having the name, even if the name is gibberish. Also, you have to admit that "Slidshocking Krow" is a very interesting name. That and every so often it makes a connection between name and function, like "Akroma, Legocy Guildmage" being a legendary creature with two colors.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
Here are a few cards I liked from skimming the end of MagicDad's results:
Tolarian King 3UU
Legendary Creature — Human Wizard
Mythic Rare
$THIS doesn't untap during your untap step.
At the beginning of your upkeep, you may gain control of target land or s creature with power 2 or less. If a creature dealt damage this way would die this turn, exile it instead.
3/2
# Tolarian King is a blue Legendary Human Wizard mythic rare that gains control of things? Impressive!
Flockwater Well 2UU
Creature — Faerie Wizard
Uncommon
Flying
Whenever $THIS deals combat damage to a player, you may pay 3BR. If you do, put a +1/+1 counter on each of onto the battlefield.
Prevent all combat damage that would be dealt to $THIS by artifact lands.
5GG: Until end of turn, $THIS becomes a Human Nomad Artificer with Devourer 1/7.
2/2
# Wow, flavor-splosion! I like that it has Devourer 1/7. Devour 7 things, get one +1/+1 counter. Ignore the fact that there's no way for the Devour to trigger.
Dream Illusion UU
Enchantment — Aura
Uncommon
Enchant creature
Enchanted creature can't be blocked.
When enchanted creature becomes the target of a spell or ability, sacrifice $THIS.
When $THIS leaves the battlefield, draw a card.
# Wow. This is incredibly on-point.
Vault of the Wild 1GG
Enchantment
Rare
1R, Sacrifice a creature: Draw a card.
Abbey Warrior 3B
Creature — Sliver
Uncommon
All Slivers have "{2}: This creature gets +0/+1 until end of turn."
Whenever Reveal a card in play, you win the game.
4/3
# Seemed great besides the name... then the sudden win the game clause
Warden of the Onubol 3
Artifact Creature — Horse
Mythic Rare
$THIS's power and toughness are each equal to the number of nonbasic lands that player controls.
When $THIS enters the battlefield, if you have 20 or less life, you may search your library for a card named $THIS, reveal that card, put it into your hand, then shuffle your library. 1/4
Warrior of Claws 2BR
Creature — Hag
Uncommon
Morph 2WU (You may cast this card face down as a 2/2 creature for {3}. Turn it face up any time for its morph cost.)
When $THIS is turned face up, each opponent guts a winct in the card's name into his or her library.
3/4
# Guts a winct? *Shudders*
Duckcaster Goldmage (R/W)
Creature — Human Warrior
Common
Whenever you cast a Spirit or Arcane spell, $THIS gets +3/+3 until end of turn.
G: $THIS gets +1/+0 until end of turn.
2/3
# I guess Spirit and Arcane spells are ducks? Also this has grass-breathing! Chase common right here.
Goblin Sorcerer R
Creature — Goblin Shaman
Common
{T}, Sacrifice $THIS: Add RR to your mana pool.
1/1
# Is this already a card? If not, it should be.
Yawgmoth's Soul 4B
Sorcery
Uncommon
Search your library for up to two basic land cards, reveal those cards, and put one onto the battlefield tapped and the other into your hand. Then shuffle your library.
# You guys, all Yawgmoth really wants to do is Cultivate.
Guerrilla Protector (R/W)
Creature — Insect
Common
Flying
R, {T}: Regenerate target creature.
1/1
# Move over Mother of Runes! I like that it has Protector in the name.
cEDH: [G(U/R) Animar] - [(U/B)(G/W) Redless Wheels] - [(G/U)(W/B) Redless Pod] - [(B/G)W Ghave Metapod]
It's definitely an idea with some merit.
Currently Playing:
Legacy: Something U/W Controlish
EDH Cube
Hypercube! A New EDH Deck Every Week(ish)!
Undefined X is another drink in the RNN drinking game.
This thing is blowing my mind. It's a 100% blue effect, it has "Illusion" in the name and has the illusion effect of Illusionary Armor, it replaces itself so it doesn't have the Aura-card-disadvantage issue or penalize you too harshly when your opponent removes it immediately, and in my judgment it has an appropriate cost and rarity. If the name were a little different, I would totally believe this was a spoiled card from Magic Origins. If the system starts producing things like this consistently, maybe Mark Rosewater's job is in jeopardy.
Oh, and a question - is this sort of AI capable of learning higher-level pattern matching, like replicate costs always matching the card's mana cost even if that cost isn't something that's shown up on a replicate card, or if it decides to give something advisorwalk it should add reminder text of "(This creature can't be blocked as long as defending player controls an Advisor.)"? Can it extrapolate structures, or only assemble things out of literal patterns it's already seen?
Drink whenever:
You get an ability missing half of its effect, like equipment without an equip cost, kicker without an "if it was kicked" clause, X payment without the X defined, etc
You get a rambling, incoherent line
You get P/T or an ability that makes zero sense on an Instant/Sorcery (like "prevent all damage that would be dealt to $THIS", or an activated ability that gives flying)
You get no P/T on a creature
You get a mismatched reminder text
Currently Playing:
Legacy: Something U/W Controlish
EDH Cube
Hypercube! A New EDH Deck Every Week(ish)!
Creatures with abilities like "Target artifact gains shroud" with no ETB trigger.
AND
Instants with abilities like "When $THIS enters the battlefield put a +1/+1 counter on it."
I do agree that we can't afford to eliminate too much of the training sample, and that we want to empower the NN to cross-reference spell abilities when making creatures and vice versa. But so far, it seems very bad at learning that the abilities take different different forms and syntax on different types. At the very least, I think that splitting things up into spells, creatures, and non-creature permanents might give some interesting results, especially if we can fit the NN for the three separately and then somehow combine the training into a single supernet with initially weak connections between the three sections. It's been a while since I studied machine learning though.
I think he has a good idea there, actually. Temperature really skewed things towards creatures for me, and very few of the lands that were not practically basics really worked.
Sounds like murder.
I like this guy's plan even more.
I think it's worth experimenting with multiple approaches, never know what we might get.
Once I get everything set up properly, my plan is to continue with the architecture we have and generate cookie-cutter inputs that thoroughly teach the basics to the network before introducing it to the real cards. I agree that part of the issue is that we are trying to get one network to serve many purposes, which is possible but also makes the training process harder. At the same time, I feel like we can get more robust and interesting inputs if we have some kind of "cross-chatter" between the different input domains. So a half-way solution like what you're suggesting may be worth considering if it can give us the best of both worlds. I guess we'll just have to try things and see. :-D
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
If this isn't something too distracting, I think it would be worth editing into the first post as I'm sure there are other people interested that could use a guide with the setting up process.
Someone posted this a few pages back, I believe.
Basically:
1) http://www.psychocats.net/ubuntu/virtualbox - Follow these directions to set up VirtualBox with 64-bit Ubuntu. Allocate at least 2GB RAM and as much Hard Drive Space as you can spare. (Skip this step if you're already in Ubuntu or OSX)
2) http://torch.ch/docs/getting-started.html#_ - Follow these directions in your terminal to download and install torch
3) In a terminal, type "luarocks install nngraph" and "luarocks install optim" to install the lua modules rnn needs
4) https://github.com/karpathy/char-rnn - Download RNN from here
5) https://drive.google.com/file/d/0B2S21COYh9P8M1hEeFYwUDZOMWM/view?usp=sharing - Download the dataset and put it into the data folder in the RNN package you downloaded
6) In a terminal, "cd /<path to where you downloaded RNN>"
7) In the same terminal, "th train.lua -data_dir data/<the folder you put the data in> -gpuid -1 -rnn_size <size> -num_layers <layers> -dropout <dropout>" to start training. This will start creating checkpoint files in the cv folder
8) In a new terminal, at any point, "th sample.lua cv/<checkpoint file name> -gpuid -1 -length <length> -temperature <temp>" to generate cards. You'll have to reformat the output to be more usable, but that's it in a nutshell.
As for what parameters to use
rnn_size - The default is 200 I believe. Talcos ran at 600. I'm kicking off a run at 1000 soon. This will depend on your hardware and how long you're willing to wait
num_layers - Defaults to 2, Talcos ran at 3, I'm going for 6. Like rnn_size will depend on your hardware and patience
dropout - I've tried .25, Talcos ran at .5. .5 is probably fine. Too much lower and you'll "overfit"
length - Default is 2000 characters, which is a good 10 cards. Increase this for more
temperature - how "risky" the created cards will be. Play around with this, anyhwere from .2 to .8 to see different results.
Currently Playing:
Legacy: Something U/W Controlish
EDH Cube
Hypercube! A New EDH Deck Every Week(ish)!
If I get super ambitious, I might write up a more detailed guide. I suppose it might also be possible to look into creating a packaged installer/script for a bunch of it that takes a lot of the manual steps out, but that's a bigger project that I'm prepared to take on this moment.
Edit: Followed your steps and kept receiving an error in terminal when trying to run the train command. Backtracked my steps a bit and found out that Torch wasn't installing correctly due to iPython being wrong version. Grabbed that but still getting a specific error. Idk, I'm a total linux n00b here, so if anybody wants to send a PM to me and let me know what I have wrong here, I won't derail this thread.
Get following info after entering "th train.lua -data_dir data/home/lasture/Desktop/char-rnn-master/data -gpuid -1 -rnn_size 600 -num_layers 3 -dropout .5"
/home/lasture/torch/install/bin/luajit: cannot open <data/home/lasture/Desktop/char-rnn-master/data/input.txt> in mode r at /home/lasture/torch/pkg/torch/lib/TH/THDiskFile.c:484
stack traceback:
[C]: at 0x7f63e017baa0
[C]: in function 'DiskFile'
./util/CharSplitLMMinibatchLoader.lua:130: in function 'text_to_tensor'
./util/CharSplitLMMinibatchLoader.lua:38: in function 'create'
train.lua:89: in main chunk
[C]: in function 'dofile'
...ture/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:131: in main chunk
[C]: at 0x00406670
Settings:
-rnn_size 300 -layers 3 -seq_length 100
Example card from my dataset:
n:Tibor and Lumia|c:[Blue,Red]|m:{1}{1}{U}{R}|cmc:4|t:[Legendary Creature]|p:3/3|s:[Human Wizard]|r:Whenever you cast a blue spell, target creature gains flying until end of turn.\nWhenever you cast a red spell, $THIS deals 1 damage to each creature without flying.
On my way to work I had a flash of insight, the mana cost really should be {U}{R}{1}{1} if we want to make it easier to learn the colors based on the mana cost.
Or are we just looking at the NN taking the same baby steps over and over?
( 0.0 )
=O ((U/R)) O=
(")(")
I'm an AI making Magic cards.
http://www.staalmedia.nl/nexus/#generate