Assuming you're in a command line terminal, use 'cd' to go into the folder that contains the script (train.lua) in this case. Then run the command from the command line terminal (th train.lua etc. etc.)
@Talcos; those are tremendously good flavour texts. Are you extracting those from the general card input file, or have you created an entirely separate input for flavour text?
Also, is your sample_hs_v2.lua script meant to work with hardcast's random field input (the one with field number tags)? I've attempted using it and it doesn't always produce correct results; telling it -manacost "{UURR}" gives a lot of results, only some of which are manacost UR.
As for the flavor text, I extracted all the flavor texts, threw delimiters around them, and fed them into the network. I wanted to see how well the network could do with just flavor text.
And no, sample_hs_v2.lua is not meant to work with the random format, but you can still prime with it if you use the "-name" option. What "-name" will do is start priming right after the first bar is hit, so you can start off a card like this: -name "4legendary|5creature|6elf warrior".
By the way, the network is still training (it's still improving over the epochs), but I'm already getting such awesome results, I have to share a few:
* "I will show you that the king yet reigns!"
* The sea is the resource of the past.
* They are lightning streams, spirits of the sphinxesply plain.
* "The Tajuru executed him." - Momir Vig
* "The clouds can be found, like and trained the clockwork in his own skin." - Zul Ashur, lich lord
* "I saw a book! We will never find them to the call. One so greed? But it's low, like shiny." - Narset
Sometimes it partially clones existing flavor texts and then changes them up:
* The most terrifying thing about the craw wurm is probably the horrible crashing sound it makes as it speeds through the forest, free of the evil from the consequent of Mirrodin's only conviction.
(compare to Craw Wurm)
Have you tried running a version that only learns from cards made starting from Lorwyn/Shadowmoor commons and uncommons to cut down on complexity, using current oracle text (so that it uses Menace properly)? Perhaps cutting out any cards that reference planeswalkers or legendary, and removing all legendary cards as well (if there are any at common or uncommon rarities)? That might let it learn about color and some of the common effects of basic card types somewhat more accurately and get less confused by the occasional complex and pie breaking cards that tend to turn up more often at higher rarities after more modern design shifts? Using Lorwyn/Shadowmoor as the cutoff point would help exclude things like Time Spiral block zaniness and pie breaking and such.
Or would that give it too little data to work off of?
Have you tried running a version that only learns from cards made starting from Lorwyn/Shadowmoor commons and uncommons to cut down on complexity, using current oracle text (so that it uses Menace properly)? Perhaps cutting out any cards that reference planeswalkers or legendary, and removing all legendary cards as well (if there are any at common or uncommon rarities)? That might let it learn about color and some of the common effects of basic card types somewhat more accurately and get less confused by the occasional complex and pie breaking cards that tend to turn up more often at higher rarities after more modern design shifts? Using Lorwyn/Shadowmoor as the cutoff point would help exclude things like Time Spiral block zaniness and pie breaking and such.
Or would that give it too little data to work off of?
That would probably be too little data. Keep in mind that we have to teach it the syntax of English as well as what different cards do (e.g. creatures have power and toughness). Without enough training data, you either end up with a network that knows nothing or a network that knows how to precisely replicate the handful of cards that you showed it.
Of course, you're right to point that having all sorts of complex, pie-breaking cards make it hard for the network to learn any fixed rules about color discipline, other than basics like the fact that red likes damage spells and blue likes countermagic.
The problem we face is that we're having to make the most of fairly limited data. For instance, if you wired a network up to cameras pointed at the outside world, it could eventually learn about things like physics, the cycles of the sun and the moon, that people open umbrellas when it rains, et cetera because it would have an endless stream of snapshots to work with. Meanwhile, our network, studying the world of Magic cards, has only around 14000 cards to learn from. That is a very small number, and we're having to use all sorts of tricks to teach the network the most we can from that.
No problem! It looks like the network is just about done training. A few more results:
* Start with elegance.
* "Boggarts are also the final stages of Paradise!"
* The price is worth a thousand swamps.
* Her commands shall be a vast shout - the promise of branches.
* The past shook, the sea takes palled for war, well, and of those who should become favor onings out of civilization.
And I can tell you it's very fun to prime this network:
Whisper: "Goblins love"
* Goblins love the strongest of shields.
* Goblins love a good war.
* Goblins love the goal fruit beyond the granded soleient.
* Goblins love the strength of our dragonlord.
Whisper: "In the pale light, I saw"
* In the pale light, I saw the Legacy Councender.
* In the pale light, I saw it consume the clouds.
* In the pale light, I saw it as death. Few that does it into living creatures as well as bones and magical creatures to master the wisdom of annoying for a carve open its bonds and blocked out forever.
* In the pale light, I saw it failed in the night. I bring you compared to the forest's wishes.
Whisper: "President Obama said to the crowd of onlookers, ""
* President Obama said to the crowd of onlookers, "Fight!"
* President Obama said to the crowd of onlookers, "I?"
* President Obama said to the crowd of onlookers, "Kill!"
* President Obama said to the crowd of onlookers, "Stand."
* President Obama said to the crowd of onlookers, "Never look to create."
* President Obama said to the crowd of onlookers, "Crovax and I should know... But I'd never seen anything as dreadful as that."
Whisper: "As the fires consumed the village, Esma thought of"
* As the fires consumed the village, Esma thought of vengeance.
* As the fires consumed the village, Esma thought of others.
* As the fires consumed the village, Esma thought of Kreel still swirling himself to the capitol.
* As the fires consumed the village, Esma thought of kavugers and seeds of victory, necromancer would return home and where the lever be so crashoows to the point of the trolls' chant, and the seal but is the earth in a shaman cry to Seric from their experiences with the magician and more rocks as a living in the arts of stealth, royal assassins choose their victims carefully, relying on timing and precision.
So, to answer your original question, yes, we can produce flavor text for cards in the way that you wanted.
Oh I can't wait to get some artificially generated artwork. Put that with random flavor text and an improved card generation process and we'll be set.
What info about the card is passed into the flavor text generator? Also, for once you have the card generation, could you have the script ask the flavor network how good it thinks the flavor text is, and then set a threshold based on the length of the text of the base card for what minimum quality of flavor text it will accept. So a card with no body text would always or almost always get a flavor text, but if it has some body text it is more willing to go without, up till the point where the card is already full and no quality of flavor text would be accepted. This seems like it would be the closest match to how actual magic cards are made.
Flavor text is awesome. I loved that part of the Weatherlight cycle when Crovax was hanging out with Obama.
Don't see a link yet, but you guys made io9 today.
Some of those are utterly magnificent. I wonder how the results would be if we trained a network on cards with their flavour text included...
My main concerns would be bleed between fields (flavor text creeping into body text), and reduced rules text quality because more computational resources would be devoted to the flavor text at the expense of everything else. Think of it this way: if a lot of cards have a lot more flavor text than rules text, and we're judging the network on how accurately it predicts the card, then we're encouraging it to focus on flavor text first and rules text second.
Flavor text is awesome. I loved that part of the Weatherlight cycle when Crovax was hanging out with Obama.
Don't see a link yet, but you guys made io9 today.
Haha, I missed that one. So we're up to three articles so far and one parody. I can see the appeal of the hilarious results of the earlier, untrained versions of the networks. When I was asked to provide card dumps for the RoboRosewater Twitter feed, I gave the person responsible a massive collection of hilarious garbage cards and an equally massive collection of cards produced by a well-trained network. There's enough humor in that data set to last for months at least, if not more.
But I have a feeling that once we bring together good rules text (which we just about have), flavor text (which we also have), and artwork (which I plan to do), we'll wow them.
What info about the card is passed into the flavor text generator? Also, for once you have the card generation, could you have the script ask the flavor network how good it thinks the flavor text is, and then set a threshold based on the length of the text of the base card for what minimum quality of flavor text it will accept. So a card with no body text would always or almost always get a flavor text, but if it has some body text it is more willing to go without, up till the point where the card is already full and no quality of flavor text would be accepted. This seems like it would be the closest match to how actual magic cards are made.
At this point, there is no way to map cards to flavor text like I previously proposed for cards to art (other than to stitch random flavor text onto random cards), but I do have a plan:
1. Compute a content vector model for rules text. (good news: we already did this!)
2. Compute a content vector model for flavor text.
3. Learn a mapping from the rules text vector space to the flavor text vector space.
4. Matchmaking!
5. Profit.
For step three, imagine we have the card Flame Slash. We compute a pair of vectors, one that describes what Flame Slash does, and another that describes Flame Slash's flavor. We treat these as equivalent in meaning, and we have to figure out a way to translate from one vector to another. It's like a translation problem between human languages, where you have a sentence in one language:
"Je parle anglais seulement."
And we have to map that sentence to another sentence in a different language that carries the same weight and meaning (note that this is almost never the literal translation):
"I am an uncultured foreigner."
Once we can find the mapping (which I'll hand-wave over for now for the sake of simplicity), we now have a function M that takes a rules text vector and gives us a flavor text vector.
Now for step four. Assume we have two networks, one generating tons of rules text for cards and another independently generating flavor text for cards. They each produce a thousand units of their products. We want to match each card with the best possible flavor text. Here's how:
1. For each card c, compute its content vector r=v(c) (the translation of c's mana cost, types, body text, etc. into a series of numbers that succinctly describe it).
2. Compute M(r), which describes where c falls into the flavor space (e.g. red instants often end up in the lightning/fire/obliteration flavor region).
3. For each flavor text f, compute the cosine distance of v(f) and M(r). You get a number from 1 (a perfect match) to -1 (a terrible match).
4. For each card, we now have a list of preferences for flavor texts. We now have to solve a variant of the Stable Marriage Problem (see this video if you're really interested in knowing more).
Ultimately, we end up with flavor text that is at worst tangentially connected to the card and at best makes the card look like it was designed by a real human being. Blue flyers should get more flavor text involving skies and oceans, green cards about trees, white cards about arrogantly imposing personal morality upon people, etc.
I'll probably work on this task last, after the artwork. I'm still working out some of the implementation details on the mapping from card vectors to artwork, but I'm sure I'll get it figured out. In the mean time, I can probably write a script that takes a dump of cards and a dump of flavor text and stitches them together in the order that they are found, like this:
Cylian Guildmage G
Creature - Elf Wizard G, T: Uncast target spell or ability that targets a permanent you control. "The soul is only as a beautiful don't know its enemies. Once they find your true music lose a matter of nightmares."
1/1
Sea Bloodbraider 2R
Creature - Human Rogue
At the beginning of each end step, if an opponent controls more creatures than you, Sea Bloodbraider deals 4 damage to you. "The mountains queer than hume's country below the gifts of the tides."
4/3
Also, when I have the chance, I'll finish that gpu2cpu script. It runs, but it complains when I try to write the results to a file, saying I'm trying to write a null value (which I most certainly am not). I'll figure that one out. Once I have that I can share GPU-trained networks with you, like the flavor text network. I mean, you're free to have it now if you want, but for most of you it would only be useful if I also provided a CPU-compatible version.
EDIT: Just a few more, some silly ones:
* "I don't not finix. But I knew it were the way?" - Elvish expression meaning "brother of the Ancient Mariner"
* "It is not in a bottle of purpose, I'm going on instincts." - Daria
* He will kill you buck fulfilly but inside.
* "Can grow them to fund the fire of justice builds us. I wear such peaceful further in its grasp. Petals! When I had we gooded ords the servants of the heart." - Aurelian Scienel
* He wishes from within.
* "I suppose you trust your enemies. Perhaps I not like to stand at a dire without savage blows." - Arcum Dagsson, Soldevi machinist
On a decent number of occasions it starts by copying the flavor text of a real card and then altering its course:
* "I have become a gourmet of sorts. Each soul, the soil big coars in hope of knees. It would be killed, we die in times of failure." (see Urborg Syphon-Mage)
* "The fatal flaw in every plan is the assumption that you know my choice had seemed to be enough." - Volrath (see Mana Leak)
* While it possesses potent venom, the Giant Spider often chooses not to paralyze its victims. Perhaps the creature enjoys the gentle rocking motion caused all of Dominaria itself. (see Giant Spider)
I don't think we need to solve the stable marriage problem. Instead, we can just generate the card and then generate flavour texts until one is within a certain bound (say >0.95). Same for art; AFAIK it doesn't take that long to generate one piece and it seems easier to generate art (using prompting, of course) until there's one that fits the card. That's not to say that you couldn't collect all the unused ones and test them against each new card so you might not need to generate new art/flavour text, but there's no need to solve a complicated problem when good enough for all cards will do.
I don't think we need to solve the stable marriage problem. Instead, we can just generate the card and then generate flavour texts until one is within a certain bound (say >0.95). Same for art; AFAIK it doesn't take that long to generate one piece and it seems easier to generate art (using prompting, of course) until there's one that fits the card. That's not to say that you couldn't collect all the unused ones and test them against each new card so you might not need to generate new art/flavour text, but there's no need to solve a complicated problem when good enough for all cards will do.
Fair enough. You're right, we can just keep generating/reiterating over a bank of flavor texts until we find one that matches each card well enough. After all, if we did step 4 like I originally suggested, I could easily end up with cards with a 0.2 match for flavor text because there were no available suitors left. It'd be easier just to generate more flavor texts to replace the low quality one.
Excellent suggestion! Thank you so much for recommending that.
The closest english equivalent is probably "Sorry, I don't speak english" (I always learn the local equivalent wherever I go, with good results).
Non, je sais (mon copain parle la langue). C'était une blague. Je voulais exprimer... le dédain, the kind that seems common among native Parisians (and I have met many in my travels).
EDIT: Oh, if you wanted to be more precise about it, the literal translation is "I speak English only." Thus the joke.
Talcos, you remember the issue with padding chars / proper backpropagating of padding / no more cutting of cards for differents batches? I'd like to know if something's ready / in progress about that.
Yes, some work was done, but it didn't deliver the results that we wanted (it was, however, very informative):
For the record, the issue with splicing can be more or less eliminated with reduplication of cards, as hardcast_sixdrop has done when he randomized the fields for each card. However, this does lead us to some overfitting problems with names and missing/duplicated fields because the network thinks that it has completed a field when in fact it skipped it or forgets that it completed a field earlier in the card.
I'm still up for doing some more padding experiments in the future but we'll need to modify the training script to skip over special padding characters. That's do-able but I haven't gotten around to it yet.
More generally, I'd like to understand the network's architecture a bit and how parameters like batch_size and dropout work.
To keep it brief, batch_size is the number of different sequences that are trained on in parallel at one point in time (either on the GPU or CPU). Dropout is a regularization technique where during each iteration of the training a certain percentage of the cells at each layer are deactivated. A moderate amount of dropout makes the network more robust and less likely to be given over to strange and unproductive thinking. More precisely, a dropout rate, of say, 0.20 on a network with 1000 cells results in us having a network that is a combination of the best performing networks of size 800.
I have evidence of a bit of a card being spliced at some point in another card and the network reproducing it, I want to recheck before being 100% confident, but it's bugging me. It looks like even one occurence of something might be learned, at least in an early learning stage.
I tried to follow karpathy's code but lua is too different from C, I see the data stream is copied in x and y, y is offset by just 1 char (presumably input / desired output), then x and y are cut into chunks with validation chunks intersped between a run of training chunks, but what happens at the boundary between training / validation / training again chunks, is it like some data was cut off? Looks like everything's happening in the nn code.
I'll have to hold off on a more thorough explanation as I have some things that I need to take care of, but I can try and walk you through the code at some point in the near future if you'd like.
Finally, I've come to the conclusion that the network is not good at learning rather strict before/after conditions with some distance, it's a lot more general than just the X issue or kicker/kicked. It's good with the global structure, with fields in a set order and all, so I thought it would also learn easily some looser pre/post conditions, but no.
I wouldn't say it's that. It's that we have a small dataset, and the only way for us to get very strict pre/post conditions is to overfit on the few examples we have. I quoted a snippet from a relevant paper in the post I linked you to explaining how very long-term dependencies can in fact be learned.
Thanks a bunch Talcos - and everyone helping - this is incredible (and hilarious). I'm only halfway in the thread, was hoping to get the conversation on flavor text, and I'm very sad/glad you guys are way ahead of me. I'll keep reading and try to see how/if I can help.
Again, amazing job. Amazing, amazing job!
I'm very happy you've enjoyed the work we have been doing here! And yes, things move very fast here so it can be very hard to keep up, haha.
Sorry for posting one after the other, but I do have some good news. I'm able to get the image and vector data put into a form that I can load and use with Torch, so that's a step in the right direction. Hopefully I'll have some preliminary results to show in the near future.
EDIT: Specifically, I can input a vector into network, get an image out, and have it correct itself if the image doesn't match what I wanted. But I'm still working out all the details.
What image data are you inputting? And by vector data do you mean the array of 200 numbers that describe the card that you mentioned a few pages ago?
Whatever it is it sounds fantastic. I can't wait to see an example...!
Yes. So what we have are crappy quality card images from mwsdata.com and vector representations of cards, like what I described earlier. We're trying to see if we can train a network to take a goblin-ish vector and produce goblin-ish art corresponding with it. After all, it's all just numbers in and numbers out, one set of numbers representing card content and the other set of numbers representing intensities of light at various points on an x/y plane.
Of course, I'm still figuring out all the inner workings of the Torch libraries, so it may be a bit, haha. But hopefully I'll have something to show for it before too long.
So what's the set up? Do you have all existing cards as vector representations and each one is matched to its image?
Then, presumably if you create a new card and get its vector representation, if it's 0.95 similar to an existing card it'll generate an image that uses the key words (type, subtype, title etc) from the existing card?
Come to think of it, wouldn't this be a good way of doing card names too, since they're flavour only just like the art?
So what's the set up? Do you have all existing cards as vector representations and each one is matched to its image?
Yes, in the way that you've described. It's all just a hideous mess of numbers, as you can see in the image below (those are the card vectors, prior to being loaded and Torch-ified).
Then, presumably if you create a new card and get its vector representation, if it's 0.95 similar to an existing card it'll generate an image that uses the key words (type, subtype, title etc) from the existing card?
So... is the vector representation of cards one-way or reversible i.e. with a vector representation of a card, can we apply an algorithm and get all the original card's information back? If it's reversible, wouldn't it be a better 'format' to store the cards in for general training (fixed length and all that)?
If we create a card that's really unique and new to the game, and there are only poor matches to its vectors, wouldn't we end up with a rather poor title/flavour text as a result, or can it get creative?
Generated Mechanics Vector - > Closest Real Mechanics Vector, then Real Flavortext - > Closest Generated Flavortext
am I understanding correctly?
I do think that being able to whisper a card title and see what it thinks a card with that title should look like is a nifty feature. Put in 'Batman' or 'Rosewater' or 'Mistborn' and see what it does.
This is off-topic, but as an RPG designer - I've often looked for ways to make magic more unique to each spellcaster and to hide spells that the players can't just look up in the handbooks - so there is truly hidden magic they discover.
A program that can generate spells would be very interesting.
Hi all, just wanted to drop a comment quick so you know I still exist. I've been a little busy (and burned out from writing too much code ) so I haven't been quite as productive as I'd like, and it looks like this coming weekend will be busy too.
I tuned up the decoder for my format a little bit, I might get a chance tomorrow to integrate some of the word2vec stuff, and maybe just plain old text search, into it so it can automatically give you spoilers of the most similar cards, etc. Should be neat.
I'm super excited to see the progress on flavor text. What about training a network on names and flavor text (or maybe name and types and flavor text?) and then using the name generated by the card generator network to whisper to the flavor text generator? If the goal is to limit the number of neurons we waste on frivolous things, then we could basically lump all of the raw flavor/creativity into the naming and use that as a key into the pure flavor pieces like flavor text and art. It seems convenient to let it have some artistic license, though the overfitting I'm getting when duplicating the data a lot is a major issue there. The crazy names are more fun.
Oh, and if you look, there a couple of different checkpoints that I posted for the randomized fields network. I'm not posting the training set for that because it's huge (250MB) but you can generate it easily with the code on my github; assuming the existence of AllSets.json from mtgjson.com, the command line is:
Anyway, you might try some of the earlier checkpoints, from what I saw they were overfitting less but still able to produce interesting cards. The validation loss doesn't really mean anything for those checkpoints because the validation set is effectively the same as the training set.
Generated Mechanics Vector - > Closest Real Mechanics Vector, then Real Flavortext - > Closest Generated Flavortext
am I understanding correctly?
I do think that being able to whisper a card title and see what it thinks a card with that title should look like is a nifty feature. Put in 'Batman' or 'Rosewater' or 'Mistborn' and see what it does.
I think it's vectors for flavor too, if I understand you correctly, but I think you have more or less the right idea.
Right now the idea is that the mechanics vector is the sole determinant of the artwork and the flavor text separately. In other words,
Perhaps a more sophisticated system would be to treat all this in a holistic way, with artwork sometimes informing mechanics, and flavor text sometimes informing artwork. But I'm avoiding that for now.
The reason why I say that is because the cardnames / flavor text can be extremely nebulous. For example, look at the card Crushing Vines. I'm fairly certain that we're all in agreement that is a mess of a card. The mechanics have very little to do with the flavor and vice versa, the mechanics themselves have very little to do with each other, and the art seems poorly directed (not a slight to the artist, but it seems like Wizards didn't know what they really wanted). Anyway, the fact that the flavor text is disconnected with everything else is not all that uncommon for Magic cards, and that's difficult to deal with. The other problem is that even when the flavor text clearly refers to the art content, it can be difficult to decipher the meaning. For example, take the flavor text of Bant Panorama:
"Bant's hearts are as pure as its air and as bright as its skies."
So air and skies play a prominent role there, but the network may not have any idea about what Bant is. In fact, this flavor text could be perfect for Grixis (in an ironic sense).
In the end, the contribution of flavor to artwork is fuzzy enough that it would be difficult to distinguish it from pure noise.
The network doesn't always get it right, but it does a decent job. The live demo is down right now, but when I used it, I fed it a picture of me smiling while holding a ferret. The description that came back was "A man smiles while holding a very small cat." The network had never seen a ferret before, so it had to get creative, but I'd count that as a success. However, to get to that level of sophistication, the network had to study thousands of images for every class of object in order for it to figure out their essence (e.g. it can figure out that a cat is a cat regardless of the angle or the lighting).
Meanwhile, for categories of objects in Magic, we have very little data to go on. For example, there are 298 Goblin cards in Magic, as they are a tribe that has been in the game since Alpha. But that's not enough to go on, especially when you consider that they are the most physically diverse race in the game.
So for now I'm trying to keep things simple, haha.
This is off-topic, but as an RPG designer - I've often looked for ways to make magic more unique to each spellcaster and to hide spells that the players can't just look up in the handbooks - so there is truly hidden magic they discover.
A program that can generate spells would be very interesting.
That's funny, I had someone else ask me the same question (not on this forum, but an associate of mine) with regards to Pathfinder campaign content generation.
If you have the data, I have the software and hardware. Just PM me with details if you're interested in me running any special projects for you when my machines are not otherwise being used. With enough data, I can churn out thousands of artificially generated spells in minutes.
Hi all, just wanted to drop a comment quick so you know I still exist. I've been a little busy (and burned out from writing too much code ) so I haven't been quite as productive as I'd like, and it looks like this coming weekend will be busy too.
I tuned up the decoder for my format a little bit, I might get a chance tomorrow to integrate some of the word2vec stuff, and maybe just plain old text search, into it so it can automatically give you spoilers of the most similar cards, etc. Should be neat.
I'm super excited to see the progress on flavor text. What about training a network on names and flavor text (or maybe name and types and flavor text?) and then using the name generated by the card generator network to whisper to the flavor text generator? If the goal is to limit the number of neurons we waste on frivolous things, then we could basically lump all of the raw flavor/creativity into the naming and use that as a key into the pure flavor pieces like flavor text and art. It seems convenient to let it have some artistic license, though the overfitting I'm getting when duplicating the data a lot is a major issue there. The crazy names are more fun.
Oh, and if you look, there a couple of different checkpoints that I posted for the randomized fields network. I'm not posting the training set for that because it's huge (250MB) but you can generate it easily with the code on my github; assuming the existence of AllSets.json from mtgjson.com, the command line is:
Anyway, you might try some of the earlier checkpoints, from what I saw they were overfitting less but still able to produce interesting cards. The validation loss doesn't really mean anything for those checkpoints because the validation set is effectively the same as the training set.
First, don't apologize! I know that you've been very busy with your internship (where I'm sure coding is a strong component), and I don't want you to overwork yourself for our sake. After all, you have done so much already, and all of us here are eternally in your debt. I'm fine with just having your moral support if you need to reserve your personal time for, well, yourself, haha.
Second, mapping names to flavor text is definitely something that I'm wanting to experiment with in the near future. I agree that idea has some potential.
Third, I also noted the overfitting on names due to the duplication. It's something we can look into.
Fourth, I'll take a look at your earlier checkpoints.
Now a question: if I ^C (completely stop a process), I know I can init from a checkpoint, but do I loose some metaparameters like learning_rate if I do?
Well, not that i want to shut down my computer in the near future Wink
Unfortunately, you have to restart from epoch 1. The training script does not pick up exactly where you left off with regards to the learning rate. However, there is a way of doing what you want. You can limit the number of epochs that you wanted to run, set the learning decay to begin at timestep one, and pick a starting learning_rate that matches the decayed learning_rate at the time that you stopped training beforehand.
# Is there some mixing of cards going on in the training process?
# How come a FULLY TRAINED network learned something that happened just once, and produces it twice? [no, thrice!]
# Is there an issue with the random number generator (repeating unlikely occurences)? What gives?
Question: Do you reserve a single character that indicates the end of a card? Like the newline character? Or does that character get used more than once within cards? And do you use any special characters indicating the start or the end of fields?
It's possible that the network could mistakenly think that it is still working on the same card for an extended period leading to field duplications. After all, the network can remember information indefinitely and if it's not properly trained to forget information, it can repeat it.
For fixing the names: I'm not sure if this is easy, but this might work to get back the variation in names that we saw earlier:
1) Train an RNN on the data (without randomized fields) with the name last.
2) Prime the network with each card a few times, and use the names it generates as the names for the duplicates of the card when randomizing the field order.
This should have the result of giving us more creative names at the cost of them being slightly less correlated to the text of the card.
Private Mod Note
():
Rollback Post to RevisionRollBack
To post a comment, please login or register a new account.
As for the flavor text, I extracted all the flavor texts, threw delimiters around them, and fed them into the network. I wanted to see how well the network could do with just flavor text.
And no, sample_hs_v2.lua is not meant to work with the random format, but you can still prime with it if you use the "-name" option. What "-name" will do is start priming right after the first bar is hit, so you can start off a card like this: -name "4legendary|5creature|6elf warrior".
By the way, the network is still training (it's still improving over the epochs), but I'm already getting such awesome results, I have to share a few:
* "I will show you that the king yet reigns!"
* The sea is the resource of the past.
* They are lightning streams, spirits of the sphinxesply plain.
* "The Tajuru executed him." - Momir Vig
* "The clouds can be found, like and trained the clockwork in his own skin." - Zul Ashur, lich lord
* "I saw a book! We will never find them to the call. One so greed? But it's low, like shiny." - Narset
Sometimes it partially clones existing flavor texts and then changes them up:
* The most terrifying thing about the craw wurm is probably the horrible crashing sound it makes as it speeds through the forest, free of the evil from the consequent of Mirrodin's only conviction.
(compare to Craw Wurm)
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
Or would that give it too little data to work off of?
That would probably be too little data. Keep in mind that we have to teach it the syntax of English as well as what different cards do (e.g. creatures have power and toughness). Without enough training data, you either end up with a network that knows nothing or a network that knows how to precisely replicate the handful of cards that you showed it.
Of course, you're right to point that having all sorts of complex, pie-breaking cards make it hard for the network to learn any fixed rules about color discipline, other than basics like the fact that red likes damage spells and blue likes countermagic.
The problem we face is that we're having to make the most of fairly limited data. For instance, if you wired a network up to cameras pointed at the outside world, it could eventually learn about things like physics, the cycles of the sun and the moon, that people open umbrellas when it rains, et cetera because it would have an endless stream of snapshots to work with. Meanwhile, our network, studying the world of Magic cards, has only around 14000 cards to learn from. That is a very small number, and we're having to use all sorts of tricks to teach the network the most we can from that.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
No problem! It looks like the network is just about done training. A few more results:
* Start with elegance.
* "Boggarts are also the final stages of Paradise!"
* The price is worth a thousand swamps.
* Her commands shall be a vast shout - the promise of branches.
* The past shook, the sea takes palled for war, well, and of those who should become favor onings out of civilization.
And I can tell you it's very fun to prime this network:
Whisper: "Goblins love"
* Goblins love the strongest of shields.
* Goblins love a good war.
* Goblins love the goal fruit beyond the granded soleient.
* Goblins love the strength of our dragonlord.
Whisper: "In the pale light, I saw"
* In the pale light, I saw the Legacy Councender.
* In the pale light, I saw it consume the clouds.
* In the pale light, I saw it as death. Few that does it into living creatures as well as bones and magical creatures to master the wisdom of annoying for a carve open its bonds and blocked out forever.
* In the pale light, I saw it failed in the night. I bring you compared to the forest's wishes.
Whisper: "President Obama said to the crowd of onlookers, ""
* President Obama said to the crowd of onlookers, "Fight!"
* President Obama said to the crowd of onlookers, "I?"
* President Obama said to the crowd of onlookers, "Kill!"
* President Obama said to the crowd of onlookers, "Stand."
* President Obama said to the crowd of onlookers, "Never look to create."
* President Obama said to the crowd of onlookers, "Crovax and I should know... But I'd never seen anything as dreadful as that."
Whisper: "As the fires consumed the village, Esma thought of"
* As the fires consumed the village, Esma thought of vengeance.
* As the fires consumed the village, Esma thought of others.
* As the fires consumed the village, Esma thought of Kreel still swirling himself to the capitol.
* As the fires consumed the village, Esma thought of kavugers and seeds of victory, necromancer would return home and where the lever be so crashoows to the point of the trolls' chant, and the seal but is the earth in a shaman cry to Seric from their experiences with the magician and more rocks as a living in the arts of stealth, royal assassins choose their victims carefully, relying on timing and precision.
So, to answer your original question, yes, we can produce flavor text for cards in the way that you wanted.
Oh I can't wait to get some artificially generated artwork. Put that with random flavor text and an improved card generation process and we'll be set.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
Don't see a link yet, but you guys made io9 today.
Wizards quality
My main concerns would be bleed between fields (flavor text creeping into body text), and reduced rules text quality because more computational resources would be devoted to the flavor text at the expense of everything else. Think of it this way: if a lot of cards have a lot more flavor text than rules text, and we're judging the network on how accurately it predicts the card, then we're encouraging it to focus on flavor text first and rules text second.
Haha, I missed that one. So we're up to three articles so far and one parody. I can see the appeal of the hilarious results of the earlier, untrained versions of the networks. When I was asked to provide card dumps for the RoboRosewater Twitter feed, I gave the person responsible a massive collection of hilarious garbage cards and an equally massive collection of cards produced by a well-trained network. There's enough humor in that data set to last for months at least, if not more.
But I have a feeling that once we bring together good rules text (which we just about have), flavor text (which we also have), and artwork (which I plan to do), we'll wow them.
At this point, there is no way to map cards to flavor text like I previously proposed for cards to art (other than to stitch random flavor text onto random cards), but I do have a plan:
1. Compute a content vector model for rules text. (good news: we already did this!)
2. Compute a content vector model for flavor text.
3. Learn a mapping from the rules text vector space to the flavor text vector space.
4. Matchmaking!
5. Profit.
For step three, imagine we have the card Flame Slash. We compute a pair of vectors, one that describes what Flame Slash does, and another that describes Flame Slash's flavor. We treat these as equivalent in meaning, and we have to figure out a way to translate from one vector to another. It's like a translation problem between human languages, where you have a sentence in one language:
"Je parle anglais seulement."
And we have to map that sentence to another sentence in a different language that carries the same weight and meaning (note that this is almost never the literal translation):
"I am an uncultured foreigner."
Once we can find the mapping (which I'll hand-wave over for now for the sake of simplicity), we now have a function M that takes a rules text vector and gives us a flavor text vector.
Now for step four. Assume we have two networks, one generating tons of rules text for cards and another independently generating flavor text for cards. They each produce a thousand units of their products. We want to match each card with the best possible flavor text. Here's how:
1. For each card c, compute its content vector r=v(c) (the translation of c's mana cost, types, body text, etc. into a series of numbers that succinctly describe it).
2. Compute M(r), which describes where c falls into the flavor space (e.g. red instants often end up in the lightning/fire/obliteration flavor region).
3. For each flavor text f, compute the cosine distance of v(f) and M(r). You get a number from 1 (a perfect match) to -1 (a terrible match).
4. For each card, we now have a list of preferences for flavor texts. We now have to solve a variant of the Stable Marriage Problem (see this video if you're really interested in knowing more).
Ultimately, we end up with flavor text that is at worst tangentially connected to the card and at best makes the card look like it was designed by a real human being. Blue flyers should get more flavor text involving skies and oceans, green cards about trees, white cards about arrogantly imposing personal morality upon people, etc.
I'll probably work on this task last, after the artwork. I'm still working out some of the implementation details on the mapping from card vectors to artwork, but I'm sure I'll get it figured out. In the mean time, I can probably write a script that takes a dump of cards and a dump of flavor text and stitches them together in the order that they are found, like this:
Cylian Guildmage
G
Creature - Elf Wizard
G, T: Uncast target spell or ability that targets a permanent you control.
"The soul is only as a beautiful don't know its enemies. Once they find your true music lose a matter of nightmares."
1/1
Sea Bloodbraider
2R
Creature - Human Rogue
At the beginning of each end step, if an opponent controls more creatures than you, Sea Bloodbraider deals 4 damage to you.
"The mountains queer than hume's country below the gifts of the tides."
4/3
Also, when I have the chance, I'll finish that gpu2cpu script. It runs, but it complains when I try to write the results to a file, saying I'm trying to write a null value (which I most certainly am not). I'll figure that one out. Once I have that I can share GPU-trained networks with you, like the flavor text network. I mean, you're free to have it now if you want, but for most of you it would only be useful if I also provided a CPU-compatible version.
EDIT: Just a few more, some silly ones:
* "I don't not finix. But I knew it were the way?" - Elvish expression meaning "brother of the Ancient Mariner"
* "It is not in a bottle of purpose, I'm going on instincts." - Daria
* He will kill you buck fulfilly but inside.
* "Can grow them to fund the fire of justice builds us. I wear such peaceful further in its grasp. Petals! When I had we gooded ords the servants of the heart." - Aurelian Scienel
* He wishes from within.
* "I suppose you trust your enemies. Perhaps I not like to stand at a dire without savage blows." - Arcum Dagsson, Soldevi machinist
On a decent number of occasions it starts by copying the flavor text of a real card and then altering its course:
* "I have become a gourmet of sorts. Each soul, the soil big coars in hope of knees. It would be killed, we die in times of failure." (see Urborg Syphon-Mage)
* "The fatal flaw in every plan is the assumption that you know my choice had seemed to be enough." - Volrath (see Mana Leak)
* While it possesses potent venom, the Giant Spider often chooses not to paralyze its victims. Perhaps the creature enjoys the gentle rocking motion caused all of Dominaria itself. (see Giant Spider)
*
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
Fair enough. You're right, we can just keep generating/reiterating over a bank of flavor texts until we find one that matches each card well enough. After all, if we did step 4 like I originally suggested, I could easily end up with cards with a 0.2 match for flavor text because there were no available suitors left. It'd be easier just to generate more flavor texts to replace the low quality one.
Excellent suggestion! Thank you so much for recommending that.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
Non, je sais (mon copain parle la langue). C'était une blague. Je voulais exprimer... le dédain, the kind that seems common among native Parisians (and I have met many in my travels).
EDIT: Oh, if you wanted to be more precise about it, the literal translation is "I speak English only." Thus the joke.
Yes, some work was done, but it didn't deliver the results that we wanted (it was, however, very informative):
http://www.mtgsalvation.com/forums/creativity/custom-card-creation/612057-generating-magic-cards-using-deep-recurrent-neural?page=36#c880
By the way, here's a graph that can help identify how different sequence lengths will affect the splicing issue:
https://plot.ly/~rmmilewi/935/character-lengths-of-cards-organized-by-type/
For the record, the issue with splicing can be more or less eliminated with reduplication of cards, as hardcast_sixdrop has done when he randomized the fields for each card. However, this does lead us to some overfitting problems with names and missing/duplicated fields because the network thinks that it has completed a field when in fact it skipped it or forgets that it completed a field earlier in the card.
I'm still up for doing some more padding experiments in the future but we'll need to modify the training script to skip over special padding characters. That's do-able but I haven't gotten around to it yet.
To keep it brief, batch_size is the number of different sequences that are trained on in parallel at one point in time (either on the GPU or CPU). Dropout is a regularization technique where during each iteration of the training a certain percentage of the cells at each layer are deactivated. A moderate amount of dropout makes the network more robust and less likely to be given over to strange and unproductive thinking. More precisely, a dropout rate, of say, 0.20 on a network with 1000 cells results in us having a network that is a combination of the best performing networks of size 800.
I'll have to hold off on a more thorough explanation as I have some things that I need to take care of, but I can try and walk you through the code at some point in the near future if you'd like.
I wouldn't say it's that. It's that we have a small dataset, and the only way for us to get very strict pre/post conditions is to overfit on the few examples we have. I quoted a snippet from a relevant paper in the post I linked you to explaining how very long-term dependencies can in fact be learned.
I'm very happy you've enjoyed the work we have been doing here! And yes, things move very fast here so it can be very hard to keep up, haha.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
EDIT: Specifically, I can input a vector into network, get an image out, and have it correct itself if the image doesn't match what I wanted. But I'm still working out all the details.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
Whatever it is it sounds fantastic. I can't wait to see an example...!
Yes. So what we have are crappy quality card images from mwsdata.com and vector representations of cards, like what I described earlier. We're trying to see if we can train a network to take a goblin-ish vector and produce goblin-ish art corresponding with it. After all, it's all just numbers in and numbers out, one set of numbers representing card content and the other set of numbers representing intensities of light at various points on an x/y plane.
Of course, I'm still figuring out all the inner workings of the Torch libraries, so it may be a bit, haha. But hopefully I'll have something to show for it before too long.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
Then, presumably if you create a new card and get its vector representation, if it's 0.95 similar to an existing card it'll generate an image that uses the key words (type, subtype, title etc) from the existing card?
Come to think of it, wouldn't this be a good way of doing card names too, since they're flavour only just like the art?
Yes, in the way that you've described. It's all just a hideous mess of numbers, as you can see in the image below (those are the card vectors, prior to being loaded and Torch-ified).
Also correct!
Hmm.. not a bad idea. Actually, we may be able to generate the flavor and the card name at the same time. That's something worth looking into.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
If we create a card that's really unique and new to the game, and there are only poor matches to its vectors, wouldn't we end up with a rather poor title/flavour text as a result, or can it get creative?
Then we would lose the ability to prime with titles, wouldn't we? I suppose it's the least-important thing to be able to prime by, but it is useful.
Generated Mechanics Vector - > Closest Real Mechanics Vector, then Real Flavortext - > Closest Generated Flavortext
am I understanding correctly?
I do think that being able to whisper a card title and see what it thinks a card with that title should look like is a nifty feature. Put in 'Batman' or 'Rosewater' or 'Mistborn' and see what it does.
A program that can generate spells would be very interesting.
Remaking Magic - A Podcast for those that love MTG and Game Design
The Dungeon Master's Guide - A Podcast for those that love RPGs and Game Design
Sig-Heroes of the Plane
I tuned up the decoder for my format a little bit, I might get a chance tomorrow to integrate some of the word2vec stuff, and maybe just plain old text search, into it so it can automatically give you spoilers of the most similar cards, etc. Should be neat.
I'm super excited to see the progress on flavor text. What about training a network on names and flavor text (or maybe name and types and flavor text?) and then using the name generated by the card generator network to whisper to the flavor text generator? If the goal is to limit the number of neurons we waste on frivolous things, then we could basically lump all of the raw flavor/creativity into the naming and use that as a key into the pure flavor pieces like flavor text and art. It seems convenient to let it have some artistic license, though the overfitting I'm getting when duplicating the data a lot is a major issue there. The crazy names are more fun.
Oh, and if you look, there a couple of different checkpoints that I posted for the randomized fields network. I'm not posting the training set for that because it's huge (250MB) but you can generate it easily with the code on my github; assuming the existence of AllSets.json from mtgjson.com, the command line is:
Anyway, you might try some of the earlier checkpoints, from what I saw they were overfitting less but still able to produce interesting cards. The validation loss doesn't really mean anything for those checkpoints because the validation set is effectively the same as the training set.
True. Well, we'll see. We can try this multiple ways and see what works best. Nothing is set in stone yet.
I think it's vectors for flavor too, if I understand you correctly, but I think you have more or less the right idea.
Right now the idea is that the mechanics vector is the sole determinant of the artwork and the flavor text separately. In other words,
generated mechanics vector -> generated artwork
generated mechanics -> generated flavor
Perhaps a more sophisticated system would be to treat all this in a holistic way, with artwork sometimes informing mechanics, and flavor text sometimes informing artwork. But I'm avoiding that for now.
The reason why I say that is because the cardnames / flavor text can be extremely nebulous. For example, look at the card Crushing Vines. I'm fairly certain that we're all in agreement that is a mess of a card. The mechanics have very little to do with the flavor and vice versa, the mechanics themselves have very little to do with each other, and the art seems poorly directed (not a slight to the artist, but it seems like Wizards didn't know what they really wanted). Anyway, the fact that the flavor text is disconnected with everything else is not all that uncommon for Magic cards, and that's difficult to deal with. The other problem is that even when the flavor text clearly refers to the art content, it can be difficult to decipher the meaning. For example, take the flavor text of Bant Panorama:
"Bant's hearts are as pure as its air and as bright as its skies."
So air and skies play a prominent role there, but the network may not have any idea about what Bant is. In fact, this flavor text could be perfect for Grixis (in an ironic sense).
In the end, the contribution of flavor to artwork is fuzzy enough that it would be difficult to distinguish it from pure noise.
Now, this being said, mapping images to text (and vice versa) is perfectly possible. For example, take a look at this (static) demo of a neural network that automatically tags and captions images (courtesy of the Toronto Deep Learning group).
The network doesn't always get it right, but it does a decent job. The live demo is down right now, but when I used it, I fed it a picture of me smiling while holding a ferret. The description that came back was "A man smiles while holding a very small cat." The network had never seen a ferret before, so it had to get creative, but I'd count that as a success. However, to get to that level of sophistication, the network had to study thousands of images for every class of object in order for it to figure out their essence (e.g. it can figure out that a cat is a cat regardless of the angle or the lighting).
Meanwhile, for categories of objects in Magic, we have very little data to go on. For example, there are 298 Goblin cards in Magic, as they are a tribe that has been in the game since Alpha. But that's not enough to go on, especially when you consider that they are the most physically diverse race in the game.
So for now I'm trying to keep things simple, haha.
That's funny, I had someone else ask me the same question (not on this forum, but an associate of mine) with regards to Pathfinder campaign content generation.
If you have the data, I have the software and hardware. Just PM me with details if you're interested in me running any special projects for you when my machines are not otherwise being used. With enough data, I can churn out thousands of artificially generated spells in minutes.
First, don't apologize! I know that you've been very busy with your internship (where I'm sure coding is a strong component), and I don't want you to overwork yourself for our sake. After all, you have done so much already, and all of us here are eternally in your debt. I'm fine with just having your moral support if you need to reserve your personal time for, well, yourself, haha.
Second, mapping names to flavor text is definitely something that I'm wanting to experiment with in the near future. I agree that idea has some potential.
Third, I also noted the overfitting on names due to the duplication. It's something we can look into.
Fourth, I'll take a look at your earlier checkpoints.
Unfortunately, you have to restart from epoch 1. The training script does not pick up exactly where you left off with regards to the learning rate. However, there is a way of doing what you want. You can limit the number of epochs that you wanted to run, set the learning decay to begin at timestep one, and pick a starting learning_rate that matches the decayed learning_rate at the time that you stopped training beforehand.
Question: Do you reserve a single character that indicates the end of a card? Like the newline character? Or does that character get used more than once within cards? And do you use any special characters indicating the start or the end of fields?
It's possible that the network could mistakenly think that it is still working on the same card for an extended period leading to field duplications. After all, the network can remember information indefinitely and if it's not properly trained to forget information, it can repeat it.
My LinkedIn profile... thing (I have one of those now!).
My research team's webpage.
The mtg-rnn repo and the mtg-encode repo.
1) Train an RNN on the data (without randomized fields) with the name last.
2) Prime the network with each card a few times, and use the names it generates as the names for the duplicates of the card when randomizing the field order.
This should have the result of giving us more creative names at the cost of them being slightly less correlated to the text of the card.