Okay, the promised second (of probably many, but trying to spread this out) post.
First, the code. I have just noticed that there isn't directories for ./logs, ./stats, and ./weights. There should be. It might explode without those. I will spend more time polishing it once I have confirmation that literally anyone but me is trying to do this stuff again.
Now, current status.
I will go over the technical bits more in a later post, but I already threw a lot of technical stuff earlier. For this one I will go over the results I've seen and things I've learned about how this type of network learns. I am doing all current experiments with just type and mana cost because those are fairly independent and doing on only two allows me to train very quickly.
a) There is value in tweaking the numbers for each individual data type. I originally gave only four nodes explicitly to type because there's not a lot of combinations that are possible. Going that low started to affect the effectiveness of mana cost. It kinda makes sense, it couldn't do a good enough job with the nodes that are dedicated, so it tries to use the general nodes... but doesn't have enough time to use them effectively, making things worse overall.
b) Length seems to be the easiest thing for it to learn. Looking at epoch 1 of this most current training set, it turned "instant" into "creatur". Creatur is the same length as instant (the right answer). It makes that mistake a lot. It takes a bit longer to fully learn length for mana cost but even there it quickly gets into the point where it's the right LENGTH, and it's the right FORMAT, but it doesn't get things fully right. It turns {3}{g}{g} -> {2}{u}{b}. Same length. It never seems to make the mistake of something like {u}{2}{b}, so it's getting the general format down pretty quickly, too. I have an experiment I'm going to run soon to see if I can nudge it into the right direction without increasing the number of nodes.
c) Planeswalkers are hard. No different from the old system, though mine does get to the point of encoding planeswalker after a while. In theory that means that it can generate them by picking specific numbers in the encoding.
d) White, for whatever reason, seems to be the easiest. I have it set up to encode/decode six specific cards chosen largely at random. It seems to get the white mana symbols much easier than other colors. It likes to turn green into either blue or black. Epoch 92 of this run with the largest numbers I've thrown at it comes back with {3}{g}{g} -> {2}{u}{b}, {w} -> {w}, {4}{u}{u} -> {1}{u}{u}, {3}{w} -> {1}{w}, and {7} -> {3}. In this case it got the double blue card right, too, but it often doesn't. It always seems to get the white right after a full training run though.
d) Before it "gets" planeswalkers, it usually wants to make them enchantments.
I had been following this thread when it was active and trying to teach myself machine learning. Periodically I would wonder why things weren't done a different way, or an idea would come up that got me super excited, but never (to my knowledge) was tried. Coming back and seeing it basically dead has encouraged me to try and implement a bunch of those things to see if I can move the project forward.
First, a pretty fundamental change: we aren't using (just) LSTMs anymore. Now we're using what's called a VAE. Feel free to read tutorials and whatnot yourself, but I always think that you don't understand a thing until you can explain it, so I will attempt to do that.
An LSTM works by keeping a bit of memory around about how it sees the world. If it's pretty smart, it might only keep a very dense bit of knowledge ("we're writing a type, it's an instant, we're on character 3") so that it doesn't really have to think very hard to keep things straight. If it's a bit dumber it can still do a good job, but in a less intelligent, more error-prone way ("we just wrote 'ins', I have no idea where we are in the card, but maybe that's instant? I guess the next letter should be a t?").
A VAE (Variational Auto-Encoder) at a high level is all about managing that idea of memory. You split the network into two parts. They go by different names, but I will call them an encoder and a decoder. You might see me refer to the decoder as the posterior later. The first part of the network, the encoder, is tasked with creating the encoded state of things. For our purposes you would expect to see the type of card encoded, as well as things like CMC. These are neural networks, so they're somewhat black boxes (though we can and hopefully will make them less of a black box later) but that's theoretically how it works.
The second half, you guessed it, decodes that memory into the original card again. Why is that useful? We're essentially squeezing the encoded representation down as far as it can possibly go. By doing so we can force it to remove redundant or useless information so that it can fit what IS necessary. Going back to CMC, it probably wouldn't actually store the CMC in the memory directly. It would probably store the mana cost -- CMC can be derived. Since it's deriving derived information, it should force things to become more self-consistent.
Now, this encoded memory has some fun properties. We can encode a card and mess with the individual numbers in the encoded memory space to see what it does. I would expect to be able to modify a number and make a card "more red" or "more creature-like." I don't get the PICK what each axis represents, but anything the machine chooses to put on an axis is a thing that I can twiddle after the fact. That sounds exciting!
As my last major tweak, I am actually using one LSTM per type of input (so mana cost, name, power, toughness, etc.) I am doing this for two reasons:
a) Memory length. LSTMs are better than older forms of RNNs, but they too suffer from a lack of very long term memory. I want to reduce the length of LSTM memory strands. If the card body is its own LSTM strand, that's about as short as it can realistically be (a fun REALLY out there project would be to try and make each sentence in the body its own LSTM... I have no idea if that could possibly work). Since the different outputs should be correlated in the memory itself, they shouldn't need to be correlated explicitly in the LSTM outputs.
b) Infilling. Infilling is a fun big concept in machine learning with images. You remove a part of an image and train a neural network to paint in the missing parts. We can (in theory) do the same with this network. I should be able to train the network to fill in the mana cost if I put in a special value that represents "I don't know" and give it the rest of the card. This is, I hope, a better way to prime the network than the previous way.
As a FINAL note on the technical parts -- this encoded information about the card... that's the perfect sort of thing to try and generate card art out of. It SHOULD contain enough info to tell that it's a creature, its cmc, its name, and anything else that's actually on the card.
In my next post I will be providing a link to my current code (it's messy, in python and Keras, I only use python for machine learning and don't really follow language conventions... sorry) and some details about my current status.
First, the code. I have just noticed that there isn't directories for ./logs, ./stats, and ./weights. There should be. It might explode without those. I will spend more time polishing it once I have confirmation that literally anyone but me is trying to do this stuff again.
Now, current status.
I will go over the technical bits more in a later post, but I already threw a lot of technical stuff earlier. For this one I will go over the results I've seen and things I've learned about how this type of network learns. I am doing all current experiments with just type and mana cost because those are fairly independent and doing on only two allows me to train very quickly.
a) There is value in tweaking the numbers for each individual data type. I originally gave only four nodes explicitly to type because there's not a lot of combinations that are possible. Going that low started to affect the effectiveness of mana cost. It kinda makes sense, it couldn't do a good enough job with the nodes that are dedicated, so it tries to use the general nodes... but doesn't have enough time to use them effectively, making things worse overall.
b) Length seems to be the easiest thing for it to learn. Looking at epoch 1 of this most current training set, it turned "instant" into "creatur". Creatur is the same length as instant (the right answer). It makes that mistake a lot. It takes a bit longer to fully learn length for mana cost but even there it quickly gets into the point where it's the right LENGTH, and it's the right FORMAT, but it doesn't get things fully right. It turns {3}{g}{g} -> {2}{u}{b}. Same length. It never seems to make the mistake of something like {u}{2}{b}, so it's getting the general format down pretty quickly, too. I have an experiment I'm going to run soon to see if I can nudge it into the right direction without increasing the number of nodes.
c) Planeswalkers are hard. No different from the old system, though mine does get to the point of encoding planeswalker after a while. In theory that means that it can generate them by picking specific numbers in the encoding.
d) White, for whatever reason, seems to be the easiest. I have it set up to encode/decode six specific cards chosen largely at random. It seems to get the white mana symbols much easier than other colors. It likes to turn green into either blue or black. Epoch 92 of this run with the largest numbers I've thrown at it comes back with {3}{g}{g} -> {2}{u}{b}, {w} -> {w}, {4}{u}{u} -> {1}{u}{u}, {3}{w} -> {1}{w}, and {7} -> {3}. In this case it got the double blue card right, too, but it often doesn't. It always seems to get the white right after a full training run though.
d) Before it "gets" planeswalkers, it usually wants to make them enchantments.
First, a pretty fundamental change: we aren't using (just) LSTMs anymore. Now we're using what's called a VAE. Feel free to read tutorials and whatnot yourself, but I always think that you don't understand a thing until you can explain it, so I will attempt to do that.
An LSTM works by keeping a bit of memory around about how it sees the world. If it's pretty smart, it might only keep a very dense bit of knowledge ("we're writing a type, it's an instant, we're on character 3") so that it doesn't really have to think very hard to keep things straight. If it's a bit dumber it can still do a good job, but in a less intelligent, more error-prone way ("we just wrote 'ins', I have no idea where we are in the card, but maybe that's instant? I guess the next letter should be a t?").
A VAE (Variational Auto-Encoder) at a high level is all about managing that idea of memory. You split the network into two parts. They go by different names, but I will call them an encoder and a decoder. You might see me refer to the decoder as the posterior later. The first part of the network, the encoder, is tasked with creating the encoded state of things. For our purposes you would expect to see the type of card encoded, as well as things like CMC. These are neural networks, so they're somewhat black boxes (though we can and hopefully will make them less of a black box later) but that's theoretically how it works.
The second half, you guessed it, decodes that memory into the original card again. Why is that useful? We're essentially squeezing the encoded representation down as far as it can possibly go. By doing so we can force it to remove redundant or useless information so that it can fit what IS necessary. Going back to CMC, it probably wouldn't actually store the CMC in the memory directly. It would probably store the mana cost -- CMC can be derived. Since it's deriving derived information, it should force things to become more self-consistent.
Now, this encoded memory has some fun properties. We can encode a card and mess with the individual numbers in the encoded memory space to see what it does. I would expect to be able to modify a number and make a card "more red" or "more creature-like." I don't get the PICK what each axis represents, but anything the machine chooses to put on an axis is a thing that I can twiddle after the fact. That sounds exciting!
As my last major tweak, I am actually using one LSTM per type of input (so mana cost, name, power, toughness, etc.) I am doing this for two reasons:
a) Memory length. LSTMs are better than older forms of RNNs, but they too suffer from a lack of very long term memory. I want to reduce the length of LSTM memory strands. If the card body is its own LSTM strand, that's about as short as it can realistically be (a fun REALLY out there project would be to try and make each sentence in the body its own LSTM... I have no idea if that could possibly work). Since the different outputs should be correlated in the memory itself, they shouldn't need to be correlated explicitly in the LSTM outputs.
b) Infilling. Infilling is a fun big concept in machine learning with images. You remove a part of an image and train a neural network to paint in the missing parts. We can (in theory) do the same with this network. I should be able to train the network to fill in the mana cost if I put in a special value that represents "I don't know" and give it the rest of the card. This is, I hope, a better way to prime the network than the previous way.
As a FINAL note on the technical parts -- this encoded information about the card... that's the perfect sort of thing to try and generate card art out of. It SHOULD contain enough info to tell that it's a creature, its cmc, its name, and anything else that's actually on the card.
In my next post I will be providing a link to my current code (it's messy, in python and Keras, I only use python for machine learning and don't really follow language conventions... sorry) and some details about my current status.