so I'm stumped as to why I can't install cutorch, kinda
What's your full system setup, if you don't mind my asking? I mean OS / hardware configuration. It looks like it's having a hard time building nvcc because it thinks you have some bizarre gpu architecture, which is interesting. Have you tried installing Torch without CUDA support to see if that works?
@hardcast: could you add a function or two in the decode.py to allow me to use it to decode text within a program? so I can eat the output of sortcards.py and decode each subset. that way the pretty HTML can be sorted.
Yes, I agree with you, both encode and decode could be more modular and expose their string translation capabilities as a separate function from the main method, and really they should share the text file / possibly json opening code from some central location as well. This is on my to-do list.
In the meantime, I should point out that the proper way to handle data is as card objects rather than text streams, and if you already have card objects then invoking the pretty formatting logic that decode.py uses is as simple as looping over the cards and having each one invoke its format() method with the right arguments. sortcards.py is a terrible example here, as it was written before card objects existed, so it isn't taking advantage of any of the modern infrastructure. I'd at least look at scripts/summarize.py, and how the indices are set up in lib/datalib.py, if you want an idea of how the infrastructure is supposed to work.
@Hardcast: I actually haven't looked at the -mse flag. is Magic Set Editor 2 the program you have to run in WINe? I was planning on just building a script to turn dumps into .html files with the icons all locally in the folder; maybe sprinkle a little .css and call it good, partially so I can have multiple columns so that I can see the scroll bar.
Yes, MSE2 requires WINe if you're running it on Linux. I found the setup with WINe to be pretty painless though, definitely worth the effort to have it available as a visualization tool. And if you have any Windows machines, it should run perfectly fine natively on one of those.
That said, having an output format that was just a plain HTML rendering that had prettified mana symbols like this forum would be totally awesome. If you build a generator for something like that, please, please give me the code so I can add it to mtgencode.
I am actually hacking together a script to format spoilers to look like they do when posted here. (that pesky 30k character limit) This will enable us, with or without the sortcards.py really, to look upon our 10M dumps and be amazed. With the sortcards.py we can easily see where we are having problems and where things seem to have gone off the rails. I would love to help get some of the harder statistical analysis going as well, but I'm not sure what we would be looking for and so could help very little. If you have any directions I could look to get hard stats going on our dumps let me know.
Two responses to this.
First, have you taken a look at Magic Set Editor 2? Using the -mse flag for decode.py to produce an mse-set file and the viewing it with MSE2 is pretty comfortable for looking at dumps, and it has some basic statistics and the ability to sort things if you're interested in a particular subset.
Second, I'm officially on the books to do a project on this work for my machine learning class (the prof apparently though text generation was so interesting a topic that he added karpathy's original blog post to the list of suggested projects after I emailed him my proposal) so I'll be putting a lot of effort into doing analysis of the output. And any generally useful code will be pushed to github.
EDIT: now to work on getting buyback and rebound and the like at the top of the rules text box. Or the sortcards.py. one of the two.
I have a set of changes currently under development that chop the rules text up into separate lines, exactly one 'ability' per line, and then reorder some of the lines where appropriate. So first enchant / equip, then french vanilla keywords like flying, first strike, etc. each on their own line, then standard text abilities and keywords like hellbent. And then a few things come last, like countertype and I think kicker. Because it's easier to start a line that says kicker when you know you have a clause that uses it than it is to start some random other line that will eventually end up in a kicker clause after you've generated the kicker cost. That's the theory, we'll see how it pans out.
Anyway, once that's implemented moving buyback and rebound around would be only a few lines of code, so you might want to hold off on hacking that for a few days.
Edit: and apparently I have a magical ability to always be the first poster on a new page, haha.
So having caught a cold or something I spend may waking hours (not many available today) working on my shell scripts. I seem to have worked out just about all the bugs, except that I don't know how to call the sampling script from inside another script. any advice on that? I tried copying and pasting the the "th sample{...}" but that just seemed to call the torch shell.
Just a quick for the record here, I don't remember having any issues calling torch from in a bash script. I just did it exactly the same as I would from the command line.
Is there anything strange going on with the path/environment or shell that's being used to run the script?
As a little side note, you can use a [code] tag inside the [spoiler] tag to put the text dumps in a monospace code font so the text tables retain their columns.
PS: the sortcards.py thought all but 10 cards were colorless and without mana.
Haha, working as intended. I'll have some real analytics scripts put up on the github at some point. And if you end up writing some code yourself, I'm happy to look at it add it to the currently almost empty set of tools in the scripts/ directory.
glad to hear it! I have a question: so I put your code on my RPi and the sorting script isn't working I get this:
pi@raspberrypi /mtgencode $ ./sortcards.py -h
./sortcards.py: line 1: import: command not found
./sortcards.py: line 2: import: command not found
./sortcards.py: line 3: import: command not found
from: can't read /var/mail/collections
./sortcards.py: line 8: syntax error near unexpected token `('
./sortcards.py: line 8: `def sortcards(cards):'
Your output makes it look like python doesn't know what an import statement is... interesting.
Have you tried running anything else? sorcards.py is out of date, probably doesn't work with modern formats, and really needs to be replaced with something better under scripts/, so it's maybe not the best test case.
The terminology 'uncasted' is actually my doing. If you turn both 'counter' and 'countered' into the same word 'uncast', then there are some situations where it's hard to figure out which one was which if you want to go back to normal magic-english terminology. So I just did the thing that was more technically sound, even if it makes for silly English.
First of all, I'm kinda sorta back to active development at the moment. I'm taking a machine learning class this semester, and now that I survived implementing gradient descent for the Lasso in the first homework, I'm trying to find a way to work this into a class project.
with the slow computer I am using an input file created by using the encode -e rmana >>input 5 times to ensure that each copy of the card had a different randomized mana cost. EDIT: Every time I continue reading the thread trying to catch up I remember that I should start using dropout, but I keep forgetting every time I set a new NN to train, Ill get there eventually, and it will probably make my results better.
I'll warn you that frequent repetition of the same cards is a recipe for overfitting. On a related note, yes, dropout is your friend.
I hear tell of Hardcast duplicating the corpus 10 or 100 times and his Checkpoints seem to come out just fine. I just wanted to make sure that the randomized mana symbols weren't all the same pattern if I used the -d command on the encode.py. I couldn't tell if they were so I hacked my own solution.
Duplicating cards in the input isn't a great idea, because the input file is the wrong place to duplicate the cards. What you really want to do is rerandomize the things that should be random for each training epoch, and possibly again each time you run the validation numbers, though that's a lot less important and of debatable value anyway.
Fortunately, the custom batcher for mtg-rnn does exactly this for you. In fact, if you give it the --randomize-mana option, it will randomize the order of the mana symbols in each cost used for training, so every single one will be unique and different from the same cost in other epochs, even if you didn't randomize the input file at all. You can go comb through the readmes if you want more detail on exactly how it works and what the rationale is.
Anyway, I should update things. The big items on my list are to regenerate the training data with the new BFZ cards (though I assume plenty of others have already done that) and see if there's any new updates from karpathy that I need to integrate into mtg-rnn. I have some work already done on a new version of the format that's probably better, but I don't want to push that out too quickly and break code people are using.
Any chance of getting another dump file... Or ten?
There are a few new 15MB MSE2 set files in this folder that I just recovered a few days ago after regaining access to one of my machines. They were generated with randomly ordered fields, so a lot of the cards will be a little messed up, but there's probably some interesting stuff in there as well.
Also, I forgot to tell you I am not using exactly the algorithm that is available in your github. The reason for that is because I've tried, but it seems like the input must be modeled for a Magic card, am I right? Therefore, I have used the neural net from Karpathy. Would you recommend me to change anything in his code?
Indeed, the code in the mtg-rnn repo will only work for input that shares certain key properties with text that looks like the encoding produced by mtgencode.
Specifically, mtg-rnn has a specialized batcher that tries to break the input up into a series of cards, and then randomize them across epochs so that as little information as possible is lost due to chopping into training batches. The effect hasn't been experimentally validated, but it works at least as well as the original and the idea seems reasonable, no? In any case, you could still use it for hearthstone cards if you structure the input in the right way. You would have to separate cards with pairs of newline characters, and not have longer sequences of newlines anywhere (though single newlines inside cards are fine). If you do that it should just work. If you separate fields within the cards with vertical bar | characters, you could use the randomize_fields option as well, just like for magic cards.
The output is pretty verbose, so you should have a good idea of how it's working. If you want to use it and run into any issues, feel free to post an issue on the github. I don't have access to any of my machines right now but I will starting next week.
Talcos, what's the status of getting stack-augmented networks to play nice with our format in Torch?
My internship just ended, so I'll be spending this week moving home. Once I'm back in the lab, I'm hoping to upgrade my Intel machine with a new GPU, and it would be nice to have something to run on it.
EDIT(6): YES! Partial victory. I got it working with just one stack. I have to go back in and add the loops so I can have multiple stacks. I needed to know that everything was working for just one before I tried more of them. And it's possible that I may have made a mistake somewhere, but what I can say is that it definitely completes every forward and backward pass without issue and I get performance numbers out so that's excellent. Hah. And it only took me around 48+ hours, lol.
Congrats! You're officially a better Torch7 hacker than I am.
I'm currently working on some improvements to the format, along the lines of this post. I'm collaborating with a friend to try and do some statistical analysis on the output to see if we can guide training parameters to something approaching optimal. Currently using R with RStudio, but I can easily convert the same data to arff format for weka. I'll provide info as things progress.
Hmmm, stack-rnns look pretty cool. I've got the whole long weekend to write some more code, so I'm thinking about pushing out a new rev of mtgencode, and trying to train my own MTG-oriented image recognition network with caffe.
How hard is it to implement this architecture on top of the existing char-rnn code? I'd try to do a port myself, but unfortunately I'm not that familiar with how these things actually work, so I'm not too confident in my ability to get it to work. I should read the papers more closely.
Also, has anyone used the ntm implementation here at all? If it's not too hard to set up, I'd be interested in training it up with the existing format just to see what it's capable of.
In the meantime, I should point out that the proper way to handle data is as card objects rather than text streams, and if you already have card objects then invoking the pretty formatting logic that decode.py uses is as simple as looping over the cards and having each one invoke its format() method with the right arguments. sortcards.py is a terrible example here, as it was written before card objects existed, so it isn't taking advantage of any of the modern infrastructure. I'd at least look at scripts/summarize.py, and how the indices are set up in lib/datalib.py, if you want an idea of how the infrastructure is supposed to work.
That said, having an output format that was just a plain HTML rendering that had prettified mana symbols like this forum would be totally awesome. If you build a generator for something like that, please, please give me the code so I can add it to mtgencode.
First, have you taken a look at Magic Set Editor 2? Using the -mse flag for decode.py to produce an mse-set file and the viewing it with MSE2 is pretty comfortable for looking at dumps, and it has some basic statistics and the ability to sort things if you're interested in a particular subset.
Second, I'm officially on the books to do a project on this work for my machine learning class (the prof apparently though text generation was so interesting a topic that he added karpathy's original blog post to the list of suggested projects after I emailed him my proposal) so I'll be putting a lot of effort into doing analysis of the output. And any generally useful code will be pushed to github.
Anyway, once that's implemented moving buyback and rebound around would be only a few lines of code, so you might want to hold off on hacking that for a few days.
Edit: and apparently I have a magical ability to always be the first poster on a new page, haha.
Is there anything strange going on with the path/environment or shell that's being used to run the script?
#!/bin/bash CHECKPOINT=${1} TEMP=${2} SIZE=${3} SEED=${4} NAME=${5} if [ -z "$TEMP" ]; then TEMP="0.8" fi if [ -z "$SIZE" ]; then SIZE="2000" fi if [ -z "$SEED" ]; then SEED="123" fi if [ -n "$NAME" ]; then NAME=${NAME}"." fi CMD="th sample.lua -gpuid 0 -temperature ${TEMP} -length ${SIZE} -seed ${SEED} ${CHECKPOINT}" TEE="tee ${CHECKPOINT}.output.${NAME}txt" echo "Sampling checkpoint: "${CHECKPOINT} echo " temperature: "${TEMP} echo " length: "${SIZE} echo " seed: "${SEED} echo " name: "${NAME} echo " command: "${CMD}" | "${TEE} ${CMD} | ${TEE}As a little side note, you can use a [code] tag inside the [spoiler] tag to put the text dumps in a monospace code font so the text tables retain their columns.
Haha, working as intended. I'll have some real analytics scripts put up on the github at some point. And if you end up writing some code yourself, I'm happy to look at it add it to the currently almost empty set of tools in the scripts/ directory.
Your output makes it look like python doesn't know what an import statement is... interesting.
Have you tried running anything else? sorcards.py is out of date, probably doesn't work with modern formats, and really needs to be replaced with something better under scripts/, so it's maybe not the best test case.
The terminology 'uncasted' is actually my doing. If you turn both 'counter' and 'countered' into the same word 'uncast', then there are some situations where it's hard to figure out which one was which if you want to go back to normal magic-english terminology. So I just did the thing that was more technically sound, even if it makes for silly English.
Duplicating cards in the input isn't a great idea, because the input file is the wrong place to duplicate the cards. What you really want to do is rerandomize the things that should be random for each training epoch, and possibly again each time you run the validation numbers, though that's a lot less important and of debatable value anyway.
Fortunately, the custom batcher for mtg-rnn does exactly this for you. In fact, if you give it the --randomize-mana option, it will randomize the order of the mana symbols in each cost used for training, so every single one will be unique and different from the same cost in other epochs, even if you didn't randomize the input file at all. You can go comb through the readmes if you want more detail on exactly how it works and what the rationale is.
Anyway, I should update things. The big items on my list are to regenerate the training data with the new BFZ cards (though I assume plenty of others have already done that) and see if there's any new updates from karpathy that I need to integrate into mtg-rnn. I have some work already done on a new version of the format that's probably better, but I don't want to push that out too quickly and break code people are using.
There are a few new 15MB MSE2 set files in this folder that I just recovered a few days ago after regaining access to one of my machines. They were generated with randomly ordered fields, so a lot of the cards will be a little messed up, but there's probably some interesting stuff in there as well.
Indeed, the code in the mtg-rnn repo will only work for input that shares certain key properties with text that looks like the encoding produced by mtgencode.
Specifically, mtg-rnn has a specialized batcher that tries to break the input up into a series of cards, and then randomize them across epochs so that as little information as possible is lost due to chopping into training batches. The effect hasn't been experimentally validated, but it works at least as well as the original and the idea seems reasonable, no? In any case, you could still use it for hearthstone cards if you structure the input in the right way. You would have to separate cards with pairs of newline characters, and not have longer sequences of newlines anywhere (though single newlines inside cards are fine). If you do that it should just work. If you separate fields within the cards with vertical bar | characters, you could use the randomize_fields option as well, just like for magic cards.
The output is pretty verbose, so you should have a good idea of how it's working. If you want to use it and run into any issues, feel free to post an issue on the github. I don't have access to any of my machines right now but I will starting next week.
My internship just ended, so I'll be spending this week moving home. Once I'm back in the lab, I'm hoping to upgrade my Intel machine with a new GPU, and it would be nice to have something to run on it.
Congrats! You're officially a better Torch7 hacker than I am.
I'm currently working on some improvements to the format, along the lines of this post. I'm collaborating with a friend to try and do some statistical analysis on the output to see if we can guide training parameters to something approaching optimal. Currently using R with RStudio, but I can easily convert the same data to arff format for weka. I'll provide info as things progress.
How hard is it to implement this architecture on top of the existing char-rnn code? I'd try to do a port myself, but unfortunately I'm not that familiar with how these things actually work, so I'm not too confident in my ability to get it to work. I should read the papers more closely.
Also, has anyone used the ntm implementation here at all? If it's not too hard to set up, I'd be interested in training it up with the existing format just to see what it's capable of.