First full spoiler with copyable forum code and hoverable names is here.
As a warning, if you open that file in a browser it will spin forever trying to preload 500,000 images from magiccards.info. I'll look into a fix for that (namely, loading the images on demand) and improvements to general readability. Any feedback is appreciated.
Alright, html spoilers are pushed to github, huge thanks to reimannsum for figuring most of it out. I'm not an html wizard, so if anyone has any complaints / suggestions I'm happy to hear them. There are a few tweaks I want to add myself, mostly a hover-over spoiler (like the [ card ] tag on here) to make the closest-card and closest-name info more useful.
An example of a spoiler is here, it's another 10MB dump from my most recent size-768 network sampled at temperature 0.8.
EDIT: I added some new technology to display visual spoilers on hovering over the names of the nearest cards by function or by name, as they appear in the creativity analysis. Currently I'm working on computing those statistics for my existing dumps (I have 60MB total) which is going to take a while even parallelized on my 8-core intel machine. I'll of course post all of those once they're done cooking. Hooray for strong processors - if only Python were as easy to accelerate on a GPU as the training of neural nets.
EDIT2: After several attempts, this version of my parallel computation of the distance metrics actually makes things faster! Here's the numbers from my i7-3770k (4 cores, 8 threads) processing 960 cards:
real 14m14.701s
user 14m13.521s
sys 0m0.284s
real 3m25.739s
user 24m52.410s
sys 0m3.636s
As you can see, we actually get about a 4.15x speedup in real time - better than linear! - due to the miracles of Intel's hyperthreading. The total user time increases by almost a factor of two, though, because the threads have to wait for each other to share a smaller number of physical cores.
Anyway, it's great to see numbers like that for real applications. Due to the intricacies of Python's Global Interpreter Lock and the overhead of creating multiple processes to get around it, I had to jump through some hoops to exploit parallelism as coarsely as possible. But, it is definitely cool to have a high level language that lets me express parallelism easily when I want it, even if there are some conditions attached to take full advantage of it.
things I would still like to do with this, but others can help (in order of importance, I feel)
Have sorting on the cards, probably borrowing heavily from the sortcards.py
have a hover panel with dumped data for each card, (vdump data, creativity info, maybe even a text box with forum code)
color the border of the according to their card border (gradient borders with CSS3)
Maybe generate a folder with files of specific types of cards in them so it can load faster (generate an Index file with a nav bar and subfiles display in an Iframe maybe)
Quick comments on these ideas.
The second is already implemented, just not as a hover panel. I just stick all of the additional info directly into the div, including creativity distance metrics with visual spoilers on hover, and a forum-encoded spoiler you can copy to share cards on here if you ask for it with -f. I think it's ok as it is, but I have many enormous monitors to display lots of large divs. If anyone has a better way to format things, I'm open to ideas.
The third is something I think I know how to do (not gradients :p) but I haven't gotten around to yet. Basically you make a few little style classes that change the color of the border and specify one of them based on what colors the card thinks it is.
Sorting and folder structure will both also be useful. Now that we've got human-readable output pretty much under control (the HTML spoilers are awesome to read) I'll be switching most of my attention to the very vague problem of evaluating the neural nets, which will certainly involve some sorting. As usual I'll post any major updates here, and probably stick the code in the scripts/ folder on github once it's stable.
so I just tried to whisper to my network with bodytext_prepend with the new format. It was prepending rebound in to the mane field. Does the sample_hs_v3.lua m ap th the labled fields?
No, I believe it just counts '|' characters and assumes the fields are ordered according to 'old' or 'noname'. In theory it will still work if you give it the name of the field in the right position, and remember to include the identifier number first or you'll confuse the network a lot. That's another thing that I should rewrite and maintain with the rest of the mtg-rnn codebase. I probably won't get around to that for a few days though.
Talcos, how hard is it to change that script? Ideally it could a setting to look for identifier characters like ('0', '1', '2', etc.) and then you could just tell it a sequence of identifiers and what you want it to insert for each one, and then it doesn't need to know the order ahead of time.
Oh, wait, there's also the fact that I gave that big talk to all the computer science undergraduate students about the project. They would have been more likely to seek out the github page. I mentioned you by name and showered praise on you. That might be why.
I thought of this as well, though I'd have expected the spike to be closer to when you gave the talk.
In any case, I certainly won't complain about the free publicity.
I like your new format, I have a question: the choice spells are formatted as [#ofchoices =choice =choice] how will this hold up to the new confluence cycle? will it need custom processing? maybe a each choice can be selected more than once\[3 =option =option =option]
Yes, the current syntax doesn't really have a great way of representing that. I suspect that given the different wording, the encoding will just leave it alone and not put it into [ # = choice ] syntax at all. The clever thing to do is probably make the choice syntax more english-friendly so there's some general way to attach modifiers to it.
EDIT: Talcos, have you been posting things on reddit again? The number of stars I have on mtgencode has gone way up in the past couple of days, and in the past that's been tied to the free advertising you provide on reddit.
So as I was reading through this extensive thread I remember reading some about card formatting and the position on the elements.
This is definitely an interesting area of study. Changing the order of the fields should have no effect on the training sampling process, and it should be pretty easy to do if you're comfortable opening up encode.py and decode.py and changing the arguments they give to the Card class. As long as the fields are all labeled, then decode.py should work fine no matter what order they're in. If you don't include the explicit labels, then you'll have to make sure decode.py knows the order or it will be confused. If you're having a hard time with the code, I can try to explain or provide some examples.
The mtgencode changeset that I'm working on right now (and that was used to create the two 10MB dumps I posted most recently) changes the field order to be better than it was before, but I've by no means done all of the necessary experimentation to figure out what's best.
Also, I'll note that if you want to be able to prime cards in any order, then you can train a network while randomizing the order of the (labeled) fields, and this works pretty well in practice, though on average cards are less likely to be well-formed and there are some interesting effects on remembering X and things like that.
EDIT: the latest mtgencode and mtg-rnn versions have been pushed to github. It's quite likely that something in there is a little broken, if anyone runs into any issues I'll try to get them fixed as quickly as possible.
They also seem to have utilized the semantic vectors of the responses themselves as a pre-filter, so they wouldn't converge too much - you want options that, while meaningful to the context, have some degree of variation.
Exactly. We want exactly the same things here - new cards that are similar enough to existing cards to be plausible, but different enough to be interesting.
I'm hoping I can address this a bit with my class project. The big question is, how do you define 'different enough,' and then once you do, how do you provide it? One option is just to build a filter, as Google is doing. I can do a similar thing with my own word2vec data, basically just taking all of the cards from a dump and separating out the ones that are too similar to existing cards or are obviously invalid. The hardest part there is trying to identify word salad.
Really what you want, though, is to somehow feed this information back into the training process, and I have no idea how to do that, other than by looking for parameters that produce good output. The thing with the word2vec distance is that although it does seem to give us a pretty good idea of how creative a card is, it's very expensive to compute.
I keep getting out of memory errors, Does that mean I should prune back my network?
I got one using a 512x3 network, I can (and am) running that on my 5 year old aleinware LAPTOP I'm confused.
with a 398x3 network I get 0.5s batch times!! I'M SO HAPPY!!
When training on a GPU, you're limited by the RAM that's available on the graphics card, not the whole RAM installed on your system. I bet your old laptop has at least 8 GB of RAM, while a 970 probably has like 2 or 3. My GTX Titan has 6GB, which is enough to train a 512x3 with seq_length of 200 and a batch size of 50. If I move up to a 768x3 network, I need to bump the batch size down to 40 or it doesn't fit.
There are a couple of things you can do to try to make it fit in your graphics card's memory - obviously reducing the size of the network is one, but you can also make the seq_length and batch_size smaller (though this will make it take longer to train). In general I find changing batch_size doesn't affect the results much (it just controls how many seq_length sized batches are trained on in at one time in parallel).
You can run 'nvidia-smi' to see the resource utilization of processes running on your GPU, and what the limits are, to get an idea of how things are working. You can even bring up a small terminal on the side somewhere and run 'nvidia-smi -l 5' to make it rerun every 5 seconds, sort of like 'top' for GPUs.
I would recommend installing CUDA before you do anything else, and only the version that you get from the NVIDIA developers website.
That is to say, look at their installation instructions, and install their .deb package or whatever, and then don't touch it at all after that. You shouldn't be using apt-get to install any graphics-driver related software at all.
whineless response (rare) U
instant ~ arcane
return target creature to its owner's hand. if @ was kicked, remove a % counter from that creature.
kicker 2U
## That name though. Seriously, this network loves weird names.
Haha, those names are pretty great. I think the craziness there is mostly due to the fact that I sampled at a temperature of 1.0. If people have specific requests, I can produce more data, my machines are currently sitting idle. I'm hoping to add multiprocessing to the word2vec analysis code so that I can run it 8x faster on my other big Intel core.
EDIT: Some truly irresistible cards from the 768 dump:
bone (rare) 1B
instant
destroy target creature or planeswalker.
## That name though...
armored assault (common) 1B
enchantment B, sacrifice a creature: untap target permanent.
## Cool idea other than the rarity and name.
worm wand (uncommon) 1
artifact
whenever an opponent shuffles his or her library, you may pay 1. if you do, draw a card.
## Of course.
akroan seggend (uncommon)
_NOCOST_
land T, pay 1 life: add GU to your mana pool. if that mana is spent on a creature spell, it gets +3/+3 and has first strike.
## Seems a bit too strong.
academy retribution (rare)
_NOCOST_
land T: add 1 to your mana pool. 10: choose a color. add one mana of that color to your mana pool.
## Seems a bit too weak, outside of use with infinite mana combos. Correct (though short distance) choose and chosen though.
animan mentor (rare)
_NOCOST_
land T: add 1 to your mana pool. 10, T, sacrifice @: search the chosen player's library for up to seven cards, reveal those cards, and put them into your hand. then shuffle your library.
## Here's a slightly better 10-mana ability on a land. Only slightly messed up the choosing of a player, and of course the usual issue of having other people's cards in your hand.
rendword pack (common) 2(U/R)
instant
convoke
heroic ~ whenever you cast a spell that targets @, put it on top of its owner's library.
## Reasonable heroic nonpermanent. Can anybody think of a way to break this?
earthwapmer (mythic rare) 2(U/R)U
creature ~ nightmare dragon
flying
whenever @ attacks, return a blue creature you control to its owner's hand.
(6/6)
## I kinda like it.
bone spring (common) 2(U/R)
creature ~ sliver
sliverpy 3R
when @ enters the battlefield, return two target lands an opponent controls to their owner's hand.
(2/2)
## This would be miserable to play against. I wonder what sliverpy does.
victiminity (common) 2BU
sorcery
as an additional cost to cast @, sacrifice a creature.
gain control of target creature until end of turn.
draw a card.
## Fascinating. It doesn't give haste, so you'd have to be more creative than just attacking with whatever it was.
terparia (rare) 3(W/B)(W/B)
enchantment
as @ enters the battlefield, choose a creature type.
all nontoken creatures of the chosen type get -2/-0.
creatures can't attack.
all attacking player chooses and can block any number of creatures.
## Hmmm. I think it should have stopped a bit sooner.
humble backace (rare) 2RR
creature ~ human knight
whenever an artifact enters the battlefield under your control, you may sacrifice it. if you do, @ deals 5 damage to target creature or player.
(2/2)
## In case you just wanted Shrapnel Blasts.
So I have uploaded my cron scripts and forked mtgencode to add support for decoding to a pretty html document; although I'm not sure how to include the image files (I'm still new to git).
Thanks so much for the pull request! I'll get that merged this weekend, then try to get my own changes and the new version of the format pushed out. I generated a few networks to test it, and I'm seeing promising results for both kicker and X / XX in manacosts, using my custom batcher and the latest char-rnn code from karpathy. It's still hit or miss, but that means there are some hits where the network actually figures it out. I'll post both annotated dumps and checkpoints as soon as the changes go live on github.
EDIT: new checkpoints / dumps are available on my google drive. The format is a little different, but older versions of mtgencode should be able to understand it just fine because the fields are labeled. I've got two 10MB dumps, complete with full MSE2 sets that even include all of the creativity comparison info against existing cards (these took almost a whole day to compute). For space reasons I'm taking down the old stuff from my google drive folder, you can bother me if you still need some of it.
Opening encoded card file: ../mtg-rnn-staging/cv/dev/std-512-lines/lm_lstm_epoch50.00_0.2673.t7.output.txt
====================
60812 valid cards, 230 invalid cards.
61042 cards parsed, 88 failed to parse
--------------------
57942 unique card names
--------------------
6 represented colors (including colorless as 'A'), 32 combinations
Breakdown by color:
A B G R U W
6955 14643 13217 10320 12405 11448
Breakdown by number of colors:
0 1 2 3 4 5
6955 47041 6254 696 84 12
--------------------
45 unique card types, 52 combinations
Breakdown by type:
creature instant enchantment sorcery artifact land tribal planeswalker hand artifact.4
31550 8815 8628 6901 5014 1926 214 156 5 4
--------------------
953 unique subtypes, 2199 combinations
-- Popular subtypes: --
human 6387
aura 3540
soldier 2623
warrior 1796
wizard 1763
shaman 1649
spirit 1591
elemental 1565
elf 1338
zombie 1231
-- Top combinations: --
aura 3498
human soldier 1442
spirit 1148
human wizard 983
elemental 967
beast 931
human cleric 599
dragon 568
insect 553
zombie 542
--------------------
27 unique supertypes, 28 combinations
Breakdown by supertype:
legendary snow world basic megaace angel incasper megance battalion engant
2361 181 54 18 1 1 1 1 1 1
--------------------
15 different CMCs, 4228 unique mana costs
Breakdown by CMC:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 15
2099 7180 12227 13877 11624 7356 3984 1729 653 202 76 22 8 3 2
-- Popular mana costs: --
none 1950
{G} 1427
{W} 1336
{U} 1247
{3} 1195
{B} 1182
{1}{G} 1136
{2} 1058
{1}{U} 994
{1}{W} 958
--------------------
170 unique p/t combinations
Largest power: 21, largest toughness: 85
-- Popular p/t values: --
1/1 5969
2/2 5782
3/3 2774
2/1 2407
4/4 1872
2/3 1029
3/2 1002
5/5 962
1/2 959
3/4 643
--------------------
Loyalty values:
3 57
4 45
5 18
2 8
6 5
\ashand 1
s 1
7 1
legendary 1
is the converted mana cost 3 or less. 1
--------------------
Card text ranges from 0 to 633 characters in length
Card text ranges from 0 to 9 lines
-- Line counts by frequency: --
1 28221
2 22469
3 7260
4 1572
0 1105
5 325
6 74
7 13
8 2
9 1
====================
********************
Overview of indices:
Index Name Keys Total Members
by_cmc 15 61042
by_power 24 31572
by_loyalty 11 139
by_textlen 477 61042
by_cost 4228 61042
by_supertype 28 2627
by_color_inclusive 6 68988
by_name 57942 61026
by_pt 170 31599
by_supertype_inclusive 27 2637
by_subtype_inclusive 953 52368
by_toughness 34 31572
by_type 52 61041
by_textlines 10 61042
by_type_inclusive 45 63257
by_color 32 61042
by_color_count 6 61042
by_subtype 2199 36276
********************
Shortest Cardname: (2)
un
Longest Cardname: (40)
quest for the luntless polley battlesing
-- Most duplicated names: --
flesh 27
sandstorm 26
riptide 20
dismiss 17
carrion 15
aurochs 15
excavation 15
seeker 15
reverence 13
urborg 13
--------------------
Longest card type: (24)
hribat artifact creature
Longest subtype: (29)
worm\whenever tarret artifact
Longest supertype: (13)
snow creature
--------------------
Longest mana cost: (26)
{6}{B}{W}{B}{U}{U}{B}{B}{B}{W}
|creature||leviathan||&^^^^^^^^^^/&^^^^^^^^^^|trample\double strike\you may pegaas the top card of your library.|{^^BBWWBBUUUU^^^BBBB^BBWW}|Y|avengra, kor spirit|
Largest cmc: (15)
{6}{B}{W}{B}{U}{U}{B}{B}{B}{W}
|creature||leviathan||&^^^^^^^^^^/&^^^^^^^^^^|trample\double strike\you may pegaas the top card of your library.|{^^BBWWBBUUUU^^^BBBB^BBWW}|Y|avengra, kor spirit|
--------------------
Largest creature power: 21
|artifact creature||golem||&^^^^^^^^^^^^^^^^^^^^^/&^^^^^^^^^^^^^^^^^|trample\@ has flying and with untapped creatures and thrull of your choice until end of turn.|{^^^^^^}|N|vermicale|
Largest creature toughness: 1\and and defending player can't cast spells with the same name as thu exiled bidder.
|creature||human rogue||&^^^/&^\and and defending player can't cast spells with the same name as thu exiled bidder.||{UUUU}|A|moralent field|
--------------------
Most lines of text in a card: 9
|creature||minotaur shaman||&^^/&^^|level up {^GG} \level &^~&^^^\&^^/&^^^^\level &^^^^^^~\&^^^^/&^^^^^^\level &^^^^~\&^^^^^/&^^^^^^^^^\first strike\banding|{^WW^}|O|nestinate|
Most chars in a card text: 633
|creature||human wizard||&^/&^^|{UU}, T: prevent the next &^ damage that would be dealt to target creature or player this turn.\T can't be blocked by walls.\each creature gets +&/+&^ counters at the beginning of the next end step. activate this ability only once each turn.\threshold ~ as long as seven or more cards are in your graveyard, @ has "when @ enters the battlefield, search your library for any number of cards with the same name as that creature. reveal them. put the revealed cards into your hand instead of any or the number of allies you control. then shuffle your library.\{^^BBBBBB}: exile @. return it to the battlefield under its owner's control.|{^WW^}|A|cruulcers' herald|
--------------------
There were 230 invalid cards.
Not summarizing.
--------------------
There were 88 unparsed cards.
Not summarizing.
====================
Opening encoded card file: ../mtg-rnn-staging/cv/dev/std-768-lines/lm_lstm_epoch49.83_0.2636.t7.output.txt
====================
60956 valid cards, 140 invalid cards.
61096 cards parsed, 46 failed to parse
--------------------
59033 unique card names
--------------------
6 represented colors (including colorless as 'A'), 32 combinations
Breakdown by color:
A B G R U W
8078 11312 10901 14667 10564 11663
Breakdown by number of colors:
0 1 2 3 4 5
8078 47472 5066 423 51 6
--------------------
45 unique card types, 37 combinations
Breakdown by type:
creature enchantment instant sorcery artifact land tribal planeswalker until rat
33102 7750 7030 6863 5968 2218 124 89 5 4
--------------------
855 unique subtypes, 2335 combinations
-- Popular subtypes: --
human 7617
aura 3424
soldier 2764
wizard 2232
warrior 2172
beast 1619
elemental 1545
cleric 1504
shaman 1493
spirit 1390
-- Top combinations: --
aura 3355
human soldier 1464
beast 1254
human wizard 1197
spirit 1155
equipment 973
human cleric 954
elemental 937
dragon 685
angel 609
--------------------
13 unique supertypes, 14 combinations
Breakdown by supertype:
legendary world snow basic lo nog elemental snot wegendary now
2523 133 119 16 1 1 1 1 1 1
--------------------
16 different CMCs, 3708 unique mana costs
Breakdown by CMC:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2229 6773 11886 13894 11502 7567 4159 1912 742 288 88 39 12 2 2 1
-- Popular mana costs: --
none 2064
{2} 1590
{3} 1568
{W} 1397
{R} 1337
{1}{W} 1136
{G} 1044
{B} 1036
{4} 977
{1}{R} 971
--------------------
198 unique p/t combinations
Largest power: 17, largest toughness: 155
-- Popular p/t values: --
2/2 5971
1/1 5898
3/3 3161
2/1 2298
4/4 1883
3/2 1125
1/2 1064
5/5 1041
2/3 930
4/3 724
--------------------
Loyalty values:
3 45
4 27
2 6
6 4
5 4
s 1/1 1
--------------------
Card text ranges from 0 to 754 characters in length
Card text ranges from 0 to 10 lines
-- Line counts by frequency: --
1 30396
2 20956
3 6570
0 1428
4 1358
5 281
6 67
7 27
8 10
10 2
====================
********************
Overview of indices:
Index Name Keys Total Members
by_cmc 16 61096
by_power 27 33089
by_loyalty 6 87
by_textlen 495 61096
by_cost 3708 61096
by_supertype 14 2795
by_color_inclusive 6 67185
by_name 59033 61076
by_pt 198 33111
by_supertype_inclusive 13 2800
by_subtype_inclusive 855 55763
by_toughness 43 33089
by_type 37 61095
by_textlines 11 61096
by_type_inclusive 45 63199
by_color 32 61096
by_color_count 6 61096
by_subtype 2335 38322
********************
Shortest Cardname: (2)
bo
Longest Cardname: (94)
snow~covered ~ basic lands are places on the chosen color would deal &^^^ or less you control.
-- Most duplicated names: --
aurochs 25
abomination 12
golgari guildmage 11
experiment one 11
choke 10
seraph 9
armistice 9
kookus 9
replica 9
mobilize 8
--------------------
Longest card type: (147)
until end of turn and cast target attacking creature exiled with @.\{^^WW^}: creatures you control get +&^^^/+& and gain trample until end of turn.
Longest subtype: (29)
human barbarian werewolf monk
Longest supertype: (19)
legendary elemental
--------------------
Longest mana cost: (22)
{R}{W}{G}{R}{W}{G}{W}{B}{G}{U}
|creature||elemental cat||&^^^^^^^/&^^^^|haste\cascade\creature spells your opponents cast that share a card type with enchanted creature.|{RRWWGGRRWWGGWWBBGGUU}|A|metamorphose|
Largest cmc: (15)
{11}{G}{W}{U}{G}
|instant|||||target creature gets -&^^^^^^^^^^^^^^^^/-&^^^^^^^^^^^^^^^^ until end of turn.|{^GG^^^^^^WWUU^GG^^^}|A|tawon weatherward|
--------------------
Largest creature power: 17
|creature||leviathan||&^^^^^^^^^^^^^^^^^/&^^^^^^^^^^^^^^^^^|when @ dies, exile it instead of put into your graveyard.|{UU^^^UUUUUU}|A|catacomb ghost|
Largest creature toughness: 1\when @ dies, target player reveals cards from the top of his or her library until a creature card named @. put one of those cards on top of your library.
|creature||human warrior||&^^^/&^\when @ dies, target player reveals cards from the top of his or her library until a creature card named @. put one of those cards on top of your library.||{BB^}|O|satyr conscripts|
--------------------
Most lines of text in a card: 10
|creature||skeleton||&^^/&^|level up {RR^} \level &^~&^^^\&^^^^^^^/&^^^^^\level &^^ &^^^\&^^^^^/&^^^\level &^^^^~\&^^^^^/&^^^\flying\protection from black.\{^^BBUU}, T: destroy target warrior with converted mana cost &^^^ or less.|{^^BB}|N|frostwood invocation|
Most chars in a card text: 754
|artifact|||||landfall ~ whenever a land enters the battlefield under your control, [&^ = you gain &^ life. = you gain &^^ life. = target opponent discards two cards. = exile the top three cards of target player's library. put a card for mana pool in that player's mana pool rathor that player, reveal the top card of your library, put all other revealed on the bottom of your library in any order.\T: until end of turn, lands you control enchanted land becomes a &^^/&^^ blue beast creature until end of turn.\whenever a mountain an opponent controls with three % counters on it, sacrifice it.\{^^^^}, T: copy @ in target opponent's graveyard.\whenever a player discards a card, you may draw a card. if you do, you may play that card for as long as it remains exiled.|{UU^UU^^}|A|pharemind preyence|
--------------------
There were 140 invalid cards.
Not summarizing.
--------------------
There were 46 unparsed cards.
Not summarizing.
====================
I'm reluctant to push out mtgencode just yet, as the new changes will effectively break the old format unless you give it some compatibility flags, and I'm not quite confident that all of the new stuff is correct. I'll try to get it out later this weekend.
Ok, I vaguely remember that I did that origionally and that's when I couldn't even get torch to begin with, but I'll give it a second shot. I'm planning on reinstalling tonight.
so I'm still a little shaky as to how to handle the Card class, but I think I'll just steal your code that cuts the text into the Cards array and then work card at a time as plain-text (aka. card.format() and then make my modifications). I'll post it as a fork for you to look over; I may make a batch file as well so that each html file has the same src location for the image files. or that could be a variable in the top of the file so people can easily change it looks for the images. That's one of the problems is that the html file isn't standalone.
That sounds fine. If you write the text transformation, I can easily move it to the right place in the Card class code. As you point out, the hard part is figuring out how to manage the other assets that will need to go along with the html to render pretty mana symbols - unfortunately I don't have any suggestions for you there, my experience with html is quite limited.
What are your thoughts on making a flag where the user inputs a list of field names and the encoder, (and there for decoder) processes only those field of the card so we can easily try a "no card names" approach, or a "flavor text first" approach?
This is already sort of implemented, as you can pass an arbitrary list of fields and labels when encoding Card class objects or creating them from raw text. I don't support getting those arguments directly from the command line, but the rational there was that if you were going to go through the work to come up with your own format, then it wouldn't be too much extra effort to just directly write the python code to handle it, for example by adding another format in encode.py and decode.py.
Is anyone interested in my scripts for getting multiple computers to all dump samples into one computer? with another script that the central dump computer runs to decode said outputs? or am I the only one with old computers that I can let run training for days, to harvest larger, slower training setups?
I'm mildly curious if it's not too hard for you to provide them. I have a number of machines, though generally my toolchain is to just train / generate large output dumps quickly with my GPU, and then scp them around if I want them somewhere else.
Aha, I think I know what your problem is. Using the package manager is great for most things, when it's ok to use stable versions of software that's a few years out of date. However, for gpu drivers this is probably not a good idea. What it looks like happened is that apt gave you some 5.x version of CUDA which seems to be compatible with Torch but actually isn't - looking back at your error log, it was complaining about the version being 5.2 or something, which makes sense because it requires 5.5 or higher.
I'd recommend carefully purging all nvidia driver packages from your system (or reinstalling the OS if you want to be absolutely sure) and then downloading the linux version of the cuda drivers directly from NVIDIA here. That will give you version 7.5 of cuda, which should be compatible with Torch.
To confirm what version you have, run those commands I used above ('cat /proc/driver/nvidia/version' and 'nvcc --version') and see what comes out.
so you're recommendation for the card parser is to basically add it as a flag in the decode.py? or even just modify the cardlib.py to include an additional format option?
Yes, the right way to do this is to add two things: a flag for the card class's format method in cardlib.py that makes it produce the right html, and then a flag for decode that causes it to pass that flag when formatting the cards and also emit the necessary css / other wrappings for a full, viewable html spoiler.
What command can I run to easily get you that?
ubuntu 15.04
gtx 970
fresh install.
That's mostly what I wanted. And I'm guessing an Intel CPU?
For the record, I'm running:
Ubuntu 14.04 LTS
Intel core i7 3770k @ 4.4GHz, 32GB DDR3, terrible, complicated disk situation dual booting with Windows 8
GTX Titan
My nvidia driver version appears to be the following:
$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 346.96 Sun Aug 23 22:29:21 PDT 2015
GCC version: gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)
And my cuda version: (apparently it's slightly out of date, the latest is 7.5 I believe)
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Mon_Feb_16_22:59:02_CST_2015
Cuda compilation tools, release 7.0, V7.0.27
I've installed both the cuda driver from the nvidia website here, as well as nvidia's proprietary neural net libraries cudnn, which are NOT required to run Torch. You have to jump through some hoops signing up as a developer to get your hands on them, but other than that they're free. Probably not worth your time, I had torch working perfectly fine on my GPU before I installed them, there was just some other code for image processing that wanted them.
EDIT: one thought - what does your path look like?
I have the following in my ~/.profile to set up the path on my system.
# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.
# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022
# if running bash
if [ -n "$BASH_VERSION" ]; then
# include .bashrc if it exists
if [ -f "$HOME/.bashrc" ]; then
. "$HOME/.bashrc"
fi
fi
# guard path manipulation to prevent double prepending by tmux
if [ "$PATHS" != "true" ]; then
export PATHS="true"
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
PATH="$HOME/bin:$PATH"
fi
# caffe
PYTHONPATH=/home/me/private/img/caffe/python:$PYTHONPATH
# CUDA 7.0
PATH=/usr/local/cuda-7.0/bin:$PATH
LD_LIBRARY_PATH=/usr/local/cuda-7.0/lib64:$LD_LIBRARY_PATH
# Added automatically by torch-dist
PATH=/home/me/private/torch/install/bin:$PATH
LD_LIBRARY_PATH=/home/me/private/torch/install/lib:$LD_LIBRARY_PATH
DYLD_LIBRARY_PATH=/home/me/private/torch/install/lib:$DYLD_LIBRARY_PATH
export PATH
export LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH
export PYTHONPATH
fi
If you're not familiar with how Linux and bash work, I can try to explain all that in detail. Basically, if you do 'echo $PATH' from a terminal and there isn't something in the output that looks like '/usr/local/cuda-7.something/bin' then something is wrong, and none of the cuda binaries will be available.
That said, if your issue is that nvcc exists but doesn't want to compile anything for you because it doesn't understand the system, then I highly doubt the fault is with your system path. Though an incorrect LD_LIBRARY_PATH could have something to do with that, I have no idea.
As a warning, if you open that file in a browser it will spin forever trying to preload 500,000 images from magiccards.info. I'll look into a fix for that (namely, loading the images on demand) and improvements to general readability. Any feedback is appreciated.
An example of a spoiler is here, it's another 10MB dump from my most recent size-768 network sampled at temperature 0.8.
EDIT: I added some new technology to display visual spoilers on hovering over the names of the nearest cards by function or by name, as they appear in the creativity analysis. Currently I'm working on computing those statistics for my existing dumps (I have 60MB total) which is going to take a while even parallelized on my 8-core intel machine. I'll of course post all of those once they're done cooking. Hooray for strong processors - if only Python were as easy to accelerate on a GPU as the training of neural nets.
EDIT2: After several attempts, this version of my parallel computation of the distance metrics actually makes things faster! Here's the numbers from my i7-3770k (4 cores, 8 threads) processing 960 cards:
Anyway, it's great to see numbers like that for real applications. Due to the intricacies of Python's Global Interpreter Lock and the overhead of creating multiple processes to get around it, I had to jump through some hoops to exploit parallelism as coarsely as possible. But, it is definitely cool to have a high level language that lets me express parallelism easily when I want it, even if there are some conditions attached to take full advantage of it.
EDIT3: Quick comments on these ideas.
The second is already implemented, just not as a hover panel. I just stick all of the additional info directly into the div, including creativity distance metrics with visual spoilers on hover, and a forum-encoded spoiler you can copy to share cards on here if you ask for it with -f. I think it's ok as it is, but I have many enormous monitors to display lots of large divs. If anyone has a better way to format things, I'm open to ideas.
The third is something I think I know how to do (not gradients :p) but I haven't gotten around to yet. Basically you make a few little style classes that change the color of the border and specify one of them based on what colors the card thinks it is.
Sorting and folder structure will both also be useful. Now that we've got human-readable output pretty much under control (the HTML spoilers are awesome to read) I'll be switching most of my attention to the very vague problem of evaluating the neural nets, which will certainly involve some sorting. As usual I'll post any major updates here, and probably stick the code in the scripts/ folder on github once it's stable.
Talcos, how hard is it to change that script? Ideally it could a setting to look for identifier characters like ('0', '1', '2', etc.) and then you could just tell it a sequence of identifiers and what you want it to insert for each one, and then it doesn't need to know the order ahead of time.
In any case, I certainly won't complain about the free publicity.
EDIT: Talcos, have you been posting things on reddit again? The number of stars I have on mtgencode has gone way up in the past couple of days, and in the past that's been tied to the free advertising you provide on reddit.
The mtgencode changeset that I'm working on right now (and that was used to create the two 10MB dumps I posted most recently) changes the field order to be better than it was before, but I've by no means done all of the necessary experimentation to figure out what's best.
Also, I'll note that if you want to be able to prime cards in any order, then you can train a network while randomizing the order of the (labeled) fields, and this works pretty well in practice, though on average cards are less likely to be well-formed and there are some interesting effects on remembering X and things like that.
EDIT: the latest mtgencode and mtg-rnn versions have been pushed to github. It's quite likely that something in there is a little broken, if anyone runs into any issues I'll try to get them fixed as quickly as possible.
I'm hoping I can address this a bit with my class project. The big question is, how do you define 'different enough,' and then once you do, how do you provide it? One option is just to build a filter, as Google is doing. I can do a similar thing with my own word2vec data, basically just taking all of the cards from a dump and separating out the ones that are too similar to existing cards or are obviously invalid. The hardest part there is trying to identify word salad.
Really what you want, though, is to somehow feed this information back into the training process, and I have no idea how to do that, other than by looking for parameters that produce good output. The thing with the word2vec distance is that although it does seem to give us a pretty good idea of how creative a card is, it's very expensive to compute.
There are a couple of things you can do to try to make it fit in your graphics card's memory - obviously reducing the size of the network is one, but you can also make the seq_length and batch_size smaller (though this will make it take longer to train). In general I find changing batch_size doesn't affect the results much (it just controls how many seq_length sized batches are trained on in at one time in parallel).
You can run 'nvidia-smi' to see the resource utilization of processes running on your GPU, and what the limits are, to get an idea of how things are working. You can even bring up a small terminal on the side somewhere and run 'nvidia-smi -l 5' to make it rerun every 5 seconds, sort of like 'top' for GPUs.
That is to say, look at their installation instructions, and install their .deb package or whatever, and then don't touch it at all after that. You shouldn't be using apt-get to install any graphics-driver related software at all.
Haha, those names are pretty great. I think the craziness there is mostly due to the fact that I sampled at a temperature of 1.0. If people have specific requests, I can produce more data, my machines are currently sitting idle. I'm hoping to add multiprocessing to the word2vec analysis code so that I can run it 8x faster on my other big Intel core.
EDIT: Some truly irresistible cards from the 768 dump:
bone (rare)
1B
instant
destroy target creature or planeswalker.
## That name though...
armored assault (common)
1B
enchantment
B, sacrifice a creature: untap target permanent.
## Cool idea other than the rarity and name.
worm wand (uncommon)
1
artifact
whenever an opponent shuffles his or her library, you may pay 1. if you do, draw a card.
## Of course.
akroan seggend (uncommon)
_NOCOST_
land
T, pay 1 life: add GU to your mana pool. if that mana is spent on a creature spell, it gets +3/+3 and has first strike.
## Seems a bit too strong.
academy retribution (rare)
_NOCOST_
land
T: add 1 to your mana pool.
10: choose a color. add one mana of that color to your mana pool.
## Seems a bit too weak, outside of use with infinite mana combos. Correct (though short distance) choose and chosen though.
animan mentor (rare)
_NOCOST_
land
T: add 1 to your mana pool.
10, T, sacrifice @: search the chosen player's library for up to seven cards, reveal those cards, and put them into your hand. then shuffle your library.
## Here's a slightly better 10-mana ability on a land. Only slightly messed up the choosing of a player, and of course the usual issue of having other people's cards in your hand.
rendword pack (common)
2(U/R)
instant
convoke
heroic ~ whenever you cast a spell that targets @, put it on top of its owner's library.
## Reasonable heroic nonpermanent. Can anybody think of a way to break this?
earthwapmer (mythic rare)
2(U/R)U
creature ~ nightmare dragon
flying
whenever @ attacks, return a blue creature you control to its owner's hand.
(6/6)
## I kinda like it.
bone spring (common)
2(U/R)
creature ~ sliver
sliverpy 3R
when @ enters the battlefield, return two target lands an opponent controls to their owner's hand.
(2/2)
## This would be miserable to play against. I wonder what sliverpy does.
victiminity (common)
2BU
sorcery
as an additional cost to cast @, sacrifice a creature.
gain control of target creature until end of turn.
draw a card.
## Fascinating. It doesn't give haste, so you'd have to be more creative than just attacking with whatever it was.
terparia (rare)
3(W/B)(W/B)
enchantment
as @ enters the battlefield, choose a creature type.
all nontoken creatures of the chosen type get -2/-0.
creatures can't attack.
all attacking player chooses and can block any number of creatures.
## Hmmm. I think it should have stopped a bit sooner.
humble backace (rare)
2RR
creature ~ human knight
whenever an artifact enters the battlefield under your control, you may sacrifice it. if you do, @ deals 5 damage to target creature or player.
(2/2)
## In case you just wanted Shrapnel Blasts.
EDIT: new checkpoints / dumps are available on my google drive. The format is a little different, but older versions of mtgencode should be able to understand it just fine because the fields are labeled. I've got two 10MB dumps, complete with full MSE2 sets that even include all of the creativity comparison info against existing cards (these took almost a whole day to compute). For space reasons I'm taking down the old stuff from my google drive folder, you can bother me if you still need some of it.
That sounds fine. If you write the text transformation, I can easily move it to the right place in the Card class code. As you point out, the hard part is figuring out how to manage the other assets that will need to go along with the html to render pretty mana symbols - unfortunately I don't have any suggestions for you there, my experience with html is quite limited.
This is already sort of implemented, as you can pass an arbitrary list of fields and labels when encoding Card class objects or creating them from raw text. I don't support getting those arguments directly from the command line, but the rational there was that if you were going to go through the work to come up with your own format, then it wouldn't be too much extra effort to just directly write the python code to handle it, for example by adding another format in encode.py and decode.py.
I'm mildly curious if it's not too hard for you to provide them. I have a number of machines, though generally my toolchain is to just train / generate large output dumps quickly with my GPU, and then scp them around if I want them somewhere else.
I'd recommend carefully purging all nvidia driver packages from your system (or reinstalling the OS if you want to be absolutely sure) and then downloading the linux version of the cuda drivers directly from NVIDIA here. That will give you version 7.5 of cuda, which should be compatible with Torch.
To confirm what version you have, run those commands I used above ('cat /proc/driver/nvidia/version' and 'nvcc --version') and see what comes out.
Yes, the right way to do this is to add two things: a flag for the card class's format method in cardlib.py that makes it produce the right html, and then a flag for decode that causes it to pass that flag when formatting the cards and also emit the necessary css / other wrappings for a full, viewable html spoiler.
For the record, I'm running:
Ubuntu 14.04 LTS
Intel core i7 3770k @ 4.4GHz, 32GB DDR3, terrible, complicated disk situation dual booting with Windows 8
GTX Titan
My nvidia driver version appears to be the following:
And my cuda version: (apparently it's slightly out of date, the latest is 7.5 I believe)
I've installed both the cuda driver from the nvidia website here, as well as nvidia's proprietary neural net libraries cudnn, which are NOT required to run Torch. You have to jump through some hoops signing up as a developer to get your hands on them, but other than that they're free. Probably not worth your time, I had torch working perfectly fine on my GPU before I installed them, there was just some other code for image processing that wanted them.
EDIT: one thought - what does your path look like?
I have the following in my ~/.profile to set up the path on my system.
That said, if your issue is that nvcc exists but doesn't want to compile anything for you because it doesn't understand the system, then I highly doubt the fault is with your system path. Though an incorrect LD_LIBRARY_PATH could have something to do with that, I have no idea.