For those getting errors downloading, you need to start from an empty database.
I have successfully downloaded all sets except Release Promos, which fails at Brisela (separate post coming for that).
Note: I skipped all the "online" sets (cube, masters edition, vintage masters, tempest remastered), as I am using GEx to maintain my inventory spreadsheet.
When attempting to open the "Export" menu, unhandled exception error:
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.InvalidCastException: Operator '=' is not defined for type 'DBNull' and type 'Boolean'.
at Microsoft.VisualBasic.CompilerServices.Operators.ConditionalCompareObjectEqual(Object Left, Object Right, Boolean TextCompare)
at Gatherer_Extractor.FenetrePrincipale.ExporterToolStripMenuItem_DropDownOpening(Object sender, EventArgs e)
at System.Windows.Forms.ToolStripDropDownItem.OnDropDownShow(EventArgs e)
at System.Windows.Forms.ToolStripMenuItem.OnDropDownShow(EventArgs e)
at System.Windows.Forms.ToolStripDropDownItem.ShowDropDownInternal()
at System.Windows.Forms.ToolStripMenuItem.OnMenuAutoExpand()
at System.Windows.Forms.MenuTimer.OnTick(Object sender, EventArgs e)
at System.Windows.Forms.Timer.OnTick(EventArgs e)
at System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.
Error with Release Promo Brisela, Voice of Nightmares:
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.ArgumentException: Argument 'Start' must be greater than zero.
at Microsoft.VisualBasic.Strings.InStr(Int32 Start, String String1, String String2, CompareMethod Compare)
at Gatherer_Extractor.Extract_Fct.Get_Logs_side(String Logs, String websource)
at Gatherer_Extractor.Extract_Fct.Get_Data(DataRow TempRow, String Logs, String LogsSF, String websource, String lang)
at Gatherer_Extractor.FenetrePrincipale.Treatment_MAJ(Int32 nbre_tache, BackgroundWorker worker, DoWorkEventArgs e)
at Gatherer_Extractor.FenetrePrincipale.BWkMAJ_DoWork(Object sender, DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.
After successfully saving those downloaded sets to CSV, when attempting to reload that database, same error received as when trying to Export:
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.InvalidCastException: Operator '=' is not defined for type 'DBNull' and type 'Boolean'.
at Microsoft.VisualBasic.CompilerServices.Operators.ConditionalCompareObjectEqual(Object Left, Object Right, Boolean TextCompare)
at Gatherer_Extractor.FenetrePrincipale.Perform_Test_Fully_Handled_Sets()
at Gatherer_Extractor.FenetrePrincipale.BGW_OpenBDD_RunWorkerCompleted(Object sender, RunWorkerCompletedEventArgs e)
at System.ComponentModel.BackgroundWorker.OnRunWorkerCompleted(RunWorkerCompletedEventArgs e)
at System.ComponentModel.BackgroundWorker.AsyncOperationCompleted(Object arg)
I confirm that you must start from an empty database. Otherwise, there will be conflicting IDs or unknown IDs with outdated csv databses that raise errors.
For the bug related to the export menu and when reopening a saved database, it is because of the cards with alternates, for which I forgot to update the status properties to "True", meaning all data have been downloaded. So the property remains at a value of DBNull (from the row declaration) and all the errors you encountered (Operator '=' is not defined for type 'DBNull' and type 'Boolean') are consequently raised. Mea culpa.
For the RLS Brisela bug, it is now fixed.
The 6.0_beta is released and all the reported bugs should be fixed. I do not guarantee it is free of bugs, as more than 50% of the code needed a rework because of the source transfer from magiccards.into to scryfall.com
For some reason, the 15th Anniversary cards are not showing up. They are set P15A on Scryfall.
When I tried to download them, they said they completed, set is green, I got no errors, but the cards were not added to the database.
I fix all the bugs regarding Scryfall set codes, and robustness of the picture extraction has been improved.
I do not guarantee that it is ree of bugs. It should be stable, but I need to finish the benchmark to make it sure.
Downloading GRN, the tokens' images are downloaded instead of the cards at numbers 1, 2, 3, 4, 5, 7, 8. Not sure why it skipped 6... a low quality image for Conclave Tribunal is downloaded at 6.
@swallowtail23: thank you for your bug report. This was due to priority errors in the search of the scan. This will be fixed in the next release. When scans are stored by card numbers, I need to do something in order to avoid a conflit between normal cards and tokens. The best solution would be to change somehow the numbers of the tokens. It will be implemented in the next release as well.
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.InvalidCastException: Conversion from type 'DBNull' to type 'String' is not valid.
at Microsoft.VisualBasic.CompilerServices.Conversions.ToString(Object Value)
at Gatherer_Extractor.Extract_Fct.Get_Data(DataRow TempRow, String Logs, String LogsSF, String websource, String lang)
at Gatherer_Extractor.FenetrePrincipale.Treatment_MAJ(Int32 nbre_tache, BackgroundWorker worker, DoWorkEventArgs e)
at Gatherer_Extractor.FenetrePrincipale.BWkMAJ_DoWork(Object sender, DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.
@ BrunoTeTa : Thanks for reporting bugs. However, I cannot reproduce the bug. May you please tell me for which card/ID the error is raised? Is it from a fresh extraction, or did you load/reuse a csv database prior to the download?
Gather Extractor offers 3 options:
- original : same image as Scryfall's "large" but with a much bigger file size (ex: Hardened Scales KTK 672x936 579KB)
- full : the same image with the border completely cropped and an even bigger file size (ex: Hardened Scales KTK 620x886 636KB)
- cropped : art crop only
I'd like to ask you to make two improvements to GEx:
1. Please fix the file size of "original". I suppose that the file size difference comes from the compression level which must be modified by GEx at some point, but I'm no expert.
2. Please add more options, at the very least Scryfall's "normal" and possibly "border_crop" as well. I hope this can be done without too much extra coding.
@ Amerigo99
Thank you for your report.
1/ by default, Gatherer Extractor uses only the 'large' png image from Scryfall. There is no compression. This is why you get much larger files. If you want the same size as the large png files from Scryfall (around 200kB per file), I would recommend you to set the Quality option to 95% or 96% instead of 100% by default.
2/ Fullscans & Crops are obtained programmatically from the large png files from Scryfall by a pixel-by-pixel analysis of the border until the inner part of the card is reached. I had to proceed like this because the size of the inner parts depends on the frame, and some sets have cards with very different frames (FNM for example). As a consequence, the size of the final "full" scans may differ from one card to another, by one or two pixels.
I can add the border_crop option with the 10 pxl width and a final resize to 480x680 if desired. This can be done with some extra coding. for the "normal" image, I see no interest in providing them if a better quality (the 'large' ones) is available. I personally don't like the "very large" pngs from Scryfall because of patterning and aliasing from the scanning process which are not completely removed in the final image.
Anyway, I will see what I can do, and I will provide the additional options in one of the next releases.
I didn't know what the scan quality slider was for. I think I understand now and that solves my problem. I was afraid it would add another layer of compression to an already jpg-compressed image. If you start from a png then it's fine.
I think you're right there's no need to add an option for additional image sizes (in pixels). It's better to keep the image at 672x936 and reduce the compression quality. The artifacts created by the jpg lossy compression aren't as noticeable as the lost resolution.
I'll play with the compression quality, no need for other options. Thanks again!
First, great program, and I am not having issues exporting or anything else. I have two entries for Guilds of Ravnica Mythic Edition. If I change one of the set symbols to be different, it changes them for both entries. No other stets have a duplicate entry.
Thank you for keeping update of this useful app. In the latest version I found I couldn't download any data after "Number of Prints" column via "download all data" option. Seems this problem occurs only if I try to get Ravnica Allegiance data; I at least confirmed I can download Japanese card names (which is my primal concern) from Ultimate Masters. Any help is much appreciated!
I was looking for the download of the Guild Kits of Ravnica Allegiance (GK2), but they are not available for download. I see that on Gatherer they have different setnames (Guild Kit Rakdos, Guild Kit Golgari etc.) and on Scryfall the set is simply called Ravnica Allegiance: Guild Kits (GK2).
Is there a way to only read the set data from scryfall?
@yekcat : Columns after "Number of prints" are about foreign data and legalities. To get them, you must check the desired foreign languages in the Options > Extracted Data menu, and the data must be available on Scryfall.
Ususally, foreign data are missing on scryfall when a set is released; those are added several weeks after the release. Concerning the issue you reported, I think patience is your best friend ^^
To get the legalities, you have to download a complete database and calculate them with the button "Compute legalities". Formats are set in the Option > Legalities menu.
@ user-100027134 : v 6.0g is released, in which Guild Kits (GK1 and GK2) data are now only extracted from Scryfall in two separated sets. I also fix minor bugs in the extraction of GK* tokens.
I have successfully downloaded all sets except Release Promos, which fails at Brisela (separate post coming for that).
Note: I skipped all the "online" sets (cube, masters edition, vintage masters, tempest remastered), as I am using GEx to maintain my inventory spreadsheet.
When attempting to open the "Export" menu, unhandled exception error:
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.InvalidCastException: Operator '=' is not defined for type 'DBNull' and type 'Boolean'.
at Microsoft.VisualBasic.CompilerServices.Operators.ConditionalCompareObjectEqual(Object Left, Object Right, Boolean TextCompare)
at Gatherer_Extractor.FenetrePrincipale.ExporterToolStripMenuItem_DropDownOpening(Object sender, EventArgs e)
at System.Windows.Forms.ToolStripDropDownItem.OnDropDownShow(EventArgs e)
at System.Windows.Forms.ToolStripMenuItem.OnDropDownShow(EventArgs e)
at System.Windows.Forms.ToolStripDropDownItem.ShowDropDownInternal()
at System.Windows.Forms.ToolStripMenuItem.OnMenuAutoExpand()
at System.Windows.Forms.MenuTimer.OnTick(Object sender, EventArgs e)
at System.Windows.Forms.Timer.OnTick(EventArgs e)
at System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3190.0 built by: NET472REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
Gatherer Extractor
Assembly Version: 3.8.0.0
Win32 Version: 3.8.0.0
CodeBase: file:///D:/Gatherer%20Extractor/Gatherer%20Extractor.exe
----------------------------------------
Microsoft.VisualBasic
Assembly Version: 10.0.0.0
Win32 Version: 14.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3190.0 built by: NET472REL1LAST_C
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3190.0 built by: NET472REL1LAST_C
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
Accessibility
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/Accessibility/v4.0_4.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
----------------------------------------
System.Runtime.Remoting
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Runtime.Remoting/v4.0_4.0.0.0__b77a5c561934e089/System.Runtime.Remoting.dll
----------------------------------------
System.Data
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3160.0 built by: NET472REL1LAST_C
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_32/System.Data/v4.0_4.0.0.0__b77a5c561934e089/System.Data.dll
----------------------------------------
System.Numerics
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Numerics/v4.0_4.0.0.0__b77a5c561934e089/System.Numerics.dll
----------------------------------------
System.Data.DataSetExtensions
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Data.DataSetExtensions/v4.0_4.0.0.0__b77a5c561934e089/System.Data.DataSetExtensions.dll
----------------------------------------
************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
For example:
<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>
When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.ArgumentException: Argument 'Start' must be greater than zero.
at Microsoft.VisualBasic.Strings.InStr(Int32 Start, String String1, String String2, CompareMethod Compare)
at Gatherer_Extractor.Extract_Fct.Get_Logs_side(String Logs, String websource)
at Gatherer_Extractor.Extract_Fct.Get_Data(DataRow TempRow, String Logs, String LogsSF, String websource, String lang)
at Gatherer_Extractor.FenetrePrincipale.Treatment_MAJ(Int32 nbre_tache, BackgroundWorker worker, DoWorkEventArgs e)
at Gatherer_Extractor.FenetrePrincipale.BWkMAJ_DoWork(Object sender, DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3190.0 built by: NET472REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
Gatherer Extractor
Assembly Version: 3.8.0.0
Win32 Version: 3.8.0.0
CodeBase: file:///F:/Gatherer%20Extractor/Gatherer%20Extractor.exe
----------------------------------------
Microsoft.VisualBasic
Assembly Version: 10.0.0.0
Win32 Version: 14.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3190.0 built by: NET472REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3190.0 built by: NET472REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
System.Runtime.Remoting
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Runtime.Remoting/v4.0_4.0.0.0__b77a5c561934e089/System.Runtime.Remoting.dll
----------------------------------------
System.Data
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3160.0 built by: NET472REL1LAST_C
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_32/System.Data/v4.0_4.0.0.0__b77a5c561934e089/System.Data.dll
----------------------------------------
System.Numerics
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Numerics/v4.0_4.0.0.0__b77a5c561934e089/System.Numerics.dll
----------------------------------------
Accessibility
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Accessibility/v4.0_4.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
----------------------------------------
System.Data.DataSetExtensions
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3056.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Data.DataSetExtensions/v4.0_4.0.0.0__b77a5c561934e089/System.Data.DataSetExtensions.dll
----------------------------------------
************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
For example:
<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>
When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.InvalidCastException: Operator '=' is not defined for type 'DBNull' and type 'Boolean'.
at Microsoft.VisualBasic.CompilerServices.Operators.ConditionalCompareObjectEqual(Object Left, Object Right, Boolean TextCompare)
at Gatherer_Extractor.FenetrePrincipale.Perform_Test_Fully_Handled_Sets()
at Gatherer_Extractor.FenetrePrincipale.BGW_OpenBDD_RunWorkerCompleted(Object sender, RunWorkerCompletedEventArgs e)
at System.ComponentModel.BackgroundWorker.OnRunWorkerCompleted(RunWorkerCompletedEventArgs e)
at System.ComponentModel.BackgroundWorker.AsyncOperationCompleted(Object arg)
I confirm that you must start from an empty database. Otherwise, there will be conflicting IDs or unknown IDs with outdated csv databses that raise errors.
For the bug related to the export menu and when reopening a saved database, it is because of the cards with alternates, for which I forgot to update the status properties to "True", meaning all data have been downloaded. So the property remains at a value of DBNull (from the row declaration) and all the errors you encountered (Operator '=' is not defined for type 'DBNull' and type 'Boolean') are consequently raised. Mea culpa.
For the RLS Brisela bug, it is now fixed.
The 6.0_beta is released and all the reported bugs should be fixed. I do not guarantee it is free of bugs, as more than 50% of the code needed a rework because of the source transfer from magiccards.into to scryfall.com
Feel free to report any bugs you may find.
Enjoy!
For some reason, the 15th Anniversary cards are not showing up. They are set P15A on Scryfall.
When I tried to download them, they said they completed, set is green, I got no errors, but the cards were not added to the database.
RLS successfully downloaded. Woot!
I fix all the bugs regarding Scryfall set codes, and robustness of the picture extraction has been improved.
I do not guarantee that it is ree of bugs. It should be stable, but I need to finish the benchmark to make it sure.
Enjoy!
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.InvalidCastException: Conversion from type 'DBNull' to type 'String' is not valid.
at Microsoft.VisualBasic.CompilerServices.Conversions.ToString(Object Value)
at Gatherer_Extractor.Extract_Fct.Get_Data(DataRow TempRow, String Logs, String LogsSF, String websource, String lang)
at Gatherer_Extractor.FenetrePrincipale.Treatment_MAJ(Int32 nbre_tache, BackgroundWorker worker, DoWorkEventArgs e)
at Gatherer_Extractor.FenetrePrincipale.BWkMAJ_DoWork(Object sender, DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
Gatherer Extractor
Assembly Version: 3.8.0.0
Win32 Version: 3.8.0.0
CodeBase: file:///D:/Jogos/Gatherer%20Extractor/Gatherer%20Extractor.exe
----------------------------------------
Microsoft.VisualBasic
Assembly Version: 10.0.0.0
Win32 Version: 14.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3081.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
System.Runtime.Remoting
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Runtime.Remoting/v4.0_4.0.0.0__b77a5c561934e089/System.Runtime.Remoting.dll
----------------------------------------
System.Data
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3081.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_32/System.Data/v4.0_4.0.0.0__b77a5c561934e089/System.Data.dll
----------------------------------------
System.Windows.Forms.resources
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms.resources/v4.0_4.0.0.0_pt-BR_b77a5c561934e089/System.Windows.Forms.resources.dll
----------------------------------------
mscorlib.resources
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/mscorlib.resources/v4.0_4.0.0.0_pt-BR_b77a5c561934e089/mscorlib.resources.dll
----------------------------------------
System.Numerics
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Numerics/v4.0_4.0.0.0__b77a5c561934e089/System.Numerics.dll
----------------------------------------
Accessibility
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Accessibility/v4.0_4.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
----------------------------------------
System.Data.DataSetExtensions
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Data.DataSetExtensions/v4.0_4.0.0.0__b77a5c561934e089/System.Data.DataSetExtensions.dll
----------------------------------------
System.resources
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.resources/v4.0_4.0.0.0_pt-BR_b77a5c561934e089/System.resources.dll
----------------------------------------
************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
For example:
<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>
When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.
First of all I'd like to thank you for creating Gatherer Extractor and taking the time to update it after the demise of magiccards.info.
It has been a wonderful tool for downloading card scans up to know and I hope I can keep using it.
I do, however, have a problem with the card images version 6.0_delta downloads from Scryfall.
Scryfall offers several image sizes and formats which are listed here : https://scryfall.com/docs/api/images
- png : very large image and file size (ex: https://img.scryfall.com/cards/png/en/ktk/133.png 745x1040 1815KB)
- large : large jpg image (ex: https://img.scryfall.com/cards/large/en/ktk/133.jpg 672x936 265 KB)
- normal : smaller jpg image, close to the size of old magiccards.info (ex: https://img.scryfall.com/cards/normal/en/ktk/133.jpg 488x680 144 KB)
- border_crop : slightly larger image than "normal" but with a large part of the border cropped, very close to what magiccards.info used to offer (ex: https://img.scryfall.com/cards/border_crop/en/ktk/133.jpg 480x680 159 KB)
- small : very small jpg image, too small for most uses (ex: https://img.scryfall.com/cards/small/en/ktk/133.jpg 146x204 16KB)
- art_crop : a crop of the art only
Gather Extractor offers 3 options:
- original : same image as Scryfall's "large" but with a much bigger file size (ex: Hardened Scales KTK 672x936 579KB)
- full : the same image with the border completely cropped and an even bigger file size (ex: Hardened Scales KTK 620x886 636KB)
- cropped : art crop only
I'd like to ask you to make two improvements to GEx:
1. Please fix the file size of "original". I suppose that the file size difference comes from the compression level which must be modified by GEx at some point, but I'm no expert.
2. Please add more options, at the very least Scryfall's "normal" and possibly "border_crop" as well. I hope this can be done without too much extra coding.
Thanks again for all the hard work!
Thank you for your report.
1/ by default, Gatherer Extractor uses only the 'large' png image from Scryfall. There is no compression. This is why you get much larger files. If you want the same size as the large png files from Scryfall (around 200kB per file), I would recommend you to set the Quality option to 95% or 96% instead of 100% by default.
2/ Fullscans & Crops are obtained programmatically from the large png files from Scryfall by a pixel-by-pixel analysis of the border until the inner part of the card is reached. I had to proceed like this because the size of the inner parts depends on the frame, and some sets have cards with very different frames (FNM for example). As a consequence, the size of the final "full" scans may differ from one card to another, by one or two pixels.
I can add the border_crop option with the 10 pxl width and a final resize to 480x680 if desired. This can be done with some extra coding. for the "normal" image, I see no interest in providing them if a better quality (the 'large' ones) is available. I personally don't like the "very large" pngs from Scryfall because of patterning and aliasing from the scanning process which are not completely removed in the final image.
Anyway, I will see what I can do, and I will provide the additional options in one of the next releases.
Thank you for your answer.
I didn't know what the scan quality slider was for. I think I understand now and that solves my problem. I was afraid it would add another layer of compression to an already jpg-compressed image. If you start from a png then it's fine.
I think you're right there's no need to add an option for additional image sizes (in pixels). It's better to keep the image at 672x936 and reduce the compression quality. The artifacts created by the jpg lossy compression aren't as noticeable as the lost resolution.
I'll play with the compression quality, no need for other options. Thanks again!
Personally I would love to get the maximum size PNG files (745x1040 I think) from Scryfall if it's not too much trouble to add in a future version.
Thanks again for this great tool!
I was looking for the download of the Guild Kits of Ravnica Allegiance (GK2), but they are not available for download. I see that on Gatherer they have different setnames (Guild Kit Rakdos, Guild Kit Golgari etc.) and on Scryfall the set is simply called Ravnica Allegiance: Guild Kits (GK2).
Is there a way to only read the set data from scryfall?
Thanks a lot!
@yekcat : Columns after "Number of prints" are about foreign data and legalities. To get them, you must check the desired foreign languages in the Options > Extracted Data menu, and the data must be available on Scryfall.
Ususally, foreign data are missing on scryfall when a set is released; those are added several weeks after the release. Concerning the issue you reported, I think patience is your best friend ^^
To get the legalities, you have to download a complete database and calculate them with the button "Compute legalities". Formats are set in the Option > Legalities menu.
@ user-100027134 : v 6.0g is released, in which Guild Kits (GK1 and GK2) data are now only extracted from Scryfall in two separated sets. I also fix minor bugs in the extraction of GK* tokens.
Best regards!