Scrape/autoadd metadata to roms?

Here you can post your ideas, scripts, dats, images, mods, themes, etc...
Tormentor667
Posts: 111
Joined: 09 Jun 2013, 13:00

Re: Scrape/autoadd metadata to roms?

Unread post by Tormentor667 » 23 Jun 2013, 15:11

Is it possible to directly migrate from ECC1 to ECC2?

User avatar
DerMicha75
ECC2 (java) Developer
Posts: 521
Joined: 05 Feb 2007, 22:14
Location: Germany
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by DerMicha75 » 23 Jun 2013, 22:55

Hi,

not direct. You have to do:

- export your metadata
- export your images
- copy over the roms in the new directories
- import the roms, import metadata and import images

Micha

Tormentor667
Posts: 111
Joined: 09 Jun 2013, 13:00

Re: Scrape/autoadd metadata to roms?

Unread post by Tormentor667 » 30 Jun 2013, 14:02

Doesn't sound too difficult, maybe I'll try that out as I am a real big fan of the autoscrape feature :)

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 01 Jul 2013, 18:58

It has to be after my vacation, haha but i will see if i can have a prototype ready to test soon!
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 31 Jul 2013, 16:38

Okey, i made a "simple" beta scraper for mobygames, this is merely a fist step :smartass: , i have to make everything from scratch, the code from D-Fend is in pascal and too complicated with XML reader stuff ect, i made this withouth to many complications, and seems to be working :yes_head: , i'll test it some more soon!

example link: http://www.mobygames.com/game/atari-2600/james-bond-007_
Result:
mobygames_data.png
mobygames_data.png (10.7 KiB) Viewed 1257 times

Code: Select all

#include <String.au3>
#include <Array.au3>

Global $String

Global $Cache = BinaryToString(InetRead("http://www.mobygames.com/game/atari-2600/james-bond-007_", 1)) ;Get data from the website

;Publisher
Global $Publisher = "Unknown" ;Default value
$Publisher_tmp_f1 = _StringBetween($Cache, "Published by</div>", "</div>") ;Get the "publisher" line
$Publisher_tmp_f2 = _StringBetween($Publisher_tmp_f1[0], "<a href=", "</a>") ;Stripdown the line some more
$Publisher_tmp_f3 = StringSplit($Publisher_tmp_f2[0], ">") ;Stripdown the line some more
If UBound($Publisher_tmp_f3) > 2 Then $Publisher = CleanString($Publisher_tmp_f3[2]) ;Get the rightmost part ([1] is the left part)

;Developer
Global $Developer = "Unknown" ;Default value
$Developer_tmp_f1 = _StringBetween($Cache, "Developed by</div>", "</div>") ;Get the "developer" line
$Developer_tmp_f2 = _StringBetween($Developer_tmp_f1[0], "<a href=", "</a>") ;Stripdown the line some more
$Developer_tmp_f3 = StringSplit($Developer_tmp_f2[0], ">") ;Stripdown the line some more
If UBound($Developer_tmp_f3) > 2 Then $Developer = CleanString($Developer_tmp_f3[2]) ;Get the rightmost part ([1] is the left part)

;Release year
$Released_tmp = _StringBetween($Cache, "Released</div>", "</a></div>") ;Get the "release" line
Global $Released = StringRight($Released_tmp[0], 4) ;Get the 4 rightmost characters

;Genre
Global $Genre = "Unknown" ;Default value
$Genre_tmp_f1 = _StringBetween($Cache, "Genre</div>", "</div>") ;Get the "genre" line
$Genre_tmp_f2 = _StringBetween($Genre_tmp_f1[0], "<a href=", "</a>") ;Stripdown the line some more
$Genre_tmp_f3 = StringSplit($Genre_tmp_f2[0], ">") ;Stripdown the line some more
If UBound($Genre_tmp_f3) > 2 Then $Genre = CleanString($Genre_tmp_f3[2]) ;Get the rightmost part ([1] is the left part)

MsgBox(64, "Found data", "Developer: " & $Developer & @CRLF & "Publisher: " & $Publisher & @CRLF & "Released: " & $Released & @CRLF & "Genre: " & $Genre)
Exit

Func CleanString($String)
$String = StringReplace($String, "&nbsp;", " ") ;Space
Return $String
EndFunc
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 02 Aug 2013, 10:17

Okey, here is an improvement of the above script, now getting more data like description, also release year has been fixed to get more then the "year" alone...

When i have a decent script i'll release it as a "test package" for you all to test it out on moby games!

Example link: http://www.mobygames.com/game/windows/i-of-the-dragon

Found data:
mobygames_scrape_02.png
mobygames_scrape_02.png (20.9 KiB) Viewed 1255 times
Mobygames scraper v0.2a:

Code: Select all

#include <String.au3>
#include <Array.au3>

Global $String

Global $Cache = BinaryToString(InetRead("http://www.mobygames.com/game/windows/i-of-the-dragon", 1)) ;Get data from the website

;Publisher
Global $Publisher = "Unknown" ;Default value
Dim $Publisher_tmp_f2, $Publisher_tmp_f3
$Publisher_tmp_f1 = _StringBetween($Cache, "Published by</div>", "</div>") ;Get the "publisher" line
If UBound($Publisher_tmp_f1) > 0 Then $Publisher_tmp_f2 = _StringBetween($Publisher_tmp_f1[0], "<a href=", "</a>") ;Stripdown the line some more
If UBound($Publisher_tmp_f2) > 0 Then $Publisher_tmp_f3 = StringSplit($Publisher_tmp_f2[0], ">") ;Stripdown the line some more
If UBound($Publisher_tmp_f3) > 2 Then $Publisher = CleanString($Publisher_tmp_f3[2]) ;Get the rightmost part ([1] is the left part)

;Developer
Global $Developer = "Unknown" ;Default value
Dim $Developer_tmp_f2, $Developer_tmp_f3
$Developer_tmp_f1 = _StringBetween($Cache, "Developed by</div>", "</div>") ;Get the "developer" line
If UBound($Developer_tmp_f1) > 0 Then $Developer_tmp_f2 = _StringBetween($Developer_tmp_f1[0], "<a href=", "</a>") ;Stripdown the line some more
If UBound($Developer_tmp_f2) > 0 Then $Developer_tmp_f3 = StringSplit($Developer_tmp_f2[0], ">") ;Stripdown the line some more
If UBound($Developer_tmp_f3) > 2 Then $Developer = CleanString($Developer_tmp_f3[2]) ;Get the rightmost part ([1] is the left part)

;Release year
Global $Released = "Unknown" ;Default value
Dim $Released_tmp_f2, $Released_tmp_f3
$Released_tmp_f1 = _StringBetween($Cache, "Released</div>", "</div>") ;Get the "released" line
If UBound($Released_tmp_f1) > 0 Then $Released_tmp_f2 = _StringBetween($Released_tmp_f1[0], "<a href=", "</a>") ;Stripdown the line some more
If UBound($Released_tmp_f2) > 0 Then $Released_tmp_f3 = StringSplit($Released_tmp_f2[0], ">") ;Stripdown the line some more
If UBound($Released_tmp_f3) > 2 Then $Released = CleanString($Released_tmp_f3[2]) ;Get the rightmost part ([1] is the left part)

;Genre
Global $Genre = "Unknown" ;Default value
Dim $Genre_tmp_f2, $Genre_tmp_f3
$Genre_tmp_f1 = _StringBetween($Cache, "Genre</div>", "</div>") ;Get the "genre" line
If UBound($Genre_tmp_f1) > 0 Then $Genre_tmp_f2 = _StringBetween($Genre_tmp_f1[0], "<a href=", "</a>") ;Stripdown the line some more
If UBound($Genre_tmp_f2) > 0 Then $Genre_tmp_f3 = StringSplit($Genre_tmp_f2[0], ">") ;Stripdown the line some more
If UBound($Genre_tmp_f3) > 2 Then $Genre = CleanString($Genre_tmp_f3[2]) ;Get the rightmost part ([1] is the left part)

;Description
Global $Description = "Unknown" ;Default value
$Description_tmp_f1 = _StringBetween($Cache, "Description</h2>", "<div class=") ;Get the "description" line
If UBound($Description_tmp_f1) > 0 Then $Description = CleanString($Description_tmp_f1[0])

MsgBox(64, "Found data", "Publisher = " & $Publisher & @CRLF & "Developer = " & $Developer & @CRLF & "Released = " & $Released & @CRLF & "Genre = " & $Genre & @CRLF & @CRLF & "Description = " & $Description)
Exit

Func CleanString($String)
$String = StringReplace($String, "&nbsp;", " ") ;Space
$String = StringReplace($String, "<i>", "")
$String = StringReplace($String, "</i>", "")
Return $String
EndFunc
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 04 Oct 2013, 16:13

Ok, i can also write data to the database... now i've got all pieces together, but the whole puzzel has to be created, so expect a data "scraping" this month for ECC :yes_head:
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 05 Oct 2013, 15:05

Ok, i've programmed a litlle bit, the first beta testing looks promising ;-), here are some screens! :sorcerer:
MGI_beta_screen_01.png
MGI_beta_screen_01.png (13.36 KiB) Viewed 1190 times
MGI_beta_screen_02.png
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 06 Oct 2013, 00:59

Okey, next screen :sorcerer:
mobygamesimporter_screen_03.png
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

User avatar
ToJPhantom
Posts: 12
Joined: 01 Mar 2010, 06:57
Location: Pleasant Plain, OH
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by ToJPhantom » 21 Oct 2013, 23:37

This looks great really excited for it to be released!
I'm the one who steps from the shadows, all trenchcoat and cigarette and arrogance, ready to deal with the madness.

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 22 Oct 2013, 01:56

ToJPhantom wrote:This looks great really excited for it to be released!
It's for 80% finished, still some work todo by addng it proper to the SQL database ("add" function), and i have to deal with some characters that are not allowed in the SQL query... like ", %, <, >, etc.
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 31 Oct 2013, 01:42

Ok, it's finished!! (and still in OCT -2013 :sweatdrop: ), you can download/update your ECC installation with eccUpdate: viewtopic.php?f=28&t=5042

Let me know if you like it! :lmfao:
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest