Scrape/autoadd metadata to roms?

Here you can post your ideas, scripts, dats, images, mods, themes, etc...
Tormentor667
Posts: 111
Joined: 09 Jun 2013, 13:00

Scrape/autoadd metadata to roms?

Unread post by Tormentor667 » 09 Jun 2013, 19:15

Just a small question, because I don't know yet if something's already there or if it would make up for a good feature request.

Is there already something like a metadata scraper/auto-adder in ECC? I think it would be a lot of work to manually add developer, year and all the other stuff to the rom metadata with ECC. A dos equivalent to ECC called D-Fend Reloaded already has something like this. It automatically searches for info over at mobygames (or other trusted sources)
snapshot.png
As a reference, it uses the name of the rom, and the user himself can choose the right one out of various possibilities.

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 09 Jun 2013, 19:28

Yes, this could be interesting, is there any info about this? (like a API for mobygames.com) so i do not have to reinvent the wheel again, just write some Autoit3 code to import some data...

Good feature request!, but i need some more info's though....i will have a look later-on!
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

Tormentor667
Posts: 111
Joined: 09 Jun 2013, 13:00

Re: Scrape/autoadd metadata to roms?

Unread post by Tormentor667 » 09 Jun 2013, 19:35

As far as I know, there is no API. Instead, D-Fend Reloaded parses the text content, which is easily achieved I guess. It checks for the search results, displays the different choices and then (if chosen), loads the info according to the results. Unfortunately I am not a programmer, so I do not have an idea how D-Fend Reloaded solves this really.

But you can take a look at the program here
http://dfendreloaded.sourceforge.net/?lang=en

I am sure it might be very useful to add something like this. I am not sure if there are other sites that are worth being implemented, but I guess MobyGames has most of the stuff that people might be interested in. At least, it works perfect for DOS games, my whole Database is fed up :)

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 10 Jun 2013, 01:17

Well i see, that dfend is only loading the page of the game (name based yes), and is not putting any content "automaticly" into the database?

The options in defend are (as read from "Bin\SearchLinks.txt")

Code: Select all

Mobygames.com;http://www.mobygames.com/search/quick?game=<GAMENAME>&p=-1&search=Go
Squakenet.com;http://www.squakenet.com/pc_games/step2.asp?game_title=<GAMENAME>
TheLegacy.de;http://www.thelegacy.de/Museum/SQLFrameset.php3?SearchValue=<GAMENAME>&searchEntity=TITLE&misc=yes&quick=yes&list=SQLlist_games.php3
KultPower.de;http://www.kultpower.de/kultpower_search.php?search=<GAMENAME>
Google.com;http://www.google.com/search?q=<GAMENAME>
Wikipedia.org;http://en.wikipedia.org/wiki/<GAMENAME>
It will not be easy to get data from the (messy) HTML output, but i'll give it a shot!, first i need to build a "proof of concept" before continue this...
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

Tormentor667
Posts: 111
Joined: 09 Jun 2013, 13:00

Re: Scrape/autoadd metadata to roms?

Unread post by Tormentor667 » 10 Jun 2013, 13:50

It doesn't put it automatically into the database without any user confirmation, because sometimes you get more than one result. But it's still better than putting everything into the database manually I guess :) I hope you can implement this, it would be awesome :)

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 10 Jun 2013, 16:22

1) Could you post a pop-up screen from dfend?
2) So dfend let's you import the data with a single mouseclick?
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

Tormentor667
Posts: 111
Joined: 09 Jun 2013, 13:00

Re: Scrape/autoadd metadata to roms?

Unread post by Tormentor667 » 10 Jun 2013, 16:26

1) What do you exactly mean with Popup screenshot?
2) Exactly, I choose the apropriate game from D-Fend Reloaded's suggestions, click "import" and then the program imports the whole info, even including screenshots.

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 10 Jun 2013, 16:41

A screenshot of the pop-up screen in dfend to import the data :lmfao:

Well, let me do some magic in the next days...
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

Tormentor667
Posts: 111
Joined: 09 Jun 2013, 13:00

Re: Scrape/autoadd metadata to roms?

Unread post by Tormentor667 » 10 Jun 2013, 21:50

No problem :)

1).
infodemo1.jpg
2).
infodemo2.jpg
infodemo2.jpg (18.92 KiB) Viewed 2178 times
3).
infodemo3.jpg

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 11 Jun 2013, 00:08

Wow he really pulled it off!, this is loads of work to build from scratch!, i will email the dfend author, to see if he wants to help us...

Stay tuned!
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

Tormentor667
Posts: 111
Joined: 09 Jun 2013, 13:00

Re: Scrape/autoadd metadata to roms?

Unread post by Tormentor667 » 11 Jun 2013, 12:19

I will :) Looking forward to this!

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 16 Jun 2013, 21:20

Ok here is the reply, i have to study his datareader .PAS file, wich i will when i found some time :yes_head: :smartass:

Code: Select all

Van: Alexander Herzog
Verzonden: zondag 16 juni 2013 19:59
Aan: phoenixinteractive
Onderwerp: Re: Import basic info from mobygames.com

Hello Sebastiaan,

I'm not using any special API to access MobyGames. I'm just reading the html code and filtering the interessting data from there. You can find the code used to interpret the MobyGames html code in the DataReader*.pas files in the D-Fend Reloaded source code zip file. The structure of the MobyGames homepage is not hard coded in the program code and can be configured via the \Tools\DataReaderServer\DataReader.xml file also available in the source code zip file.

I'm not sure the program code will help you much to add a MobyGames information reader to your program, but feel free to use the D-Fend Reloaded code for this.

Kind regards
   Alexander
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

User avatar
DerMicha75
ECC2 (java) Developer
Posts: 521
Joined: 05 Feb 2007, 22:14
Location: Germany
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by DerMicha75 » 16 Jun 2013, 21:44

Look at ECC2, there is this feature working :-)

Micha

Tormentor667
Posts: 111
Joined: 09 Jun 2013, 13:00

Re: Scrape/autoadd metadata to roms?

Unread post by Tormentor667 » 16 Jun 2013, 23:17

Awesome :)

BTW (offtopic) what's the difference between ECC1 and ECC2? (as ECC2 doesn't seem to be official)

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 16 Jun 2013, 23:28

Tormentor667 wrote:Awesome :)

BTW (offtopic) what's the difference between ECC1 and ECC2? (as ECC2 doesn't seem to be official)
Like the forum description says:
DerMicha75 wrote:I want to show you what I'm doing some evenings in my spare time.
At first I want to say that this is ONLY A PRIVATE HOBBY PROJECT, I don't want to "kill" eCC, But eventually wake up Andreas a bit ;-)
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

User avatar
DerMicha75
ECC2 (java) Developer
Posts: 521
Joined: 05 Feb 2007, 22:14
Location: Germany
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by DerMicha75 » 19 Jun 2013, 22:38

This was years ago my motivation to do something. In the meantime I think it is nearly stable, lets say "beta". So this is an alternative option. Its still my hobby project and I decide what comes in. But its usable, works, and has some nice features that I missed in ECC1. I mange my own collection with this and play my games with it.

Phoenix gave me the forum as an presentation place for this tool. I have permission to use the platform images. But this is currently not announced in other forums.

Fell free to try it out, give me feedback and I will continue to develop.

Micha

User avatar
Phoenix
ECC Developer 2006-2016
Posts: 9057
Joined: 27 Aug 2006, 01:17
Location: Deventer, The Netherlands
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by Phoenix » 20 Jun 2013, 01:01

@DerMicha,

Have you already used my API for the ICC server? :D
Sebastiaan Ebeltjes | ECC Developer 2006-2016 | ECC Forum Admin | Phoenix Interactive WebMaster
[- ECC programs -]
eccUpdate, eccScriptSystem, GtkThemeSelect, DatFileUpdater (DFU), ImagePackCenter (IPC), eccDiagnostics, 3dGallery, iccImageInject
eccKameleonCode, eccCreateStartmenuShotcut, eccThirdPartyConfig (TPC), EmuMoviesDownloader (EMD), eccVideoPlayer
MobyGamesImporter (MGI), ECC Amiga GetGemusConfig.

User avatar
DerMicha75
ECC2 (java) Developer
Posts: 521
Joined: 05 Feb 2007, 22:14
Location: Germany
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by DerMicha75 » 20 Jun 2013, 08:34

aaaehhhmmmmm.. no. Emumovies was enough for me at the moment. But I'lll give it a try.

Micha

Tormentor667
Posts: 111
Joined: 09 Jun 2013, 13:00

Re: Scrape/autoadd metadata to roms?

Unread post by Tormentor667 » 20 Jun 2013, 20:40

DerMicha75 wrote:This was years ago my motivation to do something. In the meantime I think it is nearly stable, lets say "beta". So this is an alternative option. Its still my hobby project and I decide what comes in. But its usable, works, and has some nice features that I missed in ECC1. I mange my own collection with this and play my games with it.
So what features actually? :)

User avatar
DerMicha75
ECC2 (java) Developer
Posts: 521
Joined: 05 Feb 2007, 22:14
Location: Germany
Contact:

Re: Scrape/autoadd metadata to roms?

Unread post by DerMicha75 » 21 Jun 2013, 23:04

try it out :-)

Ma favorites are:

- video playback
- configurable user interface
- extras handling (Manuals, maps, ...)
- EmuMovies downloader
- Mobygames info grabber :-)
- ...

Micha

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests