Ok,

Since I extracted the RES files for messing with the DoD GUI I've had like a billion PM's about the GCF extractor and getting a copy. So heres the current state of affairs.

I did at first have a semi working gcf extractor which interfaced with the Steam.dll to do its dirty work. However, its was really buggy and impractical and some of the code used was, how shall we say, from an "unclean" source.

However, I've since abandoned that version and decided to make a "white" version from scratch that is stand alone and doesnt require any steam dependancy.

Now because the first version just interfaced into Steam and was "aware" of the gcf file structure I didn't have much thinking to do, but now as this version has to do it all by itself I have to try and figure out the gcf format itself.

So far progress has been slow but I've had some success. I've partly worked out the gcf indexing and how to extract certain types of files but I would say I'm only about 15% of the way done at this point.

Also, the tools I'm using are complete hacks and aren't standalone applications. I have to change the source code for every extract and re-compile to tool - which is obviously NOT practical.

So, at this stage, yes, an extractor of sorts exists but its not in any releaseable form and is potentially to destructive if used by someone who doesnt know how it works.

I will release a gcf extractor or maybe a partial toolset ONCE I'm happy it works, its clean and I wont be sent to jail under the terms of the DCMA.

- Jed

Is it programmed in C++?

Originally posted by Cs42886
Is it programmed in C++?

Parts of it. I'm not a C++ programmer really. Some are actually written in Perl, one part (res extractor) is in PHP believe it or not.

- Jed

Best of luck with it Jed. If you can pull this off, I'm sure the MSA community would have your babies.:p

Good work fella!!!.

Well the thing is, if you download and install the win32 dedicated server and let it update DoD you get all the files in the GCF unpacked anyway.

The only extra files that dont come with the dedicate server are several resource files and pre-cache lists.

I know its a lot to download a 400Mb file just to get the content hence the reason for zipping up the sprites and models for you.

- Jed

If you need any help with it, I think I might be able to help you out in the C++ and 2D areas.:)

Jed, you truly are the king of kings.

Originally posted by Locomotion
Best of luck with it Jed. If you can pull this off, I'm sure the MSA community would have your babies.:p

Good work fella!!!.


Not just this MSA... If you read around other forums (ie: Counterstrike, etc...) They are all praying for a .gcf extractor too... I think Jed would have his hands full with marriage proposals... :D


...

Originally posted by Effexx
Not just this MSA... If you read around other forums (ie: Counterstrike, etc...) They are all praying for a .gcf extractor too... I think Jed would have his hands full with marriage proposals... :D


...

Usch from CS kiddies...

voice_negative


- Jed

i need the GCF Extractor ... Pls .... What is the curren Status ? in % pls :)

Originally posted by [SoS]Schlex
i need the GCF Extractor ... Pls .... What is the curren Status ? in % pls :)

While my reading skills may have degraded recently, I do believe the man said it was around 15% done...

Originally posted by Trp. Jed
Usch from CS kiddies...

voice_negative


- Jed


I agree.. I don't play CS, but while searching for models and such I've ocasionally encountered a CS forum where they are begging for such a tool... I'm sure if/when you do release your extractor it will undoubtedly filter it's way into the grubby hands of the Counterstrikers... :(


...

Originally posted by Effexx
I agree.. I don't play CS, but while searching for models and such I've ocasionally encountered a CS forum where they are begging for such a tool... I'm sure if/when you do release your extractor it will undoubtedly filter it's way into the grubby hands of the Counterstrikers... :(


...

Whats the problem with another HL community getting to use a program that helps everyone out. I don't even play CS but I think it's kinda ridiculous and childish to diss on CS every chance you get.

You look at the percentage of *******s in their community compared to the percentage of *******s in ours then I'd bet its close to the same. They just have more people total so you run into more of them.

I was going to code an extractor too, then I realised that I dont control the format of the GCF, and its very very likely to change. Especially seeing as though, Valve do not want people to be able to easily extract files from the GCF.

You will be trying to score between moving goal posts. Best of luck to you!

I'm not really bothered. I'm just using it as a mental exercise to stop me getting bored in the evenings.

Yeah, Valve can change the file format but they'd be pretty paranoid to do it just because someone wrote an extractor. I mean the GCF is a virtual file system and hence has the overhead of a FAT like system. To change that spec would require a complete re-write of the Steam GCF API and for what? Because someone found out how to read a file out of it?

I mean its not like the files aren't available via the HLDS install anyway and all thats missing is a few piddly little resource files which don't really affect much so its not exposing exploits or anything.

Anyway, if a half decent programmer wanted to they can just interface to Steam.dll anyway. The file system API is plainly visibly with any tools such as the MS dependancy viewer and it only requires a bit of working out of what functions to call in what order really.

I don't think Valve has anything to worry about reallly.

- Jed

just release what you made already under a fake name to an unknown person through a plain source, BAM done. oh wait...you want notoriety. months we wait, so be it. at least it will give you something to do in the meantime while you are home sick. feel better and good luck.

Noteriety? God know, I bloody hate the attention. Im only writing this cuz it keeps my mind of other things.

As I said before and I'll say again all I have is some hacky scripts (ones even in PHP) that automate some small tasks but a lot of it still requires a lot of manual work and combing through the GCF with a hex editor to get the numbers to punch into the scripts anyway.

Incedently I got an email from Waldo saying someone at Valve was interested in what I was doing (lord know why - the invented the format, why do they need my utility?) and could they get a copy of what I have so far.

I've tried replying to Tim but he hasn't answered his Valve e-mail address and his PM box is always full. What I wanted to ask him was to check if its OK for me to publicly publish what I know of the GCF format so far on-line so that its "open source", so to speak, so that if anyone else wants to use it as a base to write their own utilts they can.

- Jed

I for one am interested in the file spec...........

wow valve need a gcf extractor .. hmm jed !! make a good price´for this tool ......

I'll post the spec as soon as I've finished working it out and can be sure I'm not going to get myself or anyone else in trouble for doing so.

I don't feel the need to be all nefarious and release it anonymously as what i'm trying to do isn't going to destroy steam - but these days, and after the HL2 leak, the last thing I need is someone threatening me under the rules of the DCMA.

As for the spec...

Well I know HOW the files are stored in terms of the order and I've cracked the info part of the GCF header which lists the GCF cache number, revision and internal size.

I've also got quite far on cracking the file table format which is essentially the FAT of the GCF file. Basically it tells you what files are in the GCF, how big they are and what name they have.

Theres a lot of parameters I havent yet figured out what they do, its possible these are file dates or flags for if the entry is a file or directory, etc.

To be honest, what I'm looking for is the "missing link" - how to determine the start of the file in the GCF so it can be extracted. I know how to find the filename and its length, I just need to pinpoint its start and then thats enough to get the file out at least.

- Jed

Originally posted by Trp. Jed
I don't feel the need to be all nefarious and release it anonymously as what i'm trying to do isn't going to destroy steam - but these days, and after the HL2 leak, the last thing I need is someone threatening me under the rules of the DCMA. Not like there'd be any mystery at this point anyway. ;)

Hmmm seems the files are split into 8k blocks but their not sequential.

client.dll for example is split over 8 places in the GCF in chunks of 32k and 64k.

Now if I could just figure out if it actually stores the offset where each block starts or has and index saying "file X is composed of blocks 1,2,7,8, etc"

- Jed

Not that I know anything about the subject, but that seems like a ver inefficient method of storing files, perhaps a means of keeping people from peeking inside?

No idea to be honest. I mean GCF is technically a virtual file system and I suppose file systems get fragmented.

Look at it this way, when we get an update via steam, what happens?

Say anzio.bsp took up 128 8k blocks, then some revisions to it make it larger, so now it needs 132 8k blocks.

Now, if everything was sequential that would mean that you need to take everything AFTER where anzio.bsp is stored, move it to ram or somewhere else temporary, replace the file and then put everything back at the end and re-index where they all start. And thats just for one file...

If you devide your file into 8k blocks you can overwrite the ones the file is already using, then tag a few more 8k blocks to the end of the file and then you just have to update your file table to tell it in which blocks the file exists.

That and make sure that your filetable has room for more files than you need. At the moment DoD has about 1600 files in the GCF but by my calculations the FAT of the GCF allows for around 41,000 (odd number, 32,768 would of seemed more likely).

Ah well, we shall see. I'm really not sure how I'm going to figure the block allocation out.

- Jed

You must be like a lucky charm Trigger :D

No sooner do I type out that explanation than I figure out the the block assignments.

- Jed

Keep going Jed, you'll get there in the end. What you do with it in the end, is another question entirely. :D

Theres only about 2 things I need to work out before I have enough of the spec to write a functioning extractor.

I need to work out how to calculate how large the block assignment table will be based on how many records there are in the GCF - if I can do that I'll know where the name look-up table starts and then that means I'll know where ever file is in the GCF, its name and size, where in the GCF it starts and what 8k blocks need to go together to finalise the file.

The only other part is to try and work out how it stores the layout - i.e. which folders and files go together.

- Jed

Well I'm going to take a break from working on the extractor for a few days cuz staring at a Hex editor is making me cross eyed.

Anyway, this isnt an extractor, just a tool I hacked up to help me read the GCF file header and parts of the FAT table within it.

- Jed

Oh and heres the output from it, this example is for platform.gcf as it stood tonight.

Its tab delimited so its best viewed in something Excel or OpenOffice.

- Jed

You never know about the DMCA these days. In New York, you get 3 years for walking into a movie theater with a gun and robbing it, and you get 5 years for walking into a movie theater with a video camera.

One step closer for the halflife kind.....:D

Great work Jed.

Originally posted by Effexx
sure if/when you do release your extractor it will undoubtedly filter it's way into the grubby hands of the Counterstrikers... :(
...


just code it to delete CS everytime the prgm is executed, problem solved.

Originally posted by Whiskas
just code it to delete CS everytime the prgm is executed, problem solved.

why thats so diabolical and evil...yet i love it...

but instead of deleting it, why not just make it not work on cs no matter what. the cs'ers will invade these forums if it deletes it on them.

Make sure it extracts the cs_aztec fog sprite first.

Why, 2ltben... Are you saying that you play CS? :D

Get the pitchforks and torches, guys!

Just kidding! :D

Well for those that havent seen it yet (Wolf Kahler pointed it out to me):

http://www.natural-selection.org/forums/index.php?act=ST&f=19&t=55059&

I've tried it but can't get it to work, keeps giving me error's about not being able to write to the dump directory. Its not very user friendly either.

The fact that it required steam to be running and logged on makes me a bit suspicious. Its obviously interfacing to the STEAM FS calls in Steam.dll which means either they've worked out all the functions and parameters from the DLL using a DLL viewer or used the spec from the HL2 source leak. If it was the latter, I *think* it may be legal as long as they didnt actually include any of the source.

Anyhew, theres an extractor if you need one. I'll keep fiddiling with mine, or rather I'll at least try and finish cracking the file format. The goal for mine was to be stand alone and to allow you to extract files individually, hence its taking me somewhat longer.

- Jed

Keep at it Jed :).

Maybe have a DoD background on the main window once it's done :p :D?

Originally posted by Trp. Jed
Well for those that havent seen it yet (Wolf Kahler pointed it out to me):

http://www.natural-selection.org/forums/index.php?act=ST&f=19&t=55059&

I've tried it but can't get it to work, keeps giving me error's about not being able to write to the dump directory. Its not very user friendly either.

The fact that it required steam to be running and logged on makes me a bit suspicious. Its obviously interfacing to the STEAM FS calls in Steam.dll which means either they've worked out all the functions and parameters from the DLL using a DLL viewer or used the spec from the HL2 source leak. If it was the latter, I *think* it may be legal as long as they didnt actually include any of the source.

Anyhew, theres an extractor if you need one. I'll keep fiddiling with mine, or rather I'll at least try and finish cracking the file format. The goal for mine was to be stand alone and to allow you to extract files individually, hence its taking me somewhat longer.

- Jed It works perfectly for me... :D I've dumped the hl directory and dod dir. :D Now, what can I do, tha I couldn't before?

Originally posted by Bijiy in NS forums
OMG, I can now have access to the wad used in dod_switch. THANK YOU!!!


i guess he hasn't heard of avalanche.

Originally posted by Faceman
i guess he hasn't heard of avalanche.

LMAO :D

- Jed

there is a working gcf extractor already around.
it's called steam dump-it.

i could easily extract cs 1.6 and dod 1.1

p.s: application no of dod is 31

LOL. Did you read the earlier posts? LOL

- Jed

Aren't you working on a winzip type program Jed so that people can have individual files rather than just dumping all of the files out?

So the two programs are quite different Raingrove :p.

Yes, my ultimate goal is to have something that allows you take take individual files out. Its also independant of steam, i.e. it doesnt interface to the Steam API like Steam Dump It does.

To inteface to the API you need to know what internal functions to call and in what order. Those are available in the HL2 source leak but I was a bit unsure about using that info.

So, instead I've actually cracked the GCF format using a hex editor, a calculator and hours of logical deduction.

- Jed

Got an updated ETA then? :) :D :o

Originally posted by Raingrove
there is a working gcf extractor already around.
it's called steam dump-it.

i could easily extract cs 1.6 and dod 1.1

p.s: application no of dod is 31
I am very suspicious to this program. I downloaded it to see if this was true. But the instructions says that you have to enter your email... strange. I don't trust 3rd party software that instructs you to enter such information, since it's not even documented what it does with the email, and the risk of having your account stolen.
And the bottom line in the Read Me says this: - Offered to you by OGC Cheats 2003.
No way I am running that program.

Well this is how it works...

It interfaces to Steam.dll, which exports the following functions to it:

SteamMountFilesystem
SteamUnmountFilesystem
SteamLogin
SteamGetAppIds
SteamIsAppSubscribed
SteamClearError
SteamFindFirst
SteamFindNext
SteamFindClose
SteamOpenFile
SteamSizeFile
SteamReadFile
SteamCloseFile
SteamLogout
SteamCleanup

Now, it needs steam running first of so that it will have access to the VFS.

The reason it needs your email is because thats what your user folder is called in steam (using SteamLogin). It needs to pass that to Steam to recieve back the path to where the GCF files are and what GCF Application ID's are currently installed for that user (SteamGetAppIds and SteamIsAppSubscribed) it can then mount the GCF and read the files out.

I've run Steamdump it and I have a very secure firewall and at NO TIME did it attempt to transmit or recieve anything over the network.

So I am confident all it needs your email address for is so it can pass it to Steam to get the paths and locations of your GCF files.

- Jed

Originally posted by Red Squirrel
Got an updated ETA then? :) :D :o

No *sigh*

Im getting to the parts now which are really giving me a headache - i.e block allocations per file and working out how it stores the directory structure... :(

Truth is, if my C++ was better and I wasn't using Visual Basic I could probably of done the same as Steam DumpIt and interfaced into the steam.dll. Trouble is I've never had to do external libraries in VB before and I'm rusty enough as it is.

- Jed

Originally posted by Trp. Jed
So, instead I've actually cracked the GCF format using a hex editor, a calculator and hours of logical deduction.

- Jed

You're a mad man.:eek:

Indeed. Super smart:eek:

And super bored and super sick of it...

I'm just mad because if my C++ was better I could of done this ages ago but Im effectively using a mallet to crack a walnut here.

I'll probably put up what I've found of the spec on my website soon. Im losing motivation to keep on cracking the rest and maybe someone else can spot the missing link I cant find.

- Jed

Originally posted by Zyndrome
Offered to you by OGC Cheats 2003.

Where do you see that? I don't see it in The Thing's post, nor in the readme.txt, nor in the other .txt that comes with it...

Or are you talking about a completely different one? Raingrove is talking about Steam Dump-It, which I thought was the one I told Jed about [that's what its files are called, anyway]... Of course, the link The Thing gives is a direct download and not the file's proper release page, so I can't see its full documentation, which is why I'm asking, since I don't know.

Anything with the OGC label on it is automatically on my black list. There's no way in hell I'm supporting that sort of organization.

It's called the "I havn't been sleeping for 48 hours and my IQ is below zero plus I have no food at home" phenomena, thank you very much. :P

we all hate OGC for making cheat tools... but we gotta admit that they are damned smart.

Well if it is from the same guys that make the cheats then I wouldn't be suprised if they just used the HL2 source code to figure out how to write the extractor. All the API functions for file acess in Steam are in the filesystem_stdio branch.

- Jed

That is why you're slaving over hex code, so not to use that.

How are your eyes treating you? Feel better and take your time.

My eyes are better now I've finally got new glasses. LOL.

Trouble is the medicine I have to take for my illness ups my serotonin so my body clocks all out and I keep falling asleep at the keyboard.

Narcolepsy anyone? :D

- Jed

Originally posted by Trp. Jed
Well if it is from the same guys that make the cheats then I wouldn't be suprised if they just used the HL2 source code to figure out how to write the extractor. All the API functions for file acess in Steam are in the filesystem_stdio branch.

- Jed

even just reading the source code is illegal, but oh well...

be careful what you are discussing here please ;)

I don't know what it's doing, but if you use the steam dumpit program, you'll find that it connects to steam and downloads the content from the steam servers; I opened up my steam monitor when I dumped the cs gcf and the monitor showed me as updating CS.

It doesn't really EXTRACT the gcf, I'm thinking, so much as download the content from the steam server.

Well thats pretty dumb...

They should of checked to see it if it was locally available using a function I dare not mention less I be spanked by a mod :D

- Jed

:cool:

Is this dead in the water then?

:( :( :(

Originally posted by Red Squirrel
Is this dead in the water then?

:( :( :(

Nope, I just havent had a lot of free time lately. Im going to put what I've found of the GCF format up on my website soon anyway.

- Jed

gl with it, it's nice to see someone trying to give back to the community. hope that you can get done what ya need to get done.


and, i believe that it's legal to read the hl2 source just not to use any of it for the purpose of profit.

?
link (http://www.svencoop.com/forums/showthread.php?threadid=16857)

another beta GCF extractor at
http://countermap.counter-strike.net/Nemesis/index.php?p=1

it is by Nemisis, who many of you know already for using Nem's batch compiler for compiling maps.

here is his thread where he talks about what he decoded, just like Jed:
http://countermap.counter-strike.net/Nemesis/index.php?c=73#c304

Of limited interest, but I started writing in "human readable form" what I've de-crypted of the GCF format so far.

http://www.wunderboy.org/3d_games/utils/gcfformat.php

I've forwarded this info to Ryan Greg (Nem) to see if it helps him and I'm hoping we can collaborate on cracking the format.

Better to put your heads together than duplicate your work.

- Jed

Progress ;)

And as usual, a well written article.

O.K. In case anyone missed it, Ryan (Nem) has relased a working Beta of GCFScape that lets you browse and extract files from the GCF individually. Not it is still a beta and still WIP.

http://countermap.counter-strike.net/Nemesis/

Also Ryan and I have now agreed to work together on cracking the GCF format. He's done a lot of .NET work and I'm currently going over his notes and writing out a more human friendly set of documention for the specification.

- Jed

OH, MY GOD! HE'S BEEN WORKING ON THAT FORMAT FOR TWO OR THREE YEARS!!! WHAT AN INVESTMENT! O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O O_O

Of course, I suppose I can sort of say I've been working on my clan's site for two years now... I'm sure it's not hardly any kind of comparison, but wow.

Thanks for sticking to it, Jed! :D [yes, I'm aware Jed and Nem are different people]

Well Nem and I had cracked the same stuff by early December but Nem made more headway up to Xmas.

As most of his stuff is in .NET, something I'm not familar with I let him take the programming lead of things and Im just acting as another set of eyes to try and figure a few final bits out and document it really.

GCFScape has some bugs but it will extract MDLs ok.

- Jed

I suppose .NET is now the new VB in terms of having to download special drivers to get it to work?

As in, I need to get the .NET Framework in order to use this, despite having the newest Windows?

Well I have XP Pro and I didn't have the .NET framework installed by default and had to install it. Sadly its about 23 Mb :(

- Jed

Thank you for the rapid reply. :)

That's a damned shame, though. Ah, well.

At least there are none of the legal snares (far as I can see) in GCFScape that there are in the DumpIt program, which must connect to Steam to work. You HAVE to turn off Steam to use Scape.

I've extracted all sorts of things using it, and it seems to work fine. Even the entire sound folder. It's no more buggy than the old PAK Explorer.

While I acknowledge that it does suck that you have to download a 23 MB runtime just to run an executable that is only a few KB I make no apologies for it. I spend a great deal of my time developing utilities and I use whatever language best suits the application. The .NET framework supports several languages (C++, C#, VB and more) and is extremely robust and comprehensive. If it means I can shave several hours off of routine programming I see no reason not to use it.

I'm pretty sure no one would complain about it if it came loaded with your OS like most of the other common runtimes out there. The main reason it doesn't is because it was released after Windows XP but rest assured it will be part of future Windows Operating Systems.

Hell you can always look at it like this: It would be a hell of a lot more work on my part to rewrite all my .NET applications then it would be on your part just to download the bloody thing.

Nem

Nooooooooooooooooooooooooooo! If I came across as if I were complaining or berating you in ANY WAY, I sincerely appologize. I did not intend to sound that way AT ALL.

I have nothing but respect for the work you've done. Seeing Jed talk about it, I've gotten a clear picture of at least how difficult the GCF format is to figure out.

My reserves about the .NET framework have NOTHING to do with you or ANYTHING you've done, Nem.

Again, I'm terribly sorry if I gave you a personally negative impression in the least.

No worries. I didn’t mean to come off as though I was singling you out or indeed attacking you at all. I’ve had a fair amount of understandable complaints (namely from 56 K users) and I just wanted to address why I used the .NET framework and that it isn’t just a matter of switching the .NET code off.

How bout a funny picture to lighten the mood?

http://countermap.counter-strike.net/Nemesis/f/47.jpg

Nem
=

hehe another one...




-----------------


ok enough of that. glad you guys are working together. 2 heads are better than 1. ;)